AI Explainability in Banking Operations: What Regulators and RFPs Are Asking

Ask a bank's model risk team, its compliance department, and the procurement lead running an AI vendor RFP what "explainability" means, and you'll get three different answers. The examiner wants documentation proving someone can reconstruct why a model did what it did. Compliance wants specific, defensible reasons they can put in a customer-facing letter. Procurement wants a checkbox in a vendor questionnaire that a legal team will accept. None of these are technical requests for SHAP values or attention-weight visualizations — and treating explainability as a modeling problem, rather than a documentation and process problem, is why so many banking AI projects stall in review.

What "Explainability" Actually Means for a Bank

In machine learning research, explainability usually means being able to point to which input features drove a specific output — a SHAP value, an attention map, a feature-importance score. That's technical explainability, and it matters, but it's the smallest part of what a bank is actually being asked to produce.

Three other kinds of explainability sit alongside it, and each has its own audience and its own evidence standard:

• Governance explainability — can the model risk function reconstruct, from documentation and logs, why a model was approved, what it was tested against, and who owns it. This is what SR 11-7 examiners are actually checking for, and it's covered in depth in our breakdown of SR 11-7 for LLMs.
• Decision-level explainability — for any single output (a declined loan, a flagged transaction, a fraud alert), can you produce the specific factors that drove it, in language a customer or a regulator can read.
• Consumer-facing explainability — the legally mandated version of decision-level explainability. In the US, the Equal Credit Opportunity Act and Regulation B require lenders to give applicants specific, principal reasons for an adverse credit action, not a generic statement that a model produced a low score.

What Regulators Are Actually Asking For

None of the major frameworks bearing on bank AI use demand a specific explainability technique. They demand evidence that a decision can be reconstructed and defended after the fact.

Under SR 11-7, explainability shows up inside the "conceptual soundness" pillar of model risk management: examiners expect documentation of why a model works, what data and assumptions it embeds, and independent validation that it behaves as intended. We cover the full three-pillar translation for LLMs in SR 11-7 Model Risk Management for LLMs.

Under the EU AI Act, most banking AI use cases (credit scoring, fraud detection) fall into the high-risk category, which carries explicit technical documentation and logging obligations, plus a requirement that high-risk systems be designed to allow human oversight — including the ability for a human overseer to correctly interpret the system's output before acting on it.

And under US consumer protection law, the Equal Credit Opportunity Act's Regulation B requires specific, accurate reasons on every adverse action notice. "The model said no" is not a valid reason. If a bank cannot trace an AI-assisted credit decision back to the specific factors that drove it, in plain language, it has a fair lending and Reg B compliance gap regardless of how accurate the underlying model is.

What Shows Up in RFPs

Procurement teams evaluating an AI vendor for a banking use case have started putting explainability questions directly into RFPs, and the pattern is consistent enough now to list. The questions that show up most often:

• Can the system produce a plain-language explanation for any individual decision, on demand, without engineering involvement?
• What is logged for every decision — inputs, retrieved context, intermediate reasoning, and the final output — and for how long is it retained?
• Can the buyer's own model risk or compliance team query that log independently, or does every explanation request have to go back through the vendor?
• What happens when the underlying model or prompt changes — is a previously generated explanation still reproducible after the update?
• For credit or lending use cases specifically: can the system map its output to the specific reason codes a Reg B adverse action notice requires?

A vendor that can't answer the first two questions concretely is not ready for a regulated deployment, whatever its model accuracy looks like on a benchmark.

Building for Explainability from Day One

Explainability is far cheaper to build in than to retrofit. A few practical patterns we see work in production banking deployments:

• Log the reasoning, not just the output. A one-line action summary satisfies nobody. Capture the retrieved context, the relevant policy or underwriting criteria referenced, and the intermediate steps that led to the final answer — the same tamper-evident logging discipline covered in our agent audit framework.
• Separate the decision factors from the prose. If a model's output feeds a credit or account decision, extract the specific factors driving it into a structured field before it ever becomes customer-facing language. That structured field is what a Reg B reason code, or an examiner's sampling request, actually needs.
• Version the explanation alongside the model. When the base model, prompt, or retrieval corpus changes, treat it as a new version for explainability purposes too — a previously generated explanation should stay reproducible against the model version that produced it, not silently reinterpreted by whatever is running today.
• Build the human-review checkpoint into the architecture, not the policy document. The EU AI Act's human-oversight requirement, and most internal banking policies, only hold up if a human reviewer can actually see the explanation before the action takes effect — not just theoretically escalate to one.

Where This Fits in the Broader Governance Picture

Explainability isn't a standalone workstream — it's one input into the model risk and governance structure covered in AI Agent Governance in Banking and tested directly by the sampling methodology in How to Audit AI Agent Decisions in Banking. An auditor sampling agent decisions is, in practice, testing explainability evidence: can this specific decision be reconstructed and defended. Treat the three as one program, not three separate compliance exercises.

Explainability failures in banking AI are rarely modeling failures — they're logging, documentation, and process failures that show up as an examiner finding, a fair-lending complaint, or a stalled RFP. Talk to a Sphere AI Engineer about architecting an AI deployment that produces the explanation, audit-log, and human-oversight evidence regulators and procurement teams are already asking for.