Banking Got Its AI Explainability Rules First. Your Industry Is Next

At some point, a manager, customer, or regulator will question a specific decision made by your AI agent. If your company or team can’t explain the reasoning behind it, trust can erode quickly, and the agent(s) that once made your company efficient is now a liability.

Being able to show why a system produced a particular result is called AI explainability, and it’s quickly becoming the difference between an AI program people can trust and one that poses unexpected risk.

Banking hit the AI explainability wall first.

Last April, the Federal Reserve, the Office of the Comptroller of the Currency, and the FDIC issued revised interagency guidance on model risk management, published by the Federal Reserve as SR 26-2. This guidance replaces a 2011 standard created before most current models existed. Generative and agentic AI are outside the scope of the guidance, but its principles apply to other AI models. Regulation B has long required lenders to provide specific reasons for denying credit, and “the algorithm said no” isn’t a good answer.

The U.S. has generally addressed AI developments reactively, but there are signs this is starting to shift. In July, the ranking member of the House Financial Services Committee requested input on AI in financial services to help develop a modern federal framework that encourages responsible innovation while protecting consumers and markets.

State regulators are adopting a model bulletin from the National Association of Insurance Commissioners, which expects insurers to maintain a written program governing AI use and outlines what regulators may request during examinations. As of April 2026, 25 jurisdictions have adopted it. Internationally, the EU AI Act established a unified framework across all sectors, with obligations that scale to the system's risk level.

Industries are converging on the same requirements. They want managers to document how their AI tools work, name who owns them, and be ready to explain a single decision months later.

As a rule of thumb, regulated industries receive these rules first, while others typically adopt similar standards later. If your company is involved with consumer finance, health, or safety, it’s best to begin preparing for AI explainability requirements now.
 

Take The Marshmallow Test

On the surface, moving fast looks like the best response to competitive pressure, especially when new technology promises to boost productivity. But when companies implement tools before establishing clear guardrails, departmental managers discover, test, and adopt solutions on their own. The approach accelerates progress at first, but it can also create an unmanageable patchwork of AI tools and inconsistent practices that are harder to reverse than they would have been to prevent.

Governance may at first seem like the adult version of the marshmallow test, but putting enough structure in place to move forward responsibly is what keeps the rollout from turning into chaos later. Therefore, yes, adopt technology, but do it in this order:
 

• Governance

• Inventory

• Evaluation

• Implementation

• Monitoring

The good news is that each step tends to reduce the cost of the next.
 

Governance Comes Before The Tool

First, assign ownership of the AI tool. Second, determine which data may leave your environment. Third, establish a policy for when human approval is required and how those decisions are documented.

Finally, run the exercise most governance frameworks skip. Write down what could go wrong with each tool, who it would affect, and what you would do about it. If a customer-facing assistant says something it shouldn’t, the response plan matters, and auditors treat a documented what-if as evidence that someone considered risk before it occurred.

You don’t need to start your governance plan from scratch. The AI Risk Management Framework from the National Institute of Standards and Technology is free and thorough enough for organizations of any size. The Financial Services AI Risk Management Framework tailors that approach for financial services.

An initial draft of the framework is sufficient, provided it is in place before you deploy the tools. This process should take a couple of weeks.
 

Take Inventory Of The AI Already Inside Your Company

This step usually surprises executives. Microsoft and LinkedIn’s 2024 Work Trend Index found that 75% of knowledge workers were already using AI at work, often without informing management. McKinsey’s 2025 State of AI survey reported organizational adoption at 88%, up from 78% the previous year.

Employees may be using AI on their own to speed up work. Vendors have built it into products you licensed for other purposes, and departments often develop their own workarounds. You may already be using dozens of AI tools and not know it.

Cataloging current AI use is deeply unglamorous work, but every step that follows depends on knowing what tools you’re already using.
 

Evaluate The Work, Then Evaluate The AI Vendor

Rather than asking, “Where should we use AI?” focus on identifying repetitive, daily tasks across the organization.

Know that AI cares about tasks, not org charts. For example, contract reviews happen in nearly every organization and crosses departments. Legal or procurement staff review vendor agreements, compare them to standard terms, flag discrepancies, and summarize the results for approval. The work is time-consuming and largely standardized, making it seem like a strong candidate for automation.

But exposure alters that assessment. The contract may create a long-term commitment. It could be subject to legal discovery or contain confidential pricing. In those cases, a person must stay in the decision chain. AI could produce the first read and the first draft of the summary, but whoever signs that agreement will need to own the judgment.

Next, evaluate the vendor. Since most organizations don’t train foundation models in-house, much of your explainability depends on someone else outside of your company. Your primary tool will be the questionnaire you provide before entering an agreement.

Banking has a head start here, and its buyers have settled on a set of questions worth asking any AI vendor. It should cover every part of your governance framework:
 

• Which model is being used, and who trained it?

• Who’s allowed to see our data, and what data actually gets fed to the model?

• What data leaves our environment, where does it go, and which country hosts it?

• Is customer data retained, and under what retention policy?

• Can outputs be explained, and at what level of detail?

• Where does human review sit in the workflow?

• How does the vendor monitor hallucinations and model drift?

• What documentation, audit logging, and update history does the vendor provide so we don’t have to build them ourselves?

Pay particular attention to the final question. Vendors serving regulated markets are accustomed to detailed questionnaires, so ask to see the documentation itself. You want documentation, not empty reassurance.
 

Decide Who Owns The AI Program

Select a use case with quick returns and run it with a small team, since good habits are easier to establish at that scale.

Before expanding beyond the initial tool, designate a single individual to oversee the AI program, including governance and compliance. This person doesn’t need to be the CTO but must be able to address explainability questions from managers or regulators.

Communicate with end users about the tool’s purpose, its limitations, and how it will affect their daily work. People rarely resist the technology itself. They resist not knowing what it means for them.
Training should extend beyond basic operation, too. Users must understand how the system functions, recognize an answer that looks right but isn’t, and know when to override the result.
 

Monitor For Model Drift

AI systems evolve. Models are updated regularly, and outputs that were accurate in March may drift by September. New risks arise as regulations change.

Monitoring should be part of your regular operations, alongside policy reviews and internal audits. Measure actual usage rather than assuming adoption, because tools may be underused or used in ways you never intended.
 

What Does An Auditor Actually Ask About AI?

During a first AI audit, companies are often caught off guard by how little the auditor looks at the technology. Attention falls on the processes and controls surrounding it.

Auditors want to know whether you can explain outputs, monitor the system, maintain documentation, assign ownership, and defend decisions to customers or in court. Those criteria apply equally to credit grading and resume screening.

Documentation will save you, but it’s also the part companies most often treat as busywork. A written record of what you decided and why gives you something to stand on.

Follow these steps, and you can move fast without breaking things.

Abrigo provides risk management, financial crime prevention, and lending software to more than 2,400 financial institutions.