Banks are rapidly introducing generative and agentic AI into customer-facing apps, support systems and operational processes, raising difficult questions about how these technologies should be tested, monitored and controlled.
Unlike traditional software, autonomous AI systems may respond differently when presented with the same or similar inputs. A successful test run therefore provides evidence of past performance, but no guarantee that the system will behave identically during its next customer interaction.
For quality assurance teams, this demands a significant shift in approach. Fixed expected outcomes must increasingly be supplemented by statistical testing, adversarial probing, continuous production monitoring and runtime controls capable of stopping unsafe actions as they happen.
Karanjot Jaswal, co-founder and CTO of Cinchy, argued that banks must combine these disciplines with comprehensive telemetry and clearly defined permission boundaries if they are to deploy AI safely and produce credible evidence for regulators.
“Start from the fact that these systems are non-deterministic,” he said. “A clean test record is evidence, not a guarantee, because the same input can produce a different output tomorrow.”
That changes the role of quality assurance teams. Instead of simply comparing expected and actual results against a fixed specification, testers must examine patterns of behaviour across large numbers of interactions and determine whether the resulting risk remains acceptable.
“Testing then shifts to a statistical discipline layered on top: running scenarios at volume and asserting on pass rates and distributions to understand how the system behaves across the range of inputs and probing the guardrails to confirm they hold,” Jaswal explained.
“Defining what’s ‘acceptable’ is now a risk decision QA owns jointly with compliance, not a spec it inherits.”
Testing the AI disclosure layer
The need for more robust assurance is becoming increasingly pressing as the EU AI Act introduces transparency requirements for certain AI systems.
Banks will need to demonstrate that customers are properly informed when they are interacting with AI, regardless of whether the conversation takes place through a mobile app, chatbot or customer-support platform.
Jaswal said firms should treat those disclosures as a testable software component rather than relying on a standard message added during development.
“Banks should build a disclosure test suite like a regression suite: direct questions (‘am I talking to a human?’), adversarial attempts to get the system to claim it’s human, mid-conversation handoffs and channel switches,” he said.
“A clean test record is evidence, not a guarantee.”
Karanjot Jaswal
Those tests should run continuously because changes to a model, system prompt or surrounding application can alter the way the disclosure behaves.
“Run it continuously, because every model or prompt update can silently change behaviour,” Jaswal warned.
Cross-channel consistency will also be critical. Customers should receive the same fundamental disclosure regardless of which application, chatbot or support channel they use.
“Cross-channel consistency is important; the same question should produce a materially equivalent disclosure,” he said. “Where channels are run by different vendors, the risk of gaps is greatest.”
Reconstructing the causal chain
Testing whether an AI system produced an acceptable response is only one part of the assurance challenge. Banks must also be able to establish why the response was generated and which data, systems and permissions were involved.
Jaswal said the appropriate standard is “full session reconstruction”, covering the customer’s input, the instructions available to the AI, information returned by retrieval systems and every subsequent action.
“The standard is full session reconstruction: what the customer said, what context and instructions the agent had, what the retrieval layer surfaced and from where, every tool or API call with inputs and outputs, and any policy check that fired,” he said.
Achieving that level of traceability requires telemetry across the entire AI architecture rather than logs captured only at the model interface.
“That requires logging at the model, retrieval, action and decision layers, held in tamper-evident storage with correlation IDs tying AI telemetry back to bank records,” Jaswal explained.
Such evidence could become crucial when customers challenge an automated payment, account change or other AI-assisted action long after the original interaction.
“When a customer disputes an action six months later, the question won’t be ‘what did the model say’ but ‘show me the complete causal chain from utterance to account change.’”
Runtime controls become the first line of defence
Because autonomous AI behaviour cannot be guaranteed through pre-release testing, Jaswal said hard constraints must be enforced while the system is operating.
“The hard constraints, such as never disclosing another customer’s data or acting outside permitted tools and limits, have to be enforced at runtime by guardrails that impose those boundaries and validate outputs on every interaction, independent of the model,” he said.
The same principle applies after deployment. Historical testing results cannot, on their own, provide a sufficient safety case for a system whose behaviour may change between interactions.
“Past performance isn’t a safety case, so the first line of defence is runtime enforcement,” Jaswal stressed.
“Guardrails need to constrain and validate every interaction as it happens, and high-consequence actions like payments and credit decisions should always be human gated.”
“Testing autonomous systems demands statistical thinking, adversarial creativity and risk judgment.”
– Karanjot Jaswal
Continuous production testing should operate alongside those controls. Jaswal recommended synthetic testing against known scenarios, ongoing assessment of real customer interactions and drift monitoring across both system inputs and outputs.
“Continuous testing then sits alongside that including synthetic probing of the live system against known scenarios, scoring of real interactions against the same rubrics used pre-release, and drift detection on inputs and outputs, with quality degradation triggering graduated human review,” he said.
Certain failures should trigger an immediate shutdown rather than further automated processing.
“The kill switch should fire on data leakage between customers, actions outside entitlements, successful prompt injection, hallucination of account facts, or anomalous tool-call volumes.”
Evidence must be built into the architecture
Banks will also need reliable evidence that AI-driven decisions stayed within approved permissions, internal policies and risk limits.
That evidence cannot be assembled after an incident if the original system architecture did not enforce and record the relevant controls, Jaswal warned.
“You can’t produce that evidence retroactively if the architecture doesn’t enforce it,” he said.
“The strongest evidence is that the model couldn’t have exceeded its limits: scoped, least-privilege entitlements enforced by the platform, so an out-of-limit request fails at the authorisation layer regardless of what the model generates.”
That technical boundary should be supported by documented links between policies and controls, immutable records of the checks applied during each interaction and continuing adversarial tests.
“Layer on documented policy-to-control mappings, immutable per-interaction records of which checks ran and their outcomes, and results of ongoing adversarial testing showing the controls hold,” Jaswal said.
“‘The model is well-behaved’ is not an assurance statement; ‘the model operates inside a permission boundary we control, test and can evidence’ is.”
Third-party models as untrusted
The assurance problem becomes more complex when financial institutions use foundation models supplied and updated by third parties.
Vendor evaluations and red-team reports can provide useful information, but they cannot demonstrate how a model will perform within a specific bank, customer journey or data environment.
“The honest starting point is that you cannot fully assure a third-party model,” Jaswal said.
“Vendor evaluations, red-team reports are useful context, but they describe the model in general, not your application, your data and your customers, and they can be invalidated overnight by an update you didn’t control.”
Banks should consequently treat external models as untrusted components and retain control over application-specific testing, guardrails and telemetry.
“The assurance strategy should treat the model as an untrusted component and build the controls around it: your own testing on your own use cases, your own runtime guardrails validating what goes in and comes out, and your own telemetry, so your evidence doesn’t depend on the vendor’s cooperation,” Jaswal said.
Procurement teams should meanwhile negotiate version pinning, advance notification of model changes, clear data-handling provisions and defined incident-reporting requirements.
“What procurement should then secure from the vendor is not proof of good behaviour but the conditions that make your controls workable, e.g., version pinning and advance notice of changes, so your test results stay valid, clear data-handling terms, and incident notification,” he explained.
“You can outsource the model, you cannot outsource the accountability, and the contract should be written with that asymmetry in mind.”
For QA professionals, the rise of autonomous AI will ultimately broaden the testing discipline rather than diminish its importance.
“The industry conversation is dominated by what models can do,” Jaswal concluded. “Assurance and testing discipline are becoming the licence to deploy, and the banks that build that muscle will ship faster, not slower.”
“For QA teams, this is an expansion of the profession, not a threat. Testing autonomous systems demands statistical thinking, adversarial creativity and risk judgment, and it deserves a discipline of its own rather than a checkbox bolted onto existing processes.”
NEXT MONTH
REGISTER TODAY – SIMPLY CLICK HERE
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
READ MORE
- Goldman puts AI coding to the test
- How to test AI models that banks do not control
- OpenAI, Filigran and SunTec: the latest vendor and product news
- Sygnum: Testing AI is ‘a measurement problem’
- Banks’ ‘code for all’ push raises testing risks
QA FINANCIAL PODCASTS
CLICK HERE TO LISTEN TO OUR EXCLUSIVE CONVERSATIONS
