Banks have spent the past decade transforming software testing. Test automation, quality engineering and DevOps have become deeply embedded across financial services, helping firms release software faster while maintaining resilience and control.
But a new generation of software is exposing the limits of those approaches.
Applications are increasingly capable of producing different outputs from the same input, reasoning across vast amounts of information and changing their behaviour over time, sometimes without any code changes by the organisation itself.
In some cases they fail not through crashes or error messages, but by producing convincing, plausible answers that happen to be wrong.
For testing teams, that presents a profound challenge. How do you prove that a system is behaving correctly when there is no single correct answer? How do you perform regression testing on software that can subtly degrade even when nothing in the application has changed?
A new paper from researchers at global software and technology consultancy Thoughtworks warns that these questions require an entirely new discipline: evaluation engineering.
The paper, AI Assurance: A Comprehensive Testing Strategy for AI Systems, contends that the systems now entering production “introduce a class of risks that traditional software quality assurance was never designed to address.”
The authors argued that existing approaches to software testing are suffering from “a structural mismatch between the assurance model and the system it is trying to verify.”
That message is likely to resonate with banks, many of which are simultaneously accelerating software delivery, experimenting with autonomous agents and copilots, and grappling with how to assure systems whose behaviour cannot be validated through traditional deterministic testing.
“The core problem,” the authors wrote, “is that these systems are probabilistic infrastructure.”
The end of verification
For decades, software testing has been built around a relatively simple principle: given the same input, a system should produce the same output.
The new generation of systems challenges that assumption. “The same input can produce different outputs on different runs,” the paper noted. “These properties make traditional verification impossible and make continuous evaluation mandatory.”
That observation may prove uncomfortable for many quality engineering teams. Much of modern software testing, from automated regression suites to pass-fail release gates, relies on determinism. If outputs can vary and behaviour can shift over time, the entire philosophy of testing begins to change.
The paper argued that organisations need to abandon the notion that these systems can ever be proven correct.
“The goal of testing AI systems is not to certify that the system is correct,” the authors stressed. Rather, testing should “reduce uncertainty”, “bound failure” and “build confidence over time,” they shared.
In other words, testing becomes less about verification and more about risk management. That shift could have profound implications for financial institutions.
A conventional software defect generally produces visible symptoms. A transaction fails. An interface breaks. A service becomes unavailable.
The new failure modes are very different. The paper points to “confident hallucinations, silent behavioural drift and coordination failures that produce wrong answers visually indistinguishable from correct ones.”
For banks, those are not merely software defects. They are potential conduct risks, operational resilience risks and, increasingly, regulatory risks.
Regression testing
One of the paper’s strongest messages concerns regression testing. In traditional software engineering, regression testing typically accompanies code changes and release cycles. The assumption is that if nothing changes, behaviour should remain stable.
That assumption no longer holds. According to the authors, cloud providers can update models on timelines outside the control of development teams, while “a model update by a cloud provider can silently change behaviour across every prompt in the application.”
The paper stated that “prompt regression testing running continuously not just at release is the only reliable defence against silent behavioural degradation.”
For banks, this could become one of the defining testing challenges of the next few years.
Many financial institutions are already asking how they should validate code generated by copilots, how they should test autonomous agents and how they can prove that controls continue to operate effectively as systems evolve.
The answer proposed by the authors is the creation of continuous evaluation infrastructure that monitors behaviour over time, identifies degradation trends and triggers investigation before problems reach production.
“A single passing run provides weak evidence,” the researchers wrote. “A consistent pattern of high scores across multiple inputs over weeks provides strong evidence.”
The comparison the authors make is particularly striking. The equivalent of testing in these environments, they argue, is no longer the traditional test suite. “It is the CI/CD pipeline.”
The emergence of evaluation engineering
Perhaps the paper’s most provocative argument is that organisations require an entirely new discipline. The authors argued that “traditional QA roles are insufficient for maintaining evaluation infrastructure at scale.”
Instead, they propose a new field: evaluation engineering.
The discipline sits somewhere between software engineering, data curation and behavioural science. It involves building and maintaining evaluation datasets, designing rubrics, monitoring trends, creating adversarial scenarios and converting production incidents into new forms of test coverage.
“Evaluation infrastructure should be treated as a shared platform capability,” the paper read, rather than being rebuilt separately for each project.
This could have significant implications for banks.
Over the past decade, financial institutions have invested heavily in test automation platforms and quality engineering centres of excellence. Evaluation engineering suggests the next stage of that evolution may require entirely new skills and new organisational models.
The authors draw an explicit comparison with the industry’s earlier move towards DevOps and CI/CD.
“This shift from project-level test suites to platform-level evaluation infrastructure is the organisational equivalent of the move from per-project CI scripts to a shared CI/CD platform.”
New testing architecture
The paper also proposes a new assurance model built around layers of evaluation that move from deterministic infrastructure testing at the base to business outcome testing at the top.
The message is not that traditional testing disappears.
In fact, the authors repeatedly emphasise that deterministic testing remains essential. APIs still need to be validated, schemas still need to be tested and contracts still need to be enforced. What changes is everything above those foundations.
“The goal of testing AI systems is not to certify that the system is correct.”
– Paper’s authors
Systems increasingly need to be evaluated for reasoning quality, retrieval quality, trajectory correctness, coordination between agents and business outcomes.
The paper argues that failures should be caught “as low in the pyramid as possible.” A failure identified early is cheaper to diagnose and easier to remediate. The same issue discovered at the business outcome level may already have reached production.
For banks building customer assistants, coding agents, compliance copilots and autonomous workflows, the framework provides a blueprint for thinking about testing beyond traditional end-to-end automation.
New systems, new failure modes
The paper identifies an entirely new category of failures that have little equivalent in conventional software engineering.
These include hallucinations, instruction drift, trajectory collapse, prompt injection, context loss and emergent behaviours that appear following model updates.
Some of these failures may be particularly concerning for financial services.
The paper warns of “hallucinated information delivered authoritatively”, “silent behavioural degradation at scale” and “regulatory and liability exposure.”
Again, these are not simply software defects. A customer-facing assistant that invents policy terms, a compliance tool that begins misinterpreting regulations or a workflow agent that quietly ignores instructions all have the potential to create operational and regulatory consequences.
“The cost of insufficient evaluation is measured in hallucination incidents, model drift discovered weeks after onset and adversarial failures that reach production,” the authors write.
Assurance engineering
The paper’s central argument is that the systems now entering production are not simply harder-to-test versions of traditional software.
“They are a different kind of infrastructure,” the authors write, “for which the fundamental assumptions of software quality assurance do not hold.”
The implications for testing teams are profound. “Prompts must be version-controlled and regression-tested like code. Every model migration must trigger a full evaluation run across every prompt in the system.”
Most importantly, the authors argue that confidence itself needs to be redefined. In deterministic software, confidence comes from proving that software behaves as specified.
In probabilistic systems, confidence becomes “a property of the evidence accumulated through continuous evaluation.”
For banks, that may ultimately be the most important lesson of all.
As software becomes increasingly autonomous and dynamic, the future of quality engineering may depend less on proving that systems work and more on continuously demonstrating that they remain reliable, controlled and within acceptable boundaries.
Or, as the paper concluded: “Software QA was built for a world where software does what it is told to do. AI assurance must be built for a world where systems do what they have learned to do.”
16 SEPTEMBER IN LONDON
REGISTER TODAY – SIMPLY CLICK HERE
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
QA FINANCIAL PODCASTS
CLICK HERE TO LISTEN TO OUR EXCLUSIVE CONVERSATIONS
REGULATION & COMPLIANCE
Looking for more news on regulations and compliance requirements driving developments in software quality engineering at financial firms? Visit our dedicated Regulation & Compliance page here.
READ MORE
- Inside Rabobank: Engineering resilience by design
- Can AI agents finally automate data testing?
- Continuous testing drives DORA compliance
- Why software testing may face a major rethink
- Buy or build? AI rewrites software testing for banks
WATCH NOW
