Ask a banking QA lead what success looked like five years ago and the answer would probably have been straightforward: high automation coverage, fewer production defects and faster releases.
Those metrics still matter. But they are no longer enough. Increasingly, quality assurance teams are being asked to produce something different: evidence.
Not simply evidence that software was tested, but evidence showing what was tested, why particular scenarios were selected, which controls were validated, what defects were discovered, how they were remediated and whether those controls remain effective as systems evolve.
There is no formal industry term for this shift yet, but “evidence engineering” increasingly describes what many banking QA teams are being asked to do.
Software testing is no longer just about finding bugs. It is becoming the mechanism through which banks demonstrate operational resilience, AI governance and technology risk management to regulators, auditors and boards.
Regulators’ moves
That shift is quietly appearing across almost every major regulatory initiative affecting financial services.
Under the EU’s Digital Operational Resilience Act (DORA), firms are expected to carry out resilience testing, document the results, remediate weaknesses and maintain governance over ICT risk. The emphasis is not simply on performing testing, but on being able to demonstrate that resilience has been assessed and that weaknesses have been addressed.
The UK’s Financial Conduct Authority is taking a similar approach through its AI Live Testing initiative. Rather than encouraging firms to keep artificial intelligence inside controlled proof-of-concept environments, the FCA is inviting them to test AI systems in supervised real-world conditions.
As Ed Towers, head of advanced analytics at the FCA, explained recently: “We’re providing a structured but flexible space where firms can test AI-driven services in real-world conditions, all with our regulatory support and oversight.”
Just as importantly, the FCA has expanded what it considers part of the system being tested.
“We broadly define the AI system as: the actual AI model, information on the deployment context and core risks … governance and human in the loop considerations, evaluation techniques as well as the input and output controls.”
That definition reaches well beyond software functionality. Governance, deployment context, human oversight and operational controls all become part of the assurance process.
The Bank of England is moving in the same direction. Rather than relying solely on governance policies, it has begun using simulations to understand how AI systems might behave under periods of market stress, including whether autonomous trading agents could display correlated or “herding” behaviour.
The Bank has also made its broader expectations clear through its cyber resilience work, arguing that resilience “can no longer be assumed, it must be proven.” That distinction is subtle, but profound. Traditional testing asks whether software functions correctly.
Evidence-based assurance asks whether an organisation can reconstruct, months later, exactly how resilience was tested, what scenarios were exercised, what failures occurred, how they were resolved and whether the controls remain effective today.
ECB and MAS policy
The European Central Bank is reinforcing that same philosophy. Its recent warnings around artificial intelligence and cyber resilience have consistently pointed towards continuous operational resilience rather than periodic assurance, reflecting the reality that threats evolve faster than traditional testing cycles.
Singapore’s Monetary Authority has likewise focused on operationalising AI governance through lifecycle controls, monitoring and validation.
As Sameer Gupta of DBS recently observed: “To fully realise AI’s value, governance must be treated as a strategic imperative,” while Sam Burrett, chief executive of Indago, highlighted the execution challenge facing many organisations.
“Most organisations we work with have an AI policy. Very few have actually operationalised AI governance,” he said.
That operationalisation increasingly falls to testing teams. The UK’s new Critical Third Parties regime extends the same thinking beyond banks themselves.
By bringing Amazon Web Services, Google Cloud, Microsoft and Oracle under direct resilience oversight, regulators are recognising that systemic resilience depends not only on financial institutions, but also on the technology providers that underpin them.
Yet the regime does not remove responsibility from banks. Instead, it creates two connected layers of assurance. Cloud providers must demonstrate the resilience of their critical services.
Banks must still demonstrate that their own important business services remain resilient when those services degrade or fail. For QA teams, that changes the nature of testing because the objective is no longer simply to confirm that an application works.
Increasingly, testing has to demonstrate that critical business services remain within impact tolerances, that failover mechanisms operate correctly, that recovery processes have been validated, that third-party dependencies have been exercised and that those outcomes can be evidenced if challenged by regulators or auditors.
Change in mindset
That represents a significant change in mindset. Historically, a successful test execution was often the end of the process. Increasingly, it is the beginning.
Every test execution now contributes to a body of evidence that supports operational resilience, technology governance and regulatory compliance.
The question regulators are asking is quietly changing. For years, software testing largely answered one question. Did you test it? Increasingly, regulators are asking something much more demanding. Can you prove you tested it? Can you demonstrate why those scenarios were selected?
Other questions that become more relevant are can you show what failed? Can you explain how defects were remediated? Can you demonstrate that the same controls remain effective today?
Those questions place traceability at the centre of modern QA. Requirements, controls, test cases, execution records, defect histories, approvals and remediation activities all become part of a connected evidence chain.
That evolution is also reshaping testing technology. Many vendors are investing heavily in richer audit trails, execution history, AI-generated documentation, visual evidence, automated reporting and traceability across the software lifecycle.
Those capabilities are often presented as productivity improvements. Increasingly, they also help organisations answer regulatory questions with credible evidence. Artificial intelligence makes the challenge even more significant.
AI systems may evolve as models, prompts, data and deployment environments change. Passing a test once provides only limited assurance. Instead, organisations need continuous validation supported by continuous evidence.
That pushes QA further towards monitoring, governance and lifecycle assurance rather than discrete test execution. The implications extend well beyond engineering teams. Boards increasingly want assurance that operational resilience objectives are being met. Risk functions want evidence that controls remain effective and internal audit wants traceability.
Supervisors increasingly expect firms to demonstrate not only that resilience exists, but how it has been validated so quality assurance now sits at the centre of those conversations.
16 SEPTEMBER IN LONDON
REGISTER TODAY – SIMPLY CLICK HERE
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
REGULATION & COMPLIANCE
Looking for more news on regulations and compliance requirements driving developments in software quality engineering at financial firms? Visit our dedicated Regulation & Compliance page here.
READ MORE
- AI adoption strains JPMorgan testing
- Can banks ‘outsource’ AI accountability?
- HDFC Bank raises testing stakes
- Is observability banking QA’s next discipline?
- Barclays on AI testing, telemetry and kill switches
WATCH NOW
QA FINANCIAL PODCASTS
CLICK HERE TO LISTEN TO OUR EXCLUSIVE CONVERSATIONS
