As banks accelerate the deployment of AI across customer-facing products and internal workflows, the traditional boundaries of software testing are starting to blur.
For QA teams, the challenge is no longer limited to validating systems, models and code, but increasingly extends to how humans interact with them under real-world conditions.
That shift is exposing a critical gap. While financial institutions have invested heavily in tools, controls and training, the most unpredictable, and least tested, component in the stack remains human behaviour.
Nicole Jiang, co-founder and CEO of Fable Security, argued that this gap is becoming more pronounced as AI becomes embedded in everyday decision-making across banks.
QA Financial: Nicole, despite significant investment in security tools, human error remains a leading cause of breaches. How is this risk evolving as AI becomes embedded in banking systems and workflows?
Jiang pointed to the rapid rise of unsanctioned AI usage inside organisations, drawing a direct parallel with earlier struggles around shadow IT.
“Shadow AI is the new shadow IT, and banks haven’t finished solving shadow IT yet. Employees don’t wait for sanctioned tools; they find what works and they use it.”
She described how this behaviour is already translating into tangible security incidents. “That’s how Vercel wound up receiving an alleged $2 million ransom from a threat actor, after its systems were breached when an employee downloaded a compromised and unauthorized generative AI tool.”
For QA and risk teams, the issue is not the technology itself, but how it is used in practice.
“The risk isn’t that AI is malicious. It’s that an employee pasting customer data into an unsanctioned model, or over-trusting an AI-generated summary on a high-stakes transaction, creates exposure that no firewall catches.”
“Shadow AI is the new shadow IT, and banks haven’t finished solving shadow IT yet.”
– Nicole Jiang
As AI becomes more embedded, Jiang warned, human decision-making becomes both more frequent and less friction-bound.
“AI-embedded workflows make the human the variable in more decisions, more often, with less friction warning them to slow down.”
Q: From a QA and testing perspective, are financial institutions doing enough to simulate and test human behavior as part of their security and resilience strategies?
Jiang argued that most institutions are still measuring the wrong thing. “The honest answer is that most financial institutions aren’t testing human behavior at all, they’re testing human awareness.”
She drew a clear distinction between the two, noting that awareness testing often fails to reflect real-world conditions.
“Awareness tells you whether someone can identify a threat in a controlled, low-stakes quiz environment. Behavior tells you what they actually do under time pressure, with incomplete information, inside the real systems they use every day,” Jiang explained.
This gap has direct implications for operational resilience, particularly under adversarial conditions. “Human behavior can fail, and under adversarial conditions, it fails in patterned, predictable ways.”
According to Jiang, the data to address this already exists inside most banks. “Most institutions have extensive telemetry on user actions, they just haven’t applied it to people the way they’ve applied it to systems.”
She added that the issue is less about resources and more about mindset.
“That’s not a resource problem. It’s a framing problem… Until human behavior is treated as a monitored, fixable control layer with a defined baseline, the answer to ‘are we doing enough’ will always be: we don’t actually know.”
Q: You’ve argued that traditional security training isn’t working. What are the key shortcomings, and how should banks rethink testing and validation of human risk?
Jiang was blunt in her assessment of current training approaches. “Banks spend millions on security training and then measure its success by whether employees finished it. That’s like testing a fraud detection model on clean data and calling it production-ready.”
She highlighted that the highest-risk individuals are often not the focus of targeted, behaviour-driven testing.
“The population that needs the most attention… is rarely the population getting risk-calibrated, behaviorally focused training programs or assessments.”
“QA and security teams collaborating isn’t a nice-to-have. It’s the only way banks can see whether their human controls actually hold.”
– Nicole Jiang
Instead, Jiang outlined a more targeted, QA-style approach focused on real risk scenarios. “Identify the highest-consequence human decisions in your institution… Monitor the actual risky behaviors… Build adversarial scenarios around them; and… Measure response under conditions that actually resemble the threat.”
Her conclusion was stark. “Everything else is overhead.”
Q: As AI-powered attacks become more sophisticated, how should QA and security teams collaborate to test not just systems, but also how employees interact with them?
Jiang positioned collaboration between QA and security as essential, not optional. “With AI-powered cyberattacks coming in, QA and security teams collaborating isn’t a nice-to-have at this point. It’s the only way banks or any organization can see whether their human controls actually hold.”
She pointed to AI-driven attack techniques as a real-world stress test for both technical and human controls.
“A well-executed deepfake voice authorisation or AI-generated spearphishing lure doesn’t need to beat the firewall, it needs to beat the employee.”
This shifts the testing focus squarely onto human response under pressure. “The missing piece is treating human response as a measurable output, not an assumed one.”
Q: There’s growing talk of ‘measurable human risk’. What does that actually look like in practice, and how can financial institutions integrate it into their testing and governance frameworks?
Jiang argued that banks need to treat human behaviour in the same way they treat any other control layer.
“In practice, it means human behavior gets treated as a control, not a cultural initiative, not a training metric, not a footnote in the resilience report.”
That means defining expected states, testing against them, and feeding results directly into governance processes, she stressed.
“A control has an expected state, it gets tested against that state, and variance gets reported and remediated.”
For QA teams, this creates a familiar pattern, applied to a new domain. “The governance logic is identical to what they already know how to do.”
Q: Finally, as regulators place more emphasis on operational resilience, how important is it for banks to treat human behavior as a testable and monitorable component of their overall risk and QA strategy?
Jiang framed regulation as the catalyst that will force change across the industry. “Operational resilience regulation is the forcing function that makes all of it unavoidable.”
She warned that institutions with strong technical controls but weak human-layer visibility are fundamentally exposed.
“The institutions that have the most sophisticated technical controls and the least instrumented human layer are not actually resilient.”
Ultimately, she positioned this not as a compliance issue, but a core risk management challenge.
“That’s not a compliance problem. That’s a risk management problem… It’s time they applied the same logic to their human assets, too,” Jiang concluded.
THIS MONTH
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
REGULATION & COMPLIANCE
Looking for more news on regulations and compliance requirements driving developments in software quality engineering at financial firms? Visit our dedicated Regulation & Compliance page here.
READ MORE
- Inside banking’s shift to smarter QA to tackle complexity and risk
- SmartBear CPTO on AI in banking QA: ‘Impressive metrics but no critical scenarios’
- Banks push beyond traditional QA as resilience testing gains ground
- Banking QA professionals warn AI still doesn’t know ‘where the bodies are buried’
- RECAP: The QA Financial Healthcare & Insurance Forum Philadelphia 2026
WATCH NOW
QA FINANCIAL PODCASTS
CLICK HERE TO LISTEN TO OUR EXCLUSIVE CONVERSATIONS
