As financial services firms accelerate their use of artificial intelligence, the recent case of Chinese AI startup DeepSeek is sending a stark reminder to the industry: inadequate testing can carry dangerous consequences.
DeepSeek’s open-source model, designed to rival the reasoning capabilities of top-tier systems like OpenAI’s GPT, has rapidly gained attention for its technical promise.
However, behind the innovation lies a series of glaring security lapses, with particular relevance for software testing, according to some industry experts. They are vulnerabilities that highlight the systemic risk of deploying under-tested AI in the complex software ecosystems of high-stakes sectors like banking and finance.
In fact, recent research by Cisco and the University of Pennsylvania showed that DeepSeek’s model failed to block a single one of 50 harmful prompts from the HarmBench dataset, producing a 100% attack success rate.
At the same time, data from Wiz Research discovered a publicly exposed ClickHouse database containing more than a million lines of logs, including sensitive chat histories, secret keys, and backend infrastructure information.
For financial services firms, where data privacy, digital and operational resilience are core to both compliance and reputation, such exposure could have a devastating effect, according to Nabeel Jaitapker, product marketing lead at HCLSoftware.
The threats do not stop there. DeepSeek’s mobile apps were also found to contain multiple critical vulnerabilities. These included weak encryption, SQL injection points, hardcoded keys, and a lack of proper privacy safeguards, issues that could allow attackers to decrypt data or manipulate app behavior.
Most concerning, perhaps, is that DeepSeek’s model reportedly has been used to generate functioning malware, including ransomware, with minimal user expertise.
All of this reinforces a growing truth in financial IT: AI adoption without comprehensive, continuous security testing presents a clear and present risk, Jaitapker warned.
To address these risks, financial institutions are turning to various tools, which integrate application security testing throughout the software development lifecycle.
Discussing HCL’s own solution, AppScan, “its capabilities allow for the detection of vulnerabilities in source code early in the development process,” explained Jaitapker. “This approach helps developers address security flaws before they become exploitable.”
Dynamic testing
However, identifying issues in the codebase alone is no longer sufficient as Jaitapker emphasised the importance of dynamic testing in today’s fluid, runtime-driven environments.
“By simulating attacks on running applications, tools can identify vulnerabilities that may not be apparent through static analysis alone. This ensures a more comprehensive security assessment,” Jaitapker wrote in a recent analysis.
Combining these methods yields greater insight, which is where hybrid tools come into play, he stressed.
“Then there is interactive application security testing, combining elements of both SAST and DAST, providing real-time vulnerability detection during application runtime,” said Jaitapker.
“A hybrid approach enhances the accuracy and depth of security testing.”
– Nabeel Jaitapker
As many AI systems incorporate open-source components, software composition analysis is now a vital part of any firm’s testing toolkit, he continued.
For financial services firms that are increasingly blending internal systems with AI-driven tools and APIs, as most do, the DeepSeek episode is more than a cautionary tale. Jaitapker said it should be seen as “a call to action.”
“By integrating such testing into their development workflows, organisations can enhance their security posture, protect sensitive data, and maintain user trust,” he noted.
“Regular and thorough security assessments are crucial in today’s rapidly evolving technological landscape to prevent potential breaches and safeguard against emerging threats.”
In summary, according to Jaitapker, the lesson is clear: innovation in financial services must be matched by equally advanced, rigorous testing. “Anything less is a risk most financial institutions can’t afford to take.”
Why not become a QA Financial subscriber?
It’s entirely FREE
* Receive our weekly newsletter every Wednesday * Get priority invitations to our Forum events *
REGULATION & COMPLIANCE
Looking for more news on regulations and compliance requirements driving developments in software quality engineering at financial firms? Visit our dedicated Regulation & Compliance page here.
WATCH NOW
READ MORE
- Antithesis swells finserv footprint as autonomous testing gains traction
- Deep Dive: will AI replace QA teams, or simply make them more valuable?
- BrowserStack snaps up Requestly to expand open-source testing
- Eximius leads pre-seed round in rising QA star DevAssure
- AI testing takes centre stage in Bank of England’s data transformation
