Financial systems regulators shun being overly prescriptive when formulating quality or coverage metrics for the multitude of financial institutions under their supervision.
As a result, financial firms are left to apply their own judgement and tools to assess the fulfillment of their quality obligations.
A 100%-scale tends to be a common sign of “complete” test coverage, or the degree of test automation, in assessing the quality of complex financial technology platforms.
But is it always clear what the “100%” implies and what requirements it actually satisfies? Maxim Nikiforov, senior project manager at Exactpro, is not so sure.
London-based Nikiforov, who joined the testing business that specialises in complex financial infrastructures nearly 12 years ago, argues that this approach is fundamentally at odds with what comprehensive quality assessment is, and thinks it does not lead to representative test coverage.
“The approach to quality assessment that includes ‘100%’ goals is often based on a traceability matrix featuring quality requirements based on user stories,” he explained.
The International Software Testing Qualifications Board (ISTQB) Glossary defines a user story as “a high-level user or business requirement … in the everyday or business language capturing what functionality a user needs and the reason behind this.”
Nikiforov thinks the downside of this approach is that it causes the resulting quality metrics to be driven by the high-level static descriptions of user needs that do not necessarily communicate the entire context or outline the true depth of the test coverage measure of a given requirement.
“For example,” he said, “if a user expects an order with a negative price to be rejected, then the user story will state that fact, and the resulting test will aim to confirm it.”
In contrast, “what software testing should do is challenge this requirement and try to prove the opposite, check whether there are, in fact, conditions where a negative price is accepted, that would mean detecting an issue, a bug.”
Covering the ‘price-result’ field combination in-depth would require checking as many value and parameter combinations as possible, Nikiforov continued.
“Test coverage quality depends solely on the professional qualities of the QA personnel involved.”
– Maxim Nikiforov
“This measure would ensure a truly comprehensive quality assessment, as opposed to that derived from directly following the initial requirement to the letter,” he wrote in a recent analysis in Focus, run by the industry group WFE.
Examining the nature of outages, we can conclude that a failure of a complex system is rarely traced back to an isolated error, Nikiforov added.
“It is often a combination of factors, such as a large amount of minor individual issues considered non-critical, the common context of which is being ignored, accumulating into a larger problem,” he explained.
“That’s why it is important for the test framework to encompass all possible relevant conditions and parameters that could in any way affect system behaviour, as well as their combinations,” Nikiforov stressed.
He thinks “it is the only way to account for the interconnectedness in a complex system.”
Data-first approach
Nikiforov’s approach implies putting system data first, operating on the transaction level, as opposed to the requirements level, and thinking beyond the perceived linearity between business requirements and the disparate sets of tests that are derived from them.
“Due to its universality and algorithmic nature, a ‘data-first’ approach helps mitigate common challenges of the ‘requirements-centred’ approach,” he said.
Nikiforov pointed out these include “an ambiguity” in requirements analysis, a difficulty keeping the test library up to date and requirements consistent with each other and test coverage quality depending solely on the professional qualities of the QA personnel involved.
“Admittedly, a data-driven practice is impossible to implement in the absence of a data-first mindset or a unified data repository and processing workflow,” he wrote.
“It has no place in siloed infrastructures or among undigitised processes. But with a digitisation culture coupled with powerful and now affordable CPUs, the ability to aggregate and process massive volumes of transaction data can truly transform a test approach,” Nikiforov added.
He pointed out that “a traceability matrix does play an important role” as a reporting tool across the industry’s use cases.
However, “it should be taken with a grain of salt.”
Nikiforov said it should be merely a way to visualise more complex coverage, to map the results of thousands of tests to an easy-to-view-and-interpret list of items.
“A traceability matrix should be taken with a grain of salt.”
– Maxim Nikiforov
“Surely, a matrix representation can facilitate parametric analyses and reporting, help assess coverage gaps, but in no way can it be a primary cause of proper coverage,” he said.
“In this misunderstood ‘chicken-and-egg’ causality dilemma, the proper – comprehensive, and yet traceable – test coverage is not the product of a well-defined matrix, it is rather a result of a holistic data-driven testing practice, for which the coverage matrix is merely a convenient ‘user interface’,” Nikiforov argued.
He continued by writing that such an approach developed allows for pervasive automation, and it is also universal.
“It is independent of testers’ skills and the quality of pre-defined specifications,” Nikiforov noted. “It is rooted in the quality of the data made available for testing.”
Last, but not least, he thinks the results of complex testing can be successfully mapped back to the initial requirements.
In summary, Nikiforov said the purpose of software testing is to provide accurate, relevant, interpretable and affordable information, “and not to merely confirm that the system works as expected.”
Thus, in building complex systems, it is crucial to invest in a comprehensive test coverage and evaluation framework able to fully reflect the complexity of the system being tested, he noted. “It ensures holistic data-driven quality assessment and gives appropriate attention to each item on the compliance list, while not being limited by that set of proposed testable items,” Nikiforov concluded.
UPCOMING EVENTS
DO NOT MISS
QA FINANCIAL FORUM LONDON: RECAP
In September, QA Financial held the London conference of the QA Financial Forum, a global series of conference and networking meetings for software risk managers.
The agenda was designed to meet the needs of software testers working for banks and other financial firms working in regulated, complex markets.
Please check our special post-conference flipbook by clicking here.
READ MORE
- Taking place TODAY: the QA Financial Forum Chicago 2025
- HDFC Bank turns to Katalon and QualityKiosk for QA upgrade
- Perforce’s Clinton Sprauve on AI testing for charts and graphics
- Functional and regression testing crucial for banks, says insider
- Shastic teams up with MeridianLink to target banks with AI automation
Become a QA Financial subscriber – for FREE
* Receive our weekly newsletter * Priority invitations to our Forum events
