As the annual QA Financial Forum returned to Chicago this week, where an invited audience of DevOps, testing and quality engineering leaders from financial firms heard presentations from expert speakers who covered a range of timely key topics and trends.
One of the main speakers on April 9 was Sandesh Gawande, founder and CTO of Connecticut-based Torana, the company behind data integrity management software and solutions provider iceDQ.
A serial entrepreneur, Gawande has been designing products and involved in data engineering since 1996, along the way developing and trademarking a framework for data integration, ETL Interface Architecture®.
During this decades-long journey, Gawande came to realise that, while companies were investing millions of dollars in their data projects, they were not testing their data pipelines. This causes project delays, huge labour costs, and expensive production fixes.
With this in mind, at the QA Financial Forum he reflected on a case that shook the industry nearly six years ago when, in 2018, one of Britain’s largest retail banks, TSB, botched a major customer data migration project.
Millions of customers were affected as TSB’s website went down for more than three weeks. The chaos led to a UK parliamentary enquiry and a regulatory fine of close to £250 million for TSB. The bank’s CEO at the time was sacked and personally fined £81,620.
Following his presentation in Chicago earlier this week, QA Financial checked in with Gawande to discuss this fascinating case and scrutinise what banks can, and should, do to avoid falling into the same traps as TSB.
Q: The incredible TSB scandal in 2018 was a clear case of how not to do things. The affair sent shockwaves through the industry. Clearly, data testing failed, but what other factors played a role? In other words, how could this have happened?
Gawande: The TSB bank IT migration failure in 2018 wasn’t solely caused by data testing issues, but rather a “perfect storm” of several contributing factors. While inadequate data testing undoubtedly played a significant role, other crucial aspects also contributed such as unrealistic deadlines, potentially leading to insufficient testing and a rushed implementation. Also, there were communication breakdowns: issues identified by internal controls weren’t adequately communicated between different parties involved.
“While data testing failures were undeniably a key factor, additional elements combined created a complex and ultimately disastrous situation for TSB.”
– Sandesh Gawande
Q: And what about project complexity and scope?
Gawande: Yes, the sheer volume and intricate nature of the functionalities being migrated added to the overall challenge. And there was limited internal expertise: TSB relied heavily on the expertise of SABIS, the external vendor responsible for implementing the new platform. Finally, do not forget shifting responsibilities: Changes in responsibility for testing between TSB and SABIS potentially led to confusion and gaps in the testing process.
While data testing failures were undeniably a key factor, these additional elements combined to create a complex and ultimately disastrous situation for TSB and its customers.
Q: Have there been other, similar incidents since?
Gawande: Yes, Deutsche Bank’s recent completion of its Postbank IT migration, in February of this year, took about nine years. The process, which involved transferring millions of customer contracts and data points to Deutsche Bank’s systems, experienced several issues throughout various phases. These included problems with online transfers, phone banking activation, and even statement printing.
And in November of last year, FINRA fined Merrill Lynch $6 million for suspicious activity reports between 2009 to 2019. Merrill Lynch incorrectly applied the monetary threshold applicable to national banks, rather than the threshold applicable to broker-dealers.
Q: How can and should other financial institutions learn from this to avoid such a costly mishap?
Gawande: Good question. Firstly, prioritise data testing, so treat data testing as equally important as application testing: Allocate sufficient resources and expertise to ensure data integrity and accuracy throughout the migration process.
Moreover, effective third party outsourcing management: While projects can be outsourced proper audits and internal expertise as well as oversight cannot be ignored.
Also, independent testing and certification. The team that is doing the development cannot be given sole responsibilities to test and certify the code. The QA activities should have been given to another party or kept inhouse.
“The only way to know the true status of a project is to test it. For that you need to establish quality controls.”
– Sandesh Gawande
Then there is complexity and size. Data migration is a complex project with many functionalities. it would have been wise to do incremental delivery instead of big bang approach.
In addition, risk management, meaning establish clear roles and responsibilities based on RACI matrix with effective risk management strategies.
Finally, communication: Implement effective communication between large teams with proper project management.
Q: You touch on a range of factors and things to consider. But how are these elements linked, so let’s say, how are migration planning, testing, quality controls and contingency plans inter-connected?
Gawande: The only way to know the true status of a project is to test it. For that you need to establish quality controls. Once the test gives the metrics pitfalls are found, then contingency planning can be done. And all this requires planning in advance as one cannot rely on ad hoc project management hoping things will fall in place.
Q: Thank you. Finally, anything else you’d like to share with our readers ahead of today’s event in Chicago?
Gawande: Yes, software development life cycle (SDLC) consists of planning, requirements, design, development, testing, deployments and maintenance. For application centric projects this is followed in some form or another. However, for data centric projects SDLC steps, especially testing is often ignored. I see a huge risk in data projects because of lack of expertise or ignorance in data testing.
QA Financial Forum 2024
Gawande spoke at our annual forum in Chicago earlier this week. At the conference venue – the Virgin Hotels Chicago, centrally located at 203 North Wabash Avenue – an invited audience of DevOps, testing and quality engineering leaders from financial firms heard presentations from expert speakers.
Delegate places were free for employees of banks, insurance companies, capital market firms and trading venues. More information can be found here.
Want to stay up to date and receive our news, features and interviews for free?
Our e-newsletter lands in your inbox every Friday. Sign up HERE in one simple step.
ALSO READ
