Internet giant Google published its annual DevOps Research and Assessment (DORA) at the end of last month, which has revealed some interesting findings that may leave some within the QA space and wider software development community to scratch their heads.
While artificial intelligence in software testing is being rolled out at an unprecedented rate, with banks, finance firms and other companies rushing to embrace the technology, the DORA report found that it also appears to be slowing the rate at which software is being developed, delivered and implemented.
In fact, there has been a 1.5% drop in software delivery, compared to last year, and a record 7.2% decrease in delivery stability.
Google’s annual DORA findings are eagerly watched within the QA space and the wider software development community as they track a range of DevOps metrics, including change lead times, deployment frequency, change failure rates and failed deployment recovery time.
Time for QA Financial to catch up with the lead author of the report, Nathen Harvey, DORA Lead and Developer Advocate at Google Cloud.
Your latest DORA report revealed some interesting findings. What would you say are the most direct implications of DORA for banks and other financial services firms?
There are several standout findings for banks and other financial services firms in this year’s DORA report. The highest performing teams excel across all four software delivery metrics, such as change lead time, deployment frequency, change fail percentage, and failed deployment recovery time, while the lowest performers perform poorly across all four. We see teams from every industry vertical in each of the performance clusters.
So what does this mean for banks and other financial services firms?
Top performance is achievable, even in highly regulated environments. This isn’t to suggest that there are no unique challenges across industries, but no one industry appears to be uniquely encumbered or uniquely capable when it comes to software delivery performance. Teams that adopt a mindset and practice of continuous improvement are likely to see the most benefits. Invest in building the organizational muscles required to repeat this over time.
Secondly, AI is having broad impact. AI is producing a paradigm shift in the field of software development. Early adoption is showing some promising results, tempered by caution. AI adoption benefits flow, productivity, job satisfaction, code quality, internal documentation, review processes, team performance and organisational performance.
“AI adoption also brings some detrimental effects, such as reductions to software delivery performance.”
– Nathen Harvey, Google
However, AI adoption also brings some detrimental effects. We have observed reductions to software delivery performance, and the effect on product performance is uncertain. The vast majority of respondent reported that their organisations have shifted their priorities to increase their incorporation of AI into their applications and services.
Participants from all surveyed industries reported statistically identical levels of reliance on AI in their daily work, which suggests that this rapid adoption of AI is unfolding uniformly across all industry sectors. So now is the time to start leveraging AI, if you’ve not already started. Teams should continue experimenting and learning more about the impact of increasing reliance on AI.
How would you describe the somewhat love-hate relationship between QA engineering and AI?
We have not studied this in our research program so we really don’t have data on this particular question. However, we have studied how much professionals are trusting the code that is created using generative AI. The DORA team found that 39% of developers trust the quality of gen AI output only “a little” or “not at all.” As with any tool, we see that those who trust the tools tend to have greater reliance on the tool than those that don’t. Trust and usage often create a virtuous cycle, each reinforcing the other.
The non-deterministic nature and rapid pace of innovation may also be influencing the attitudes that QA professionals have towards generative AI. These new tools require some new approaches to how teams complete their daily work and deliver value.
What DevOps trends do you foresee in the near future within the finserv space?
I see further adoption of AI as a big trend in the finserv space. We are in the early days of understanding the impact these tools can have on a team’s ability to deliver value. A lot of experimenting and learning is happening now and is likely to happen over the coming years. Experimentation always comes with both progress and setbacks; both can serve as success stories so long as teams apply the lessons learned to future work.
“The non-deterministic nature and rapid pace of innovation may also be influencing the attitudes that QA professionals have towards generative AI.”
– Nathen Harvey, Google
I also see the continuation of modernizing existing applications to take advantage of flexible infrastructure like what the cloud has to offer. Flexible infrastructure can increase organizational performance.
However, moving to the cloud without adopting the flexibility that cloud has to offer may be more harmful than remaining in the data center. Transforming approaches, processes, and technologies is required for a successful migration.
QA is increasingly climbing banks and other finance firms’ priority ladder, partly because of AI and automation, partly because of CrowdStrike-like threats and risks. Do you recognise this?
We haven’t investigated this particular phenomenon in our research but we have always seen organisations that create a climate for learning, enable fast flow, and enable fast, high-quality feedback are the organizations that tend to have the best performance.
Fair enough. Then, finally, you stated in your report that it appears AI is slowing the rate at which software is being developed, delivered and implemented. Can you elaborate?
Specifically, we see a drop in software delivery throughput and software delivery stability. That is to say that changes appear to be moving through the delivery system at a slower pace and are causing more failures that require immediate attention when those changes are introduced in production environments. This may be a side effect of generative AI’s ability to generate code faster than an engineer might do so. This, in turn, may lead to trying to move larger changes through the delivery system. We’ve seen that delivering in small batches is a way to improve overall software delivery performance. Perhaps this leaves the door open for reliance on AI beyond the code generation phase of the software development life cycle.
THIS WEEK

QA FINANCIAL FORUM LONDON: RECAP
Last month, on September 11, QA Financial held the London conference of the QA Financial Forum, a global series of conference and networking meetings for software risk managers.
The agenda was designed to meet the needs of software testers working for banks and other financial firms working in regulated, complex markets.
Please check our special post-conference flipbook by clicking here.
READ MORE
- Cognizant drags rival Infosys to court over trade secrets
- Testaify claims tool is ‘100x faster than seasoned QA architect’
- Fast-growing Newgen sets sights on banks in Middle East
- ABN Amro hires nCino and CBA for digital upgrade
- QAFF London: Lloyds’ Richard Bishop on the rise of ‘green software’
Become a QA Financial subscriber – for FREE
* Receive our weekly newsletter * Priority invitations to our Forum events