Overhyped? AI not creating ‘any clear wins’ in QA, says insider

Austin, Texas-based Mike Sonders

QA teams across the financial services space are racing to embrace AI, primarily to increase productivity, reduce costs and speed up automation.

The adoption of AI-powered solutions is high across software testing workflows but the relatively young technology is not paying off in ways that matter most to software teams. At least, not yet.

That is according to new research by software testing firm Rainforest QA.

The company found in a new report that teams open-source teams using AI are still spending just as much — if not more — time on “painful test writing and maintenance tasks” than ones not using AI, as Rainforest QA’s researchers put it.

“AI has a high adoption rate in open-source testing, but isn’t correlated with more productivity,” USA-based Mike Sonders of Rainforest QA wrote.

According to Rainforest’s survey of over 600 software developers in the U.S., Canada, the U.K., and Australia, close to three in four teams using open source frameworks for test automation are using AI to assist with test writing and maintenance.

“Test writing and maintenance, of course, are the most time-consuming and painful parts of the ongoing test automation process,” Sonders said.

He cited one software engineering lead, who shared in the survey that ”to be able to maintain automated tests, especially with a small dev team, it just takes time,” while an engineering manager at an unnamed firm said that “in the past, we gave up on testing the front end with open source because it was too difficult to maintain and the tests were very often broken.”

But Sonders continued by saying that open-source teams using AI aren’t saving time on these tedious tasks.

“In fact, it looks like teams using open-source frameworks who use AI for test creation or maintenance actually spend slightly more time on these tasks than teams who don’t use AI,” he wrote.

This might come as a surprise to anyone who’s bought into the promise of AI to improve productivity on software teams.

But it echoes the results of a recent study by Uplevel showing that developers using GitHub Copilot currently experience no clear productivity gains, and do have to deal with more bugs.

Source: Rainforest QA

One possibility could explain these results and undersell AI’s benefits: teams who choose to adopt AI may have complex products that inherently require more work than average to keep automated test suites updated.

“In which case, maybe AI is saving those teams some from spending even more time on test suite upkeep, and that effect is hiding in the results,” Sonders wrote.

“But given the large adoption rate of AI for test creation and maintenance among teams who use open-source automation frameworks, this scenario seems unlikely,” he added.

“It’s more likely that AI just isn’t currently delivering velocity benefits for these recurring test automation tasks,” Sonders explained.

However, AI is not a complete wash for teams using open source, Sonders stressed, as it does help small teams keep automated test suites up to date.

“While AI hasn’t created any clear wins for open-source teams in the time they spend on maintenance, AI might still be helpful for small teams working with open source,” he noted.

When an automated test suite is not kept up to date, it can’t reliably do its job: catching bugs and other issues before customers do.

“But test maintenance is painful and time-consuming — especially with open-source frameworks, as we’ve seen in the data — so some teams struggle to keep up with it,” Sonders shared.

He pointed out that test maintenance is a practice that requires clear definitions of ownership and enforced team policies.


“AI has a high adoption rate in open-source testing, but is not correlated with more productivity.”

– Mike Sonders

So, the smallest dev teams — the ones least likely to have formal policies and procedures in place — are understandably the least likely to keep their automated test suites updated.

“The time required to work with open-source frameworks just exacerbates the problem,” Sonders said.

But as teams grow, there is not much difference between using AI and not using AI, as the vast majority of teams report keeping their tests up to date.

So why is AI underdelivering in important areas of test writing and maintenance?

“The results don’t necessarily suggest AI is a dud for teams using open-source frameworks,” Sonders stated.

“Implementations of AI across open-source testing setups include various off-the-shelf and custom-made solutions,” he explained.

“Some solutions are certainly more effective than others, but the results suggest that teams are still trying to find the ones that work. And there are probably areas in which the technology still needs to improve.”

Source: Rainforest QA

The data showed that teams using no-code to automate their E2E tests are spending a lot less time on test maintenance tasks. They give themselves more time and resources to dedicate to shipping code.

“But not all no- and low-code tools are built the same — some are complex, require training, and have their own “language” to learn,” Sonders said.

“It’s going to be difficult to save time using any tool like that.” 

To increase velocity and free up dev team’s time, the key is to adopt a no-code testing tool that’s intuitive, he continued, requiring little or no training.

“It should produce test scripts in plain English, and deeply integrate AI in ways that demonstrably save time on the most annoying of maintenance tasks,” Sonders concluded.


NEXT MONTH


DO NOT MISS


QA FINANCIAL FORUM LONDON 2024: RECAP

In September, QA Financial held the London conference of the QA Financial Forum, a global series of conference and networking meetings for software risk managers.

The agenda was designed to meet the needs of software testers working for banks and other financial firms working in regulated, complex markets.

Please check our special post-conference flipbook by clicking here.


READ MORE


Why not become a QA Financial subscriber? It’s entirely FREE

* Receive our weekly newsletter * Priority invitations to our Forum events *

REGISTER HERE TODAY