QA space warned for ‘dark side of AI we are not taking about’

Fitz Nowlan

As software development undergoes a seismic shift with GenAI at the forefront, testing, quality assurance, and observability are being transformed in unprecedented ways.

These advancements are driving new levels of automation and efficiencies, while challenging traditional methodologies and long-held assumptions about speed, adaptability, and innovation.

As GenAI automates routine tasks and enables smarter decision-making, it is raising critical questions about oversight, reliability, and responsibility.

In this era of rapid transformation, the industry must balance GenAI’s immense potential with its inherent risks to ensure a future of sustainable progress, argues Fitz Nowlan, who is VP of AI and architecture at Boston-based software testing firm SmartBear.

“While GenAI is celebrated for its transformative potential, its adoption comes with critical pitfalls and risks that often go unaddressed,” said Nowlan, who is also a co-founder of Reflect, an advanced AI-powered provider that was acquired by SmartBear in 2024.

That deal was generally seen as an acceleration of SmartBear’s AI strategy and expansion of its GenAI capabilities.

Nowlan continued: “One major concern is the illusion of simplicity that GenAI creates.”

“By abstracting away the underlying complexity of systems, GenAI can obscure vulnerabilities that may only appear in edge cases. This false sense of security can lead teams to underestimate the challenges of debugging and maintenance,” he explained.

In addition, Villanova, Pennsylvania-based Nowlan singled out the risk of over-reliance on automation as another major concern.

“Teams that depend too heavily on AI-driven tools may overlook the rigor and low-level details essential for QA, leaving gaps that compromise reliability,” he warned.

Nowlan thinks this problem is compounded by issues of data bias and model transparency.

“AI systems are only as reliable as the data they are trained on, and biases in training data can lead to flawed outputs that undermine the quality and fairness of applications.”


“One major concern is the illusion of simplicity that GenAI creates.”

– Fitz Nowlan

Ethical and privacy concerns further complicate GenAI’s adoption, he continued

“Sensitive data used to train AI tools can increase the risk and cost of a future breach, as well as create compliance challenges when third-party models are involved,” Nowlan pointed out.

Finally, the rapid pace of AI adoption often results in escalating technical debt.

“Systems built on GenAI may be efficient in the short term but fragile over time, leading to hidden costs and long-term maintenance challenges that are difficult to resolve,” he said.

Nowlan’s observations come as GenAI is transforming how software development teams think about QA and observability.

Traditionally seen as separate domains, QA and observability now converge under the capabilities of GenAI, setting new standards for speed, adaptability, and precision.

“This integration demands a shift in how we approach and align these disciplines,” Nowlan noted.

Furthermore, he believes the growth of GenAI throughout the software development lifecycle potentially establishes a new connection between authoring and testing software.

Nowlan’s comments come only a month after SmartBear became the new owner of QMetry, a provider of an AI-enabled digital quality platform that aims to scale software quality, the software and API testing firm confirmed.

Dan Faulkner, chief product officer at SmartBear
Dan Faulkner

Financial details were not disclosed by Dan Faulkner, SmartBear’s chief product officer, nor by the sellers, QMetry’s private equity owners, Goldman Sachs Alternatives and Everstone Group.

However, Faulkner did say that “QMetry’s solutions align perfectly with our objectives [because] by adding their test management and AI-enabled tools to our portfolio, we’re expanding our technical and market footprint.”

Built on “compliance-driven” test management, as SmartBear put it, QMetry’s GenAI-enabled platform aims to scale testing efforts, reduce manual tasks, and accelerate release cycles. The company claims to serve several Fortune 500 companies.

Intent-driven quality

Analysing the current state of the testing space, traditional test automation has long relied on rigid, code-based frameworks, Nowlan said, which require extensive scripting to specify exactly how tests should run.

“GenAI upends this paradigm by enabling intent-driven testing. Instead of focusing on rigid, script-heavy frameworks, testers can define high-level intents, like ‘Verify user authentication,’ and let the AI dynamically generate and execute corresponding tests,” he explained.

This approach reduces the maintenance overhead of traditional frameworks, Nowlan continued, while aligning testing efforts more closely with business goals and ensuring broader, more comprehensive test coverage.

“At the same time, human testers remain indispensable for setting priorities, conducting exploratory testing, and overseeing AI-generated outputs,” he analysed.

“This collaboration between human intuition and AI-driven efficiency establishes a new standard for quality, one that is faster, smarter, and more reliable. When implemented thoughtfully, this strategy has the potential to redefine the role of QA in modern development,’ Nowlan summarised.

Evolving observability

As QA workflows evolve with GenAI, observability tools are also seeing a transformation with AI.
Traditional observability tools focus exclusively on tracking logs, metrics, and traces to infer system health and diagnose issues.

“While effective for conventional systems, this approach falls short in environments dominated by AI,” stated Nowlan.

“GenAI introduces new layers of abstraction, models, datasets, and generated code, that traditional observability methods rarely integrate,” he pointed out.

“To address this gap,” Nowlan continued, “AI observability is emerging as a critical discipline to interpret model behaviors, trace root causes, and validate outputs at a deeper level.”

However, this evolution comes with its own set of challenges.

Nowlan argues that the inherent opacity of AI models can hinder debugging, while third-party AI reliance raises concerns about trust, accountability, and cost.

“Teams must incorporate ethical guardrails and maintain human oversight to ensure that observability evolves in a way that supports innovation without sacrificing reliability,” he said.


“Human testers remain indispensable for setting priorities, conducting exploratory testing, and overseeing AI-generated outputs.”

– Fitz Nowlan

Zooming in on the “symbiotic future” of QA and observability, Nowlan stated that QA and observability are no longer siloed functions.

“GenAI creates a semantic feedback loop between these domains, fostering a deeper integration like never before,” he said.

The Reflect co-founder added that robust observability ensures the quality of AI-driven tests, while intent-driven testing provides data and scenarios that enhance observability insights and predictive capabilities.

“Together, these disciplines form a unified approach to managing the growing complexity of modern software systems,” he summarised.

“By embracing this symbiosis, teams not only simplify workflows but raise the bar for software excellence, balancing the speed and adaptability of GenAI with the accountability and rigor needed to deliver trustworthy, high-performing applications,” Nowlan shared.

Finally, he was keen to stress that the risks associated with GenAI should not deter its adoption but serve as a reminder to approach it with thoughtful implementation.

“By combining automation with human oversight, adopting transparent practices, and embedding ethical governance into development workflows, the industry can prepare itself to meet the challenges of a GenAI-driven future,” Nowlan argued.

“As GenAI raises the bar for speed and adaptability, the real test will be maintaining the transparency, oversight, and accountability required to ensure sustainable progress,” he concluded.


NEXT MONTH


DO NOT MISS


QA FINANCIAL FORUM LONDON: RECAP

In September, QA Financial held the London conference of the QA Financial Forum, a global series of conference and networking meetings for software risk managers.

The agenda was designed to meet the needs of software testers working for banks and other financial firms working in regulated, complex markets.

Please check our special post-conference flipbook by clicking here.


READ MORE


Become a QA Financial subscriber – for FREE

* Receive our weekly newsletter * Priority invitations to our Forum events

REGISTER HERE TODAY