Estimated reading time: 2.5 minutes
Here we present an overview of a recently published research article “Software Testing with Large Language Model: Survey, Landscape, and Vision”.
The paper, authored by Junjie Wang et al. and published to the arXiv repository in July 2023, presents a comprehensive review of the current usage of Large Language Models (LLMs) in software testing and attempts to present a comparison of their performance. The authors analyse 52 relevant studies, focusing on the intersection of software testing and LLMs, a technology in natural language processing and artificial intelligence.
Generative AI, which utilises large language models (LLMs) to generate new and unique content based on patterns found in training data, has become increasingly popular in recent months, since the launch of Microsoft artificial intelligence research lab OpenAI’s ChatGPT tool in November 2022. This technology has rapidly become integrated into a number of software testing platforms including Webb.ai’s root cause analysis platform, New Relic’s observability platform and Honeycomb’s AI query assistant, as well as seeing the creation by Australia and New Zealand banking group (ANZ) of a team to formally investigate the use of generative AI for software testing.
As the scope and complexity of software systems continues to grow, there is an increasing need for effective software testing techniques. The authors explore the software testing tasks for which LLMs are commonly used, identifying test case preparation and program repair as the most representative ones. They also analyse the types of LLMs used, the methods of prompt engineering employed, and the accompanying techniques with these LLMs.
The paper provides a detailed discussion of the current use of LLMs in software testing. It summarises the methodologies and applications, highlighting key challenges and potential opportunities as identified in the analysed studies. These include:
The use of LLMs in the early stages of testing (e.g. test planning) has not yet been explored.
Some areas of testing (e.g. regression testing) have not yet been explored using LLMs.
The lack of application of LLMs to non-functional testing.
The authors also present a roadmap for future research in this area, pointing out gaps in the current understanding and potential avenues for exploration.
[Image Source: Institute of Software, Chinese Academy of Sciences]
Related Articles:
ANZ explores generative AI for code testing
