Friday, May 3, 2024, 11am

Software testing is an integral part of software development. However, testing faces challenges due to the time-consuming and challenging nature of writing high quality tests, leading to poorly maintained test suites and lower overall software quality. Prior work for automatically generating tests, like EvoSuite and Randoop can generate high-coverage tests, however often these tests are hard to read, unrealistic, or incorrect, necessitating additional effort from developers for verification. In contrast, language models have shown promise in generating human-like, high quality code functions, benefiting tools like Copilot in code generation.

However, language models are not as successful at generating tests, struggling with both hallucination and correctly invoking internal methods present in the code under test. This is because code generation language models only consider source code immediately before the generated code, and thus miss context in the file under test. To help overcome these limitations, I focus on how we can incorporate domain specific properties of testing such as the strong coupling between source and test files along with important test execution data to improve the application of language models to software testing. I also examine how we can better evaluate test generation approaches with metrics that are more meaningful to developers. My thesis statement is: We can exploit the structure of test code and close relationship between code and test files to enable the practical application of language models to software testing in both pretraining and fine-tuning. This approach can be used to a) generate useful unit test cases b) identify weaknesses in existing test suites and c) improve test suites to overcome found weaknesses.

My thesis will make the following contributions:

1. It presents a new method for pretraining models for test generation, that considers the relationship between source code and test code.
2. It provides an approach to automatically classify mutants as detected or undetected without executing the test suite by leveraging additional test context.
3. It demonstrates the effectiveness of adding execution context to test generation models, which enables us to generate mutant killing tests.
4. It evaluates all provided techniques with metrics and experiments that are practically meaningful developers, not considered in prior work.

Work I have already completed (ASE 2023) demonstrated that pretraining language models on dual objectives of code and test generation significantly improves unit test generation. I also leveraged the joint relationship between code and tests (FSE 2023) to improve predictive mutation testing techniques, modeling mutants at the token level, and incorporating both source and test methods during fine-tuning.

I propose to further apply these insights to a specialized case of mutation testing: generating tests that kill existing live mutants. I plan to include additional execution context into test generation models and use reinforcement learning. This will enable me to automatically generate test suites that are more comprehensive and similar to what an actual developer would write than current tools. I intend to complete this work by May 2025.

Thesis Committee:
Christian Kästner (Chair)
Daniel Fried
Claire Le Goues
Alex Groce (Northern Arizona University)

Additional Information

In Person and Zoom Participation. See announcement.

Event Type: Thesis Proposals
Room Number: In Person and Virtual - ET
Building: TCS Hall 358 and Zoom
Speaker's Name: KUSH DAVE JAIN
Speaker Website: www.kushjain.com
Speaker's Professional Title: Ph.D. Student, Ph.D. Program in Softare Engineering, Software and Societal Systems Department, Carnegie Mellon University
Talk Title: Exploiting Test Structure to Enhance Language Models for Software Testing
For More Information: cherold@cs.cmu.edu | aroudebu@andrew.cmu.edu
Affiliations: Software and Societal Systems Department (S3D)
Organization(s): School of Computer Science