Influence of External Dependency Retrieval and Prompt Engineering in Test Case Generation using LLMs

Abstract

The recent rise of large language models (LLMs) has enabled the generation of higher-quality test cases by leveraging the semantics of the methods under test. However, existing LLM-based approaches still struggle to achieve high coverage levels. To mitigate this issue, we present two complementary techniques in this work: Prompt Engineering and External Dependency Retrieval for context enrichment. We evaluated our improvements through an ablation study on three open-source and four proprietary projects, encompassing 261 distinct methods. For each method, we generated test suites under four implementations and performed ten independent runs, yielding a total of 10,440 executions. Our combined approach yields an average coverage increase of 12% on industrial software, with statistically significant gains over all other variants studied in this paper. Although our enhancements increase the context (the number of input tokens rises by 66.3%), this is partially compensated by a reduction in output tokens due to fewer repair attempts, so that the overall cost overhead remains moderate at about 16%. As future work, we aim to identify the minimal necessary context that still yields significant improvements in test coverage, which could help to further reduce costs.