DEEP REINFORCEMENT LEARNING FOR TEST CASE GENERATION AND PRIORITIZATION

Abstract

With the increasing scale and complexity of software systems, improving testing efficiency and fault detection capability under limited testing resources has become a critical issue in software testing research. To address the limitations of traditional approaches, which are often static, separately designed, and unable to make full use of dynamic execution feedback in test case generation and prioritization, this paper proposes a unified testing optimization framework based on deep reinforcement learning. The proposed method formulates both test case generation and test case prioritization as a sequential decision-making problem aimed at maximizing testing utility, and develops a feedback-driven strategy learning mechanism through state representation, action selection, and reward design, enabling the model to dynamically adjust testing behavior according to coverage information, fault detection outcomes, and budget constraints. Experimental results show that the proposed method effectively improves statement coverage, branch coverage, and the number of detected faults in test case generation, while also achieving higher APFD and earlier fault detection under limited budgets in test case prioritization. Further ablation analysis demonstrates that coverage reward, fault-related reward, and budget constraint play complementary roles in optimizing the testing strategy. Overall, this study shows that deep reinforcement learning provides an adaptive and unified solution for automated decision-making in software testing and offers a promising direction for the development of intelligent software testing methods.