Context Matters: Investigating Its Impact on ChatGPT's Bug Fixing Performance

Abstract

In this study, we explore the role of contextual information in enhancing ChatGPT's capabilities in bug fixing. Our focus is specifically on the “Wrong Answer” problem, where a program executes without error but fails to produce the correct output. Our approach draws inspiration from human debugging practices, which heavily rely on understanding both the intended task of the program and the specific scenarios in which it fails, such as unit test cases. We evaluate ChatGPT's performance with various types and levels of contextual data. The results reveal three key insights. First, providing the model with a mix of correct and incorrect test cases sharpens its debugging skills. Second, giving ChatGPT detailed descriptions of the problems substantially enhances its ability to identify and resolve errors. Third, merging detailed problem descriptions with various test cases leads to a synergistic outcome. This combined approach significantly elevates the efficiency of the bug-fixing process compared to employing each type of contextual information individually. Our paper presents a thorough analysis based on these findings. It offers an extensive exploration of why and how contextual information can be strategically utilized to enhance ChatGPT's debugging effectiveness. Furthermore, this investigation enriches our comprehension of the underlying mechanisms by which contextual cues amplify the model's capacity for solving problems.

Ähnliche Arbeiten