Semantic Analysis of ChatGPT’s Behavior in Contextual Code Correction for Python

Abstract

Software debugging remains one of the most time-consuming and cognitively demanding phases in the software development lifecycle. The paper considers the analytical capabilities of transformers transformer-based language models—specifically ChatGPT—in detecting and correcting semantic faults in multi-module Python projects. Ten synthetic Python programs (100–200 lines each), containing a total of 30 deliberately injected faults (distributed among order faults, variable leakage, and edge-case omissions), were submitted to ChatGPT under a standardized prompting scheme. Model responses were benchmarked against conventional static analysis tools (Pylint, MyPy) and a human expert baseline. Quantitatively, ChatGPT achieved an average error detection rate (EDR) of 76.7%, outperforming Pylint (23.3%) and MyPy (15%) across fault categories. In repair accuracy (RA), ChatGPT resolved 62–75% of the identified errors correctly, versus 10–25% for static tools. Statistical validation using a Chi-square test (χ² = 36.27, p < 0.001) and one-way ANOVA (F(3, 27) = 14.92, p < 0.001) confirms the significance of these differences. Qualitative clarity was also assessed using ordinal metrics and validated via Kruskal-Wallis H tests (H = 11.56, p < 0.01). These results suggest that ChatGPT possesses substantial semantic reasoning capabilities, particularly in contexts requiring non-local inference across modules. However, limitations persist in handling implicit dependencies and dynamic runtime conditions. The study concludes that such models can be meaningfully integrated into debugging pipelines as assistive agents, provided their outputs are cross-validated with expert oversight and static tools. Future research should explore hybrid frameworks that combine statistical inference with formal verification techniques.

Ähnliche Arbeiten