Assessing ChatGPT’s Code Generation Capabilities with Short vs Long Context Programming Problems

Abstract

This study assesses the code generation capabilities of ChatGPT using competitive programming problems from platforms such as LeetCode, HackerRank, and UVa Online Judge. In a novel approach, we contrast ChatGPT’s performance on concise problems from LeetCode against more complex, narrative-driven problems from Codeforces. Our results reveal significant challenges in addressing the intricate narrative structures of Codeforces, with difficulties in problem recognition and strategic planning in extended contexts. While initial code accuracy for LeetCode problems stands at 72%, it drops to 31% for complex Codeforces problems using Python. Additionally, we explore the impact of targeted instructions aimed at enhancing performance, which increased LeetCode accuracy to 73.53% but saw a decrease in Codeforces performance to 29%. Our analysis further extends across multiple programming languages, examining if iterative prompting and specific feedback can enhance code precision and efficiency. We also delve into ChatGPT’s performance on challenging problems and those released post its training period. This research provides insights into the strengths and weaknesses of AI in code generation and lays groundwork for future developments in AI-driven coding tools.

Ähnliche Arbeiten