An empirical study of ChatGPT-related projects and their issues on GitHub

Abstract

Due to its powerful capabilities in natural language understanding and content generation, ChatGPT has received widespread attention since its launch in 2022. An increasing number of ChatGPT-related projects (that enhance the capabilities of ChatGPT, develop applications by calling ChatGPT APIs, etc.) are being released on GitHub and have sparked widespread discussions. However, GitHub does not provide a detailed classification of these projects to help users effectively explore interested projects. Additionally, the issues raised by users for these projects cover various aspects, e.g., installation, usage, and updates. It would be valuable to help developers prioritize more urgent issues and improve development efficiency. Unfortunately, there is currently no research focused on understanding the categories and issues of ChatGPT-related projects. To fill this gap, we retrieved 71,244 projects from GitHub using the keyword ‘ChatGPT’ and selected the top 200 representative projects with the highest numbers of stars as our dataset. By analyzing the project descriptions , we identified three primary categories of ChatGPT-related projects, namely ChatGPT Implementation & Training , ChatGPT Application , ChatGPT Improvement & Extension . We further built a classifier for automatically categorizing projects based on the 200 manually annotated projects. Next, we applied a topic modeling technique to 23,609 issues of those projects and identified ten issue topics, e.g., model reply and interaction interface . We analyzed the popularity, difficulty, and evolution of each issue topic within the three project categories and further proposed a method for recommending solutions for open issues by summarizing the pull requests associated with closed issues. Our main findings are: (1) The increase in the number of projects within the three categories is closely related to the development of ChatGPT; and (2) There are significant differences in the popularity, difficulty, and evolutionary trends of the issue topics across the three project categories. Based on these findings, we finally provided implications for project developers and platform managers on how to better develop and manage ChatGPT-related projects, such as offering more fine-grained tags to categorize projects to facilitate their exploration. • We categorized ChatGPT-related projects on GitHub and trained a classification model. • We proposed a topic modeling method for texts in both English and Chinese. • We identified ten topics from the issues of ChatGPT-related projects. • We analyzed the popularity, difficulty, and evolution of issue topics. • We proposed a solution recommendation method for open issues. • We offered suggestions for developers and open-source platforms.

Ähnliche Arbeiten