Can AI Predict Scientific Research? a Case Study on the NLP Domain

Abstract

Using literature embedding and cluster analysis methods, this study proposes a predictive framework to track the evolution of the academic literature. We used Doc2Vec to create semantic embedments for ACL papers from 2018 to 2021, and then used K-means clustering to identify key papers within those clusters. These key papers were used as hints for a large language model (llm) to predict the content of the 2022 publication. The accuracy of these predictions was assessed by comparing the AI-generated content to the actual 2,022 papers, using two main criteria: surface similarity (measured by cosine distance between embedments) and semantic consistency (assessed using BERTScore). Empirical validation shows that the LLM-generated predictions achieve a significant amount of surface similarity (0.82 mean cosine) and moderate semantic consistency (0.79 BERTScores) with the actual 2022 publication, but show a diminished ability to predict entirely new methodological paradigms.

Ähnliche Arbeiten