A scientific evaluation of the use of limited versions of AI tools as support in identifying and defining simple non-English lithological terms

Abstract

This study was prompted by the need to examine how well AI tools and large language models (LLMs) handle geological issues, particularly lithological issues, in languages other than English. The study aimed to evaluate the quality of responses in Polish generated by free versions of AI tools accessible to non-geologists with limited technological expertise. The survey, which was conducted between February and May 2025, involved people with a background in geology and students of geosciences, whose task was to evaluate each of the responses received. The lithology questions were the same for all respondents. The study involved using ChatGPT, Claude, DeepSeek AI, Google Gemini, Microsoft Copilot, Perplexity AI, and Qwen2.5. Respondents were most likely to use ChatGPT, Microsoft Copilot and Perplexity. The assessment covered the factual accuracy of the responses, the reliability of the sources referenced, and the comprehensibility of the responses received. The study revealed that not all AI tools can process the Polish language effectively, and a lack of relevant publications in Polish hinders the improvement of response quality. It was shown that more complete and complex queries that delve deeper into substantive knowledge enable higher quality and more satisfactory results. These results indicate the need to adapt algorithms to regional scientific terminology specifics, which could enhance the quality, reliability and usefulness of the content.

Ähnliche Arbeiten