Exploring Bias Evaluation Techniques for Quantifying Large Language Model Biases

Abstract

In recent years, there has been a surge in the adoption of large language models (LLMs) such as “ChatG PT” trained by OpenAI. These models have gained popularity due to their impressive performance in various real-world applications. However, research has shown that small pre-trained language models (PLMs) such as BERT exhibit biases, particularly gender bias, that mirror societal stereotypes. Given the shared architecture between LLMs and small PLMs like Transformer, there is concern that these biases may also exist in LLMs. Although some studies suggest the presence of biases in LLMs, there is no consensus on how these biases should be measured. This paper employs three internal bias metrics, namely SEAT, StereoSet, and CrowS Pairs, to evaluate nine bias involving gender, age, race, occupation, nationality, religion, sexual orientation, physical appearance and disability in five open source LLMs (Llama, Llama2, Alpaca, Vicuna, and MPT), thereby determining their specific bias level. The experimental results demonstrate varying degrees of bias within these LLMs, with some models displaying high levels of bias that could potentially lead to harm in specific domains. Interestingly, we also discover that despite their larger architectures and greater number of parameters compared to small PLMs like BERT, these LLMs exhibit a lower level of bias. We posit that the inclusion of fairness considerations during the pre-training phase of these Language Model-based Learners (LLMs) is the primary contributing factor. This involves prioritizing the use of “fair” corpora while constructing the training data, and our experimental findings confirm the effectiveness of such an approach. Finally, by identifying the presence and measuring the specific level of bias, we contribute to the ongoing discourse on the mitigation of bias and the responsible usage of LLMs in various applications.

Ähnliche Arbeiten