Neural Language Models are not Born Equal to Fit Brain Data, but\n Training Helps

Abstract

Neural Language Models (NLMs) have made tremendous advances during the last\nyears, achieving impressive performance on various linguistic tasks.\nCapitalizing on this, studies in neuroscience have started to use NLMs to study\nneural activity in the human brain during language processing. However, many\nquestions remain unanswered regarding which factors determine the ability of a\nneural language model to capture brain activity (aka its 'brain score'). Here,\nwe make first steps in this direction and examine the impact of test loss,\ntraining corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT),\non the prediction of functional Magnetic Resonance Imaging timecourses of\nparticipants listening to an audiobook. We find that (1) untrained versions of\neach model already explain significant amount of signal in the brain by\ncapturing similarity in brain responses across identical words, with the\nuntrained LSTM outperforming the transformerbased models, being less impacted\nby the effect of context; (2) that training NLP models improves brain scores in\nthe same brain regions irrespective of the model's architecture; (3) that\nPerplexity (test loss) is not a good predictor of brain score; (4) that\ntraining data have a strong influence on the outcome and, notably, that\noff-the-shelf models may lack statistical power to detect brain activations.\nOverall, we outline the impact of modeltraining choices, and suggest good\npractices for future studies aiming at explaining the human language system\nusing neural language models.\n

Ähnliche Arbeiten