Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps - Inserm - Institut national de la santé et de la recherche médicale Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps

Résumé

Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks. Capitalizing on this, studies in neuroscience have started to use NLMs to study neural activity in the human brain during language processing. However, many questions remain unanswered regarding which factors determine the ability of a neural language model to capture brain activity (aka its 'brain score'). Here, we make first steps in this direction and examine the impact of test loss, training corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT), on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that (1) untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words, with the untrained LSTM outperforming the transformerbased models, being less impacted by the effect of context; (2) that training NLP models improves brain scores in the same brain regions irrespective of the model's architecture; (3) that Perplexity (test loss) is not a good predictor of brain score; (4) that training data have a strong influence on the outcome and, notably, that off-the-shelf models may lack statistical power to detect brain activations. Overall, we outline the impact of modeltraining choices, and suggest good practices for future studies aiming at explaining the human language system using neural language models.
Fichier principal
Vignette du fichier
final_paper.pdf (34.36 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03704504 , version 1 (07-07-2022)

Identifiants

Citer

Alexandre Pasquiou, Yair Lakretz, John Hale, Bertrand Thirion, Christophe Pallier. Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps. ICML 2022 - 39th International Conference on Machine Learning, Jul 2022, Baltimore, United States. pp.18. ⟨hal-03704504⟩
133 Consultations
16 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More