Skip to Main content Skip to Navigation
Journal articles

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer 1, 2 Felix Holzmeister 3 Colin Camerer 4 Anna Dreber 5, 3 Juergen Huber 3 Magnus Johannesson 5 Michael Kirchler 3 Roni Iwanir 1 Jeanette Mumford 6 R. Alison Adcock 7 Paolo Avesani 8, 9 Blazej Baczkowski 10 Aahana Bajracharya 11 Leah Bakst 12 Sheryl Ball 13 Marco Barilari 14 Nadège Bault 15 Derek Beaton 16 Julia Beitner 17, 18 Roland Benoit 10 Ruud Berkers 10 Jamil Bhanji 19 Bharat Biswal 20, 21 Sebastian Bobadilla-Suarez 22 Tiago Bortolini 23 Katherine Bottenhorn 24 Alexander Bowring 25 Senne Braem 26, 27 Hayley Brooks 28 Emily Brudner 19 Cristian Calderon 27 Julia Camilleri 29, 30 Jaime Castrellon 7 Luca Cecchetti 31 Edna Cieslik 29, 30 Zachary Cole 32 Olivier Collignon 9, 14 Robert Cox 33 William Cunningham 34 Stefan Czoschke 18 Kamalaker Dadi 35 Charles Davis 36, 37 Alberto De Luca 38 Mauricio Delgado 19 Lysia Demetriou 39, 25 Jeffrey Dennison 40 Xin Di 20, 21 Erin Dickie 34 Ekaterina Dobryakova 41 Claire Donnat 42 Juergen Dukart 30, 29 Niall Duncan 43, 44 Joke Durnez 42 Amr Eed 45 Simon Eickhoff 30, 29 Andrew Erhart 28 Laura Fontanesi 46 G. Matthew Fricke 47 Shiguang Fu 48 Adriana Galván 49 Remi Gau 50 Sarah Genon 30, 29 Tristan Glatard 51 Enrico Glerean 52 Jelle Goeman 53 Sergej Golowin 43 Carlos González-García 27 Krzysztof Gorgolewski 42 Cheryl Grady 16 Mikella Green 7 João Guassi Moreira 49 Olivia Guest 22 Shabnam Hakimi 7 J. Paul Hamilton 54 Roeland Hancock 37 Giacomo Handjaras 31 Bronson Harry 55 Colin Hawco 56 Peer Herholz 57 Gabrielle Herman 56 Stephan Heunis 58, 59 Felix Hoffstaedter 30, 29 Jeremy Hogeveen 47 Susan Holmes 42 Chuan-Peng Hu 60 Scott Huettel 7 Matthew Hughes 61 Vittorio Iacovella 9 Alexandru Iordan 62 Peder Isager 58 Ayse Isik 63 Andrew Jahn 62 Matthew Johnson 32 Tom Johnstone 61 Michael Joseph 56 Anthony Juliano 41 Joseph Kable 64 Michalis Kassinopoulos 57 Cemal Koba 31 Xiang-Zhen Kong 65 Timothy Koscik 66 Nuri Erkut Kucukboyaci 67, 41 Brice Kuhl 68 Sebastian Kupek 3 Angela Laird 24 Claus Lamm 69 Robert Langner 30, 29 Nina Lauharatanahirun 64 Hongmi Lee 70 Sangil Lee 64 Alexander Leemans 38 Andrea Leo 31 Elise Lesage 27 Flora Li 71 Monica Li 37, 72 Phui Cheng Lim 32 Evan Lintz 32 Schuyler Liphardt 47 Annabel Losecaat Vermeer 69 Bradley Love 22, 73 Michael Mack 34 Norberto Malpica 74 Theo Marins 23 Camille Maumet 75 Kelsey Mcdonald 7 Joseph Mcguire 12 Helena Melero 74 Adriana Méndez Leal 49 Benjamin Meyer 60 Kristin Meyer 76 Glad Mihai 10, 77 Georgios Mitsis 57 Jorge Moll 23, 42 Dylan Nielson 33 Gustav Nilsonne 78, 79 Michael Notter 80 Emanuele Olivetti 8, 9 Adrian Onicas 31 Paolo Papale 31, 81 Kaustubh Patil 30, 29 Jonathan Peelle 11 Alexandre Pérez 57 Doris Pischedda 82, 83, 84 Jean-Baptiste Poline 57, 85 Yanina Prystauka 37 Shruti Ray 20 Patricia Reuter-Lorenz 62 Richard Reynolds 33 Emiliano Ricciardi 31 Jenny Rieck 16 Anais Rodriguez-Thompson 76 Anthony Romyn 34 Taylor Salo 24 Gregory Samanez-Larkin 7 Emilio Sanz-Morales 74 Margaret Schlichting 34 Douglas Schultz 32 Qiang Shen 48 Margaret Sheridan 76 Jennifer Silvers 49 Kenny Skagerlund 54 Alec Smith 13 David Smith 40 Peter Sokol-Hessner 28 Simon Steinkamp 30 Sarah Tashjian 49 Bertrand Thirion 35 John Thorp 86 Gustav Tinghög 54 Loreen Tisdall 42, 46 Steven Tompson 87 Claudio Toro-Serey 12 Juan Jesus Torre Tresols 35 Leonardo Tozzi 88 Vuong Truong 43, 44 Luca Turella 9 Anna van ‘t Veer 89 Tom Verguts 27 Jean Vettel 90, 64 Sagana Vijayarajah 34 Khoi Vo 7 Matthew Wall 39, 22 Wouter Weeda 89 Susanne Weis 30, 29 David White 61 David Wisniewski 27 Alba Xifra-Porxas 57 Emily Yearling 37 Sangsuk Yoon 91 Rui Yuan 88 Kenneth Yuen 60 Lei Zhang 69 Xu Zhang 37 Joshua Zosky 32 Thomas Nichols 25, * Russell Poldrack 42, * Tom Schonberg 1, *
* Corresponding author
1 Tel Aviv University [Tel Aviv]
2 Dartmouth College [Hanover]
3 University of Innsbruck
4 CALTECH - California Institute of Technology
5 SSE - Stockholm School of Economics
6 University of Wisconsin-Madison
7 Duke University [Durham]
8 FBK - Fondazione Bruno Kessler [Trento, Italy]
9 University of Trento [Trento]
10 IMPNSC - Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig]
11 Washington University in St Louis
12 BU - Boston University [Boston]
13 Virginia Tech [Blacksburg]
14 UCL - Université Catholique de Louvain
15 Plymouth University
16 RRI - Rotman Research Institute at the Baycrest Centre
17 UvA - University of Amsterdam [Amsterdam]
18 Goethe-University Frankfurt am Main
19 Rutgers University [Newark]
20 NJIT - New Jersey Institute of Technology [Newark]
21 UESTC - University of Electronic Science and Technology of China
22 UCL - University College of London [London]
23 D'Or Institute for Research and Education [Rio de Janeiro]
24 FIU - Florida International University
25 University of Oxford [Oxford]
26 VUB - Vrije Universiteit Brussel
27 UGENT - Ghent University [Belgium]
28 University of Denver
29 Heinrich-Heine-Universität Düsseldorf [Düsseldorf]
30 Jülich Research Centre
31 IMT - School for Advanced Studies Lucca
32 University of Nebraska [Lincoln]
33 NIH - National Institutes of Health [Bethesda]
34 University of Toronto
35 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
Inria Saclay - Ile de France, NEUROSPIN - Service NEUROSPIN
36 Lancashire Teaching Hospital
37 UCONN - University of Connecticut
38 University Medical Center [Utrecht]
39 Imperial College London
40 Temple University [Philadelphia]
41 Kessler Foundation [East Hanover]
42 Stanford University
43 Taipei Medical University
44 TMU-ShuangHo Hospital [New Taipei City]
45 Instituto de Neurociencias de Alicante
46 Unibas - University of Basel
47 The University of New Mexico [Albuquerque]
48 Zhejiang University of Technology
49 UCLA - University of California [Los Angeles]
50 UCL - Université catholique de Lille
51 Concordia University [Montreal]
52 A? - Aalto University
53 Leiden University Medical Center (LUMC)
54 LIU - Linköping University
55 UWS - Western Sydney University
56 CAMH - Centre for Addiction and Mental Health [Toronto]
57 McGill University = Université McGill [Montréal, Canada]
58 TU/e - Eindhoven University of Technology [Eindhoven]
59 Epilepsy Centre Kempenhaeghe [Heeze]
60 LIR - Leibniz-Institut für Resilienzforschung [Mainz]
61 Swinburne University of Technology (Hawthorn campus)
62 University of Michigan [Ann Arbor]
63 Max Planck Institute for Empirical Aesthetics [Frankfurt am Main]
64 University of Pennsylvania [Philadelphia]
65 Max Planck Institute for Psycholinguistics
66 Carver College of Medicine, University of Iowa
67 NJMS - Rutgers New Jersey Medical School
68 University of Oregon [Eugene]
69 University of Vienna [Vienna]
70 JHU - Johns Hopkins University
71 NJU - Nanjing University
72 Haskins Laboratories
73 The Alan Turing Institute
74 URJC - Universidad Rey Juan Carlos [Madrid]
75 Empenn
INSERM - Institut National de la Santé et de la Recherche Médicale, Inria Rennes – Bretagne Atlantique , IRISA-D5 - SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE
76 UNC - University of North Carolina [Chapel Hill]
77 TU Dresden - Technische Universität Dresden = Dresden University of Technology
78 Karolinska Institutet [Stockholm]
79 Stockholm University
80 UNIL - Université de Lausanne
81 NIN - Netherlands Institute for Neuroscience
82 Charité - Universitätsmedizin Berlin / Charite - University Medicine Berlin
83 Humboldt-Universität zu Berlin
84 TU - Technische Universität Berlin
85 University of California [Berkeley]
86 Columbia University [New York]
87 Human Research and Engineering Directorate [Aberdeen Proving Ground]
88 Stanford School of Medicine [Stanford]
89 Leiden University
90 UCSB - University of California [Santa Barbara]
91 University of Dayton
Abstract : Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Document type :
Journal articles
Complete list of metadatas
https://www.hal.inserm.fr/inserm-02914443
Contributor : Camille Maumet <>
Submitted on : Tuesday, August 11, 2020 - 9:31:53 PM
Last modification on : Friday, February 26, 2021 - 5:58:02 PM

Links full text

openaccess

Identifiers

Collections

INSERM | UNIV-RENNES1 | CEA | INRIA | CEA-UPSAY | NEUROSPIN | INSA-RENNES | JOLIOT | IRISA | CNRS | CENTRALESUPELEC | UR1-UFR-ISTIC | CEA-DRF | UNIV-RENNES

Citation

Rotem Botvinik-Nezer, Felix Holzmeister, Colin Camerer, Anna Dreber, Juergen Huber, et al.. Variability in the analysis of a single neuroimaging dataset by many teams. Nature, Nature Publishing Group, 2020, 582 (7810), pp.84-88. ⟨10.1038/s41586-020-2314-9⟩. ⟨inserm-02914443⟩

Export

BibTeX TEI DC DCterms EndNote

Share

Metrics

Record views

123