Skip to Main content Skip to Navigation
Journal articles

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer 1, 2 Felix Holzmeister 3 Colin Camerer 4 Anna Dreber 5, 3 Juergen Huber 3 Magnus Johannesson 5 Michael Kirchler 3 Roni Iwanir 1 Jeanette Mumford 6 R. Alison Adcock 7 Paolo Avesani 8, 9 Blazej Baczkowski 10 Aahana Bajracharya 11 Leah Bakst 12 Sheryl Ball 13 Marco Barilari 14 Nadège Bault 15 Derek Beaton 16 Julia Beitner 17, 18 Roland Benoit 10 Ruud Berkers 10 Jamil Bhanji 19 Bharat Biswal 20, 21 Sebastian Bobadilla-Suarez 22 Tiago Bortolini 23 Katherine Bottenhorn 24 Alexander Bowring 25 Senne Braem 26, 27 Hayley Brooks 28 Emily Brudner 19 Cristian Calderon 27 Julia Camilleri 29, 30 Jaime Castrellon 7 Luca Cecchetti 31 Edna Cieslik 29, 30 Zachary Cole 32 Olivier Collignon 9, 14 Robert Cox 33 William Cunningham 34 Stefan Czoschke 18 Kamalaker Dadi 35 Charles Davis 36, 37 Alberto De Luca 38 Mauricio Delgado 19 Lysia Demetriou 39, 25 Jeffrey Dennison 40 Xin Di 20, 21 Erin Dickie 34 Ekaterina Dobryakova 41 Claire Donnat 42 Juergen Dukart 30, 29 Niall Duncan 43, 44 Joke Durnez 42 Amr Eed 45 Simon Eickhoff 30, 29 Andrew Erhart 28 Laura Fontanesi 46 G. Matthew Fricke 47 Shiguang Fu 48 Adriana Galván 49 Remi Gau 50 Sarah Genon 30, 29 Tristan Glatard 51 Enrico Glerean 52 Jelle Goeman 53 Sergej Golowin 43 Carlos González-García 27 Krzysztof Gorgolewski 42 Cheryl Grady 16 Mikella Green 7 João Guassi Moreira 49 Olivia Guest 22 Shabnam Hakimi 7 J. Paul Hamilton 54 Roeland Hancock 37 Giacomo Handjaras 31 Bronson Harry 55 Colin Hawco 56 Peer Herholz 57 Gabrielle Herman 56 Stephan Heunis 58, 59 Felix Hoffstaedter 30, 29 Jeremy Hogeveen 47 Susan Holmes 42 Chuan-Peng Hu 60 Scott Huettel 7 Matthew Hughes 61 Vittorio Iacovella 9 Alexandru Iordan 62 Peder Isager 58 Ayse Isik 63 Andrew Jahn 62 Matthew Johnson 32 Tom Johnstone 61 Michael Joseph 56 Anthony Juliano 41 Joseph Kable 64 Michalis Kassinopoulos 57 Cemal Koba 31 Xiang-Zhen Kong 65 Timothy Koscik 66 Nuri Erkut Kucukboyaci 67, 41 Brice Kuhl 68 Sebastian Kupek 3 Angela Laird 24 Claus Lamm 69 Robert Langner 30, 29 Nina Lauharatanahirun 64 Hongmi Lee 70 Sangil Lee 64 Alexander Leemans 38 Andrea Leo 31 Elise Lesage 27 Flora Li 71 Monica Li 37, 72 Phui Cheng Lim 32 Evan Lintz 32 Schuyler Liphardt 47 Annabel Losecaat Vermeer 69 Bradley Love 22, 73 Michael Mack 34 Norberto Malpica 74 Theo Marins 23 Camille Maumet 75 Kelsey Mcdonald 7 Joseph Mcguire 12 Helena Melero 74 Adriana Méndez Leal 49 Benjamin Meyer 60 Kristin Meyer 76 Glad Mihai 10, 77 Georgios Mitsis 57 Jorge Moll 23, 42 Dylan Nielson 33 Gustav Nilsonne 78, 79 Michael Notter 80 Emanuele Olivetti 8, 9 Adrian Onicas 31 Paolo Papale 31, 81 Kaustubh Patil 30, 29 Jonathan Peelle 11 Alexandre Pérez 57 Doris Pischedda 82, 83, 84 Jean-Baptiste Poline 57, 85 Yanina Prystauka 37 Shruti Ray 20 Patricia Reuter-Lorenz 62 Richard Reynolds 33 Emiliano Ricciardi 31 Jenny Rieck 16 Anais Rodriguez-Thompson 76 Anthony Romyn 34 Taylor Salo 24 Gregory Samanez-Larkin 7 Emilio Sanz-Morales 74 Margaret Schlichting 34 Douglas Schultz 32 Qiang Shen 48 Margaret Sheridan 76 Jennifer Silvers 49 Kenny Skagerlund 54 Alec Smith 13 David Smith 40 Peter Sokol-Hessner 28 Simon Steinkamp 30 Sarah Tashjian 49 Bertrand Thirion 35 John Thorp 86 Gustav Tinghög 54 Loreen Tisdall 42, 46 Steven Tompson 87 Claudio Toro-Serey 12 Juan Jesus Torre Tresols 35 Leonardo Tozzi 88 Vuong Truong 43, 44 Luca Turella 9 Anna van ‘t Veer 89 Tom Verguts 27 Jean Vettel 90, 64 Sagana Vijayarajah 34 Khoi Vo 7 Matthew Wall 39, 22 Wouter Weeda 89 Susanne Weis 30, 29 David White 61 David Wisniewski 27 Alba Xifra-Porxas 57 Emily Yearling 37 Sangsuk Yoon 91 Rui Yuan 88 Kenneth Yuen 60 Lei Zhang 69 Xu Zhang 37 Joshua Zosky 32 Thomas Nichols 25 Russell Poldrack 42 Tom Schonberg 1
35 PARIETAL - Modelling brain structure, function and variability based on high-field MRI data
Inria Saclay - Ile de France, NEUROSPIN - Service NEUROSPIN
75 Empenn
INSERM - Institut National de la Santé et de la Recherche Médicale, Inria Rennes – Bretagne Atlantique , IRISA-D5 - SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE
Abstract : Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Document type :
Journal articles
Complete list of metadatas

https://www.hal.inserm.fr/inserm-02914443
Contributor : Camille Maumet <>
Submitted on : Tuesday, August 11, 2020 - 9:31:53 PM
Last modification on : Friday, September 25, 2020 - 2:22:02 PM

Links full text

Identifiers

Citation

Rotem Botvinik-Nezer, Felix Holzmeister, Colin Camerer, Anna Dreber, Juergen Huber, et al.. Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 2020, 582 (7810), pp.84-88. ⟨10.1038/s41586-020-2314-9⟩. ⟨inserm-02914443⟩

Share

Metrics

Record views

44