Skip to Main content Skip to Navigation
Journal articles

Designs of trials assessing interventions to improve the peer review process: a vignette-based survey

Amytis Heim 1, 2 Philippe Ravaud 1, 2, 3 Gabriel Baron 1, 2, 3 Isabelle Boutron 1, 2, 3, *
* Corresponding author
1 CRESS - U1153 - Equipe 5 : METHODS - Méthodes de l’évaluation thérapeutique des maladies chroniques
UPD5 - Université Paris Descartes - Paris 5, CRESS (U1153 / UMR_A_1125 / UMR_S_1153) - Centre de Recherche Épidémiologie et Statistique Sorbonne Paris Cité
Abstract : BACKGROUND: We aimed to determine the best study designs for assessing interventions to improve the peer review process according to experts' opinions. Furthermore, for interventions previously evaluated, we determined whether the study designs actually used were rated as the best study designs. METHODS: Study design: A series of six vignette-based surveys exploring the best study designs for six different interventions (training peer reviewers, adding an expert to the peer review process, use of reporting guidelines checklists, blinding peer reviewers to the results (i.e., results-free peer review), giving incentives to peer reviewers, and post-publication peer review). Vignette construction: Vignettes were case scenarios of trials assessing interventions aimed at improving the quality of peer review. For each intervention, the vignette included the study type (e.g., randomized controlled trial [RCT]), setting (e.g., single biomedical journal), and type of manuscript assessed (e.g., actual manuscripts received by the journal); each of these three features varied between vignettes. PARTICIPANTS: Researchers with expertise in peer review or methodology of clinical trials. OUTCOME: Participants were proposed two vignettes describing two different study designs to assess the same intervention and had to indicate which study design they preferred on a scale, from - 5 (preference for study A) to 5 (preference for study B), 0 indicating no preference between the suggested designs (primary outcome). Secondary outcomes were trust in the results and feasibility of the designs. RESULTS: A total of 204 experts assessed 1044 paired comparisons. The preferred study type was RCTs with randomization of manuscripts for four interventions (adding an expert, use of reporting guidelines checklist, results-free peer review, post-publication peer review) and RCTs with randomization of peer reviewers for two interventions (training peer reviewers and using incentives). The preferred setting was mainly several biomedical journals from different publishers, and the preferred type of manuscript was actual manuscripts submitted to journals. However, the most feasible designs were often cluster RCTs and interrupted time series analysis set in a single biomedical journal, with the assessment of a fabricated manuscript. Three interventions were previously assessed: none used the design rated first in preference by experts. CONCLUSION: The vignette-based survey allowed us to identify the best study designs for assessing different interventions to improve peer review according to experts' opinion. There is gap between the preferred study designs and the designs actually used.
Document type :
Journal articles
Complete list of metadatas

Cited literature [21 references]  Display  Hide  Download
Contributor : Myriam Bodescot <>
Submitted on : Monday, March 4, 2019 - 3:49:39 PM
Last modification on : Thursday, April 9, 2020 - 11:56:20 AM
Document(s) archivé(s) le : Wednesday, June 5, 2019 - 3:30:01 PM


Publisher files allowed on an open archive




Amytis Heim, Philippe Ravaud, Gabriel Baron, Isabelle Boutron. Designs of trials assessing interventions to improve the peer review process: a vignette-based survey. BMC Medicine, BioMed Central, 2018, 16 (1), pp.191. ⟨10.1186/s12916-018-1167-7⟩. ⟨inserm-02056430⟩



Record views


Files downloads