Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies

Sean Grant, Marika Booth, Dmitry Khodyakov

Research output: Contribution to journalArticle

10 Scopus citations

Abstract

Objectives: To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes. Study Design and Setting: Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels. Results: Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus. Conclusion: Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.

Original languageEnglish (US)
Pages (from-to)96-105
Number of pages10
JournalJournal of Clinical Epidemiology
Volume99
DOIs
StatePublished - Jul 2018
Externally publishedYes

Keywords

  • Consensus
  • Delphi process
  • Expert panel
  • Open science framework
  • Preanalysis plan
  • Preregistration

ASJC Scopus subject areas

  • Epidemiology

Fingerprint Dive into the research topics of 'Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies'. Together they form a unique fingerprint.

  • Cite this