Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis, Andrea Benedetti, Alexander W. Levis, John P.A. Ioannidis, Ian Shrier, Pim Cuijpers, Simon Gilbody, Lorie A. Kloda, Dean McMillan, Scott B. Patten, Russell J. Steele, Roy C. Ziegelstein, Charles H. Bombardier, Flavia De Lima Osório, Jesse R. Fann, Dwenda Gjerdingen, Femke Lamers, Manote Lotrakul, Sonia R. Loureiro, Bernd LöweJuwita Shaaban, Lesley Stafford, Henk C.P.M. Van Weert, Mary A. Whooley, Linda S. Williams, Karin A. Wittkampf, Albert S. Yeung, Brett D. Thombs

Research output: Contribution to journalArticle

16 Scopus citations

Abstract

In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

Original languageEnglish (US)
Pages (from-to)954-964
Number of pages11
JournalAmerican journal of epidemiology
Volume185
Issue number10
DOIs
StatePublished - May 15 2017

    Fingerprint

Keywords

  • bias
  • depression
  • diagnostic test accuracy
  • individual-patient-data meta-analysis
  • meta-analysis
  • screening
  • selective cutoff reporting

ASJC Scopus subject areas

  • Epidemiology

Cite this

Levis, B., Benedetti, A., Levis, A. W., Ioannidis, J. P. A., Shrier, I., Cuijpers, P., Gilbody, S., Kloda, L. A., McMillan, D., Patten, S. B., Steele, R. J., Ziegelstein, R. C., Bombardier, C. H., De Lima Osório, F., Fann, J. R., Gjerdingen, D., Lamers, F., Lotrakul, M., Loureiro, S. R., ... Thombs, B. D. (2017). Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool. American journal of epidemiology, 185(10), 954-964. https://doi.org/10.1093/aje/kww191