Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis, Andrea Benedetti, Alexander W. Levis, John P.A. Ioannidis, Ian Shrier, Pim Cuijpers, Simon Gilbody, Lorie A. Kloda, Dean McMillan, Scott B. Patten, Russell J. Steele, Roy C. Ziegelstein, Charles H. Bombardier, Flavia De Lima Osório, Jesse R. Fann, Dwenda Gjerdingen, Femke Lamers, Manote Lotrakul, Sonia R. Loureiro, Bernd LöweJuwita Shaaban, Lesley Stafford, Henk C.P.M. Van Weert, Mary A. Whooley, Linda S. Williams, Karin A. Wittkampf, Albert S. Yeung, Brett D. Thombs

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

Original languageEnglish (US)
Pages (from-to)954-964
Number of pages11
JournalAmerican Journal of Epidemiology
Volume185
Issue number10
DOIs
StatePublished - May 15 2017

Fingerprint

Routine Diagnostic Tests
Meta-Analysis
Health
Surveys and Questionnaires

Keywords

  • bias
  • depression
  • diagnostic test accuracy
  • individual-patient-data meta-analysis
  • meta-analysis
  • screening
  • selective cutoff reporting

ASJC Scopus subject areas

  • Epidemiology

Cite this

Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy : A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool. / Levis, Brooke; Benedetti, Andrea; Levis, Alexander W.; Ioannidis, John P.A.; Shrier, Ian; Cuijpers, Pim; Gilbody, Simon; Kloda, Lorie A.; McMillan, Dean; Patten, Scott B.; Steele, Russell J.; Ziegelstein, Roy C.; Bombardier, Charles H.; De Lima Osório, Flavia; Fann, Jesse R.; Gjerdingen, Dwenda; Lamers, Femke; Lotrakul, Manote; Loureiro, Sonia R.; Löwe, Bernd; Shaaban, Juwita; Stafford, Lesley; Van Weert, Henk C.P.M.; Whooley, Mary A.; Williams, Linda S.; Wittkampf, Karin A.; Yeung, Albert S.; Thombs, Brett D.

In: American Journal of Epidemiology, Vol. 185, No. 10, 15.05.2017, p. 954-964.

Research output: Contribution to journalArticle

Levis, B, Benedetti, A, Levis, AW, Ioannidis, JPA, Shrier, I, Cuijpers, P, Gilbody, S, Kloda, LA, McMillan, D, Patten, SB, Steele, RJ, Ziegelstein, RC, Bombardier, CH, De Lima Osório, F, Fann, JR, Gjerdingen, D, Lamers, F, Lotrakul, M, Loureiro, SR, Löwe, B, Shaaban, J, Stafford, L, Van Weert, HCPM, Whooley, MA, Williams, LS, Wittkampf, KA, Yeung, AS & Thombs, BD 2017, 'Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool', American Journal of Epidemiology, vol. 185, no. 10, pp. 954-964. https://doi.org/10.1093/aje/kww191
Levis, Brooke ; Benedetti, Andrea ; Levis, Alexander W. ; Ioannidis, John P.A. ; Shrier, Ian ; Cuijpers, Pim ; Gilbody, Simon ; Kloda, Lorie A. ; McMillan, Dean ; Patten, Scott B. ; Steele, Russell J. ; Ziegelstein, Roy C. ; Bombardier, Charles H. ; De Lima Osório, Flavia ; Fann, Jesse R. ; Gjerdingen, Dwenda ; Lamers, Femke ; Lotrakul, Manote ; Loureiro, Sonia R. ; Löwe, Bernd ; Shaaban, Juwita ; Stafford, Lesley ; Van Weert, Henk C.P.M. ; Whooley, Mary A. ; Williams, Linda S. ; Wittkampf, Karin A. ; Yeung, Albert S. ; Thombs, Brett D. / Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy : A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool. In: American Journal of Epidemiology. 2017 ; Vol. 185, No. 10. pp. 954-964.
@article{7f0f72aede0d4de1b4287c5cce2fd1b6,
title = "Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy: A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool",
abstract = "In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven {"}optimal{"} cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the {"}standard{"} PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1{\%} of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5{\%}-15{\%} for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.",
keywords = "bias, depression, diagnostic test accuracy, individual-patient-data meta-analysis, meta-analysis, screening, selective cutoff reporting",
author = "Brooke Levis and Andrea Benedetti and Levis, {Alexander W.} and Ioannidis, {John P.A.} and Ian Shrier and Pim Cuijpers and Simon Gilbody and Kloda, {Lorie A.} and Dean McMillan and Patten, {Scott B.} and Steele, {Russell J.} and Ziegelstein, {Roy C.} and Bombardier, {Charles H.} and {De Lima Os{\'o}rio}, Flavia and Fann, {Jesse R.} and Dwenda Gjerdingen and Femke Lamers and Manote Lotrakul and Loureiro, {Sonia R.} and Bernd L{\"o}we and Juwita Shaaban and Lesley Stafford and {Van Weert}, {Henk C.P.M.} and Whooley, {Mary A.} and Williams, {Linda S.} and Wittkampf, {Karin A.} and Yeung, {Albert S.} and Thombs, {Brett D.}",
year = "2017",
month = "5",
day = "15",
doi = "10.1093/aje/kww191",
language = "English (US)",
volume = "185",
pages = "954--964",
journal = "American Journal of Epidemiology",
issn = "0002-9262",
publisher = "Oxford University Press",
number = "10",

}

TY - JOUR

T1 - Selective Cutoff Reporting in Studies of Diagnostic Test Accuracy

T2 - A Comparison of Conventional and Individual-Patient-Data Meta-Analyses of the Patient Health Questionnaire-9 Depression Screening Tool

AU - Levis, Brooke

AU - Benedetti, Andrea

AU - Levis, Alexander W.

AU - Ioannidis, John P.A.

AU - Shrier, Ian

AU - Cuijpers, Pim

AU - Gilbody, Simon

AU - Kloda, Lorie A.

AU - McMillan, Dean

AU - Patten, Scott B.

AU - Steele, Russell J.

AU - Ziegelstein, Roy C.

AU - Bombardier, Charles H.

AU - De Lima Osório, Flavia

AU - Fann, Jesse R.

AU - Gjerdingen, Dwenda

AU - Lamers, Femke

AU - Lotrakul, Manote

AU - Loureiro, Sonia R.

AU - Löwe, Bernd

AU - Shaaban, Juwita

AU - Stafford, Lesley

AU - Van Weert, Henk C.P.M.

AU - Whooley, Mary A.

AU - Williams, Linda S.

AU - Wittkampf, Karin A.

AU - Yeung, Albert S.

AU - Thombs, Brett D.

PY - 2017/5/15

Y1 - 2017/5/15

N2 - In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

AB - In studies of diagnostic test accuracy, authors sometimes report results only for a range of cutoff points around data-driven "optimal" cutoffs. We assessed selective cutoff reporting in studies of the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared conventional meta-analysis of published results only with individual-patient-data meta-analysis of results derived from all cutoff points, using data from 13 of 16 studies published during 2004-2009 that were included in a published conventional metaanalysis. For the "standard" PHQ-9 cutoff of 10, accuracy results had been published by 11 of the studies. For all other relevant cutoffs, 3-6 studies published accuracy results. For all cutoffs examined, specificity estimates in conventional and individual-patient-data meta-analyses were within 1% of each other. Sensitivity estimates were similar for the cutoff of 10 but differed by 5%-15% for other cutoffs. In samples where the PHQ-9 was poorly sensitive at the standard cutoff, authors tended to report results for lower cutoffs that yielded optimal results. When the PHQ-9 was highly sensitive, authors more often reported results for higher cutoffs. Consequently, in the conventional metaanalysis, sensitivity increased as cutoff severity increased across part of the cutoff range-an impossibility if all data are analyzed. In sum, selective reporting by primary study authors of only results from cutoffs that perform well in their study can bias accuracy estimates in meta-analyses of published results.

KW - bias

KW - depression

KW - diagnostic test accuracy

KW - individual-patient-data meta-analysis

KW - meta-analysis

KW - screening

KW - selective cutoff reporting

UR - http://www.scopus.com/inward/record.url?scp=85020483813&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85020483813&partnerID=8YFLogxK

U2 - 10.1093/aje/kww191

DO - 10.1093/aje/kww191

M3 - Article

C2 - 28419203

AN - SCOPUS:85020483813

VL - 185

SP - 954

EP - 964

JO - American Journal of Epidemiology

JF - American Journal of Epidemiology

SN - 0002-9262

IS - 10

ER -