Optimal two-phase sampling design for comparing accuracies of two binary classification rules

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

In this paper, we consider the design for comparing the performance of two binary classification rules, for example, two record linkage algorithms or two screening tests. Statistical methods are well developed for comparing these accuracy measures when the gold standard is available for every unit in the sample, or in a two-phase study when the gold standard is ascertained only in the second phase in a subsample using a fixed sampling scheme. However, these methods do not attempt to optimize the sampling scheme to minimize the variance of the estimators of interest. In comparing the performance of two classification rules, the parameters of primary interest are the difference in sensitivities, specificities, and positive predictive values. We derived the analytic variance formulas for these parameter estimates and used them to obtain the optimal sampling design. The efficiency of the optimal sampling design is evaluated through an empirical investigation that compares the optimal sampling with simple random sampling and with proportional allocation. Results of the empirical study show that the optimal sampling design is similar for estimating the difference in sensitivities and in specificities, and both achieve a substantial amount of variance reduction with an over-sample of subjects with discordant results and under-sample of subjects with concordant results. A heuristic rule is recommended when there is no prior knowledge of individual sensitivities and specificities, or the prevalence of the true positive findings in the study population. The optimal sampling is applied to a real-world example in record linkage to evaluate the difference in classification accuracy of two matching algorithms.

Original languageEnglish
Pages (from-to)500-513
Number of pages14
JournalStatistics in Medicine
Volume33
Issue number3
DOIs
StatePublished - Feb 10 2014

Fingerprint

Two-phase Sampling
Binary Classification
Sampling Design
Classification Rules
Record Linkage
Specificity
Sensitivity and Specificity
Gold
Simple Random Sampling
Variance Reduction
Matching Algorithm
Prior Knowledge
Statistical method
Empirical Study
Screening
Efficiency
Directly proportional
Optimise
Heuristics
Minimise

Keywords

  • Diagnostic accuracy
  • Diagnostic test
  • Positive predicted value
  • Record linkage
  • Sensitivity
  • Specificity
  • Stratified sampling

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability

Cite this

Optimal two-phase sampling design for comparing accuracies of two binary classification rules. / Xu, Huiping; Hui, Siu; Grannis, Shaun.

In: Statistics in Medicine, Vol. 33, No. 3, 10.02.2014, p. 500-513.

Research output: Contribution to journalArticle

@article{3ec09606f767497a866ff1913c4468bd,
title = "Optimal two-phase sampling design for comparing accuracies of two binary classification rules",
abstract = "In this paper, we consider the design for comparing the performance of two binary classification rules, for example, two record linkage algorithms or two screening tests. Statistical methods are well developed for comparing these accuracy measures when the gold standard is available for every unit in the sample, or in a two-phase study when the gold standard is ascertained only in the second phase in a subsample using a fixed sampling scheme. However, these methods do not attempt to optimize the sampling scheme to minimize the variance of the estimators of interest. In comparing the performance of two classification rules, the parameters of primary interest are the difference in sensitivities, specificities, and positive predictive values. We derived the analytic variance formulas for these parameter estimates and used them to obtain the optimal sampling design. The efficiency of the optimal sampling design is evaluated through an empirical investigation that compares the optimal sampling with simple random sampling and with proportional allocation. Results of the empirical study show that the optimal sampling design is similar for estimating the difference in sensitivities and in specificities, and both achieve a substantial amount of variance reduction with an over-sample of subjects with discordant results and under-sample of subjects with concordant results. A heuristic rule is recommended when there is no prior knowledge of individual sensitivities and specificities, or the prevalence of the true positive findings in the study population. The optimal sampling is applied to a real-world example in record linkage to evaluate the difference in classification accuracy of two matching algorithms.",
keywords = "Diagnostic accuracy, Diagnostic test, Positive predicted value, Record linkage, Sensitivity, Specificity, Stratified sampling",
author = "Huiping Xu and Siu Hui and Shaun Grannis",
year = "2014",
month = "2",
day = "10",
doi = "10.1002/sim.5946",
language = "English",
volume = "33",
pages = "500--513",
journal = "Statistics in Medicine",
issn = "0277-6715",
publisher = "John Wiley and Sons Ltd",
number = "3",

}

TY - JOUR

T1 - Optimal two-phase sampling design for comparing accuracies of two binary classification rules

AU - Xu, Huiping

AU - Hui, Siu

AU - Grannis, Shaun

PY - 2014/2/10

Y1 - 2014/2/10

N2 - In this paper, we consider the design for comparing the performance of two binary classification rules, for example, two record linkage algorithms or two screening tests. Statistical methods are well developed for comparing these accuracy measures when the gold standard is available for every unit in the sample, or in a two-phase study when the gold standard is ascertained only in the second phase in a subsample using a fixed sampling scheme. However, these methods do not attempt to optimize the sampling scheme to minimize the variance of the estimators of interest. In comparing the performance of two classification rules, the parameters of primary interest are the difference in sensitivities, specificities, and positive predictive values. We derived the analytic variance formulas for these parameter estimates and used them to obtain the optimal sampling design. The efficiency of the optimal sampling design is evaluated through an empirical investigation that compares the optimal sampling with simple random sampling and with proportional allocation. Results of the empirical study show that the optimal sampling design is similar for estimating the difference in sensitivities and in specificities, and both achieve a substantial amount of variance reduction with an over-sample of subjects with discordant results and under-sample of subjects with concordant results. A heuristic rule is recommended when there is no prior knowledge of individual sensitivities and specificities, or the prevalence of the true positive findings in the study population. The optimal sampling is applied to a real-world example in record linkage to evaluate the difference in classification accuracy of two matching algorithms.

AB - In this paper, we consider the design for comparing the performance of two binary classification rules, for example, two record linkage algorithms or two screening tests. Statistical methods are well developed for comparing these accuracy measures when the gold standard is available for every unit in the sample, or in a two-phase study when the gold standard is ascertained only in the second phase in a subsample using a fixed sampling scheme. However, these methods do not attempt to optimize the sampling scheme to minimize the variance of the estimators of interest. In comparing the performance of two classification rules, the parameters of primary interest are the difference in sensitivities, specificities, and positive predictive values. We derived the analytic variance formulas for these parameter estimates and used them to obtain the optimal sampling design. The efficiency of the optimal sampling design is evaluated through an empirical investigation that compares the optimal sampling with simple random sampling and with proportional allocation. Results of the empirical study show that the optimal sampling design is similar for estimating the difference in sensitivities and in specificities, and both achieve a substantial amount of variance reduction with an over-sample of subjects with discordant results and under-sample of subjects with concordant results. A heuristic rule is recommended when there is no prior knowledge of individual sensitivities and specificities, or the prevalence of the true positive findings in the study population. The optimal sampling is applied to a real-world example in record linkage to evaluate the difference in classification accuracy of two matching algorithms.

KW - Diagnostic accuracy

KW - Diagnostic test

KW - Positive predicted value

KW - Record linkage

KW - Sensitivity

KW - Specificity

KW - Stratified sampling

UR - http://www.scopus.com/inward/record.url?scp=84891836780&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84891836780&partnerID=8YFLogxK

U2 - 10.1002/sim.5946

DO - 10.1002/sim.5946

M3 - Article

VL - 33

SP - 500

EP - 513

JO - Statistics in Medicine

JF - Statistics in Medicine

SN - 0277-6715

IS - 3

ER -