How Do You Tell Whether a Change in Surgical Technique Leads to a Change in Outcome?

Andrew J. Vickers, Angel M. Cronin, Timothy Masterson, James A. Eastham

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Purpose: Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies. Materials and Methods: We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve. Results: In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance. Conclusions: Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.

Original languageEnglish (US)
Pages (from-to)1510-1514
Number of pages5
JournalJournal of Urology
Volume183
Issue number4
DOIs
StatePublished - Apr 2010
Externally publishedYes

Fingerprint

Learning Curve
Chi-Square Distribution
Research Personnel

Keywords

  • data interpretation
  • learning
  • prostate
  • prostatectomy
  • research design
  • statistical

ASJC Scopus subject areas

  • Urology

Cite this

How Do You Tell Whether a Change in Surgical Technique Leads to a Change in Outcome? / Vickers, Andrew J.; Cronin, Angel M.; Masterson, Timothy; Eastham, James A.

In: Journal of Urology, Vol. 183, No. 4, 04.2010, p. 1510-1514.

Research output: Contribution to journalArticle

Vickers, Andrew J. ; Cronin, Angel M. ; Masterson, Timothy ; Eastham, James A. / How Do You Tell Whether a Change in Surgical Technique Leads to a Change in Outcome?. In: Journal of Urology. 2010 ; Vol. 183, No. 4. pp. 1510-1514.
@article{563782bbf43e49e28ee9877f4f72f031,
title = "How Do You Tell Whether a Change in Surgical Technique Leads to a Change in Outcome?",
abstract = "Purpose: Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies. Materials and Methods: We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve. Results: In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10{\%} decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80{\%} of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance. Conclusions: Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.",
keywords = "data interpretation, learning, prostate, prostatectomy, research design, statistical",
author = "Vickers, {Andrew J.} and Cronin, {Angel M.} and Timothy Masterson and Eastham, {James A.}",
year = "2010",
month = "4",
doi = "10.1016/j.juro.2009.12.034",
language = "English (US)",
volume = "183",
pages = "1510--1514",
journal = "Journal of Urology",
issn = "0022-5347",
publisher = "Elsevier Inc.",
number = "4",

}

TY - JOUR

T1 - How Do You Tell Whether a Change in Surgical Technique Leads to a Change in Outcome?

AU - Vickers, Andrew J.

AU - Cronin, Angel M.

AU - Masterson, Timothy

AU - Eastham, James A.

PY - 2010/4

Y1 - 2010/4

N2 - Purpose: Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies. Materials and Methods: We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve. Results: In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance. Conclusions: Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.

AB - Purpose: Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies. Materials and Methods: We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve. Results: In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance. Conclusions: Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification.

KW - data interpretation

KW - learning

KW - prostate

KW - prostatectomy

KW - research design

KW - statistical

UR - http://www.scopus.com/inward/record.url?scp=77949334917&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77949334917&partnerID=8YFLogxK

U2 - 10.1016/j.juro.2009.12.034

DO - 10.1016/j.juro.2009.12.034

M3 - Article

VL - 183

SP - 1510

EP - 1514

JO - Journal of Urology

JF - Journal of Urology

SN - 0022-5347

IS - 4

ER -