Ranking Surgical Residency Programs: Reputation Survey or Outcomes Measures?

Adam B. Wilson, Laura J. Torbeck, Gary Dunnington

Research output: Contribution to journalArticle

10 Citations (Scopus)

Abstract

Objective: The release of general surgery residency program rankings by Doximity and U.S. News & World Report accentuates the need to define and establish measurable standards of program quality. This study evaluated the extent to which program rankings based solely on peer nominations correlated with familiar program outcomes measures. Design: Publicly available data were collected for all 254 general surgery residency programs. To generate a rudimentary outcomes-based program ranking, surgery programs were rank-ordered according to an average percentile rank that was calculated using board pass rates and the prevalence of alumni publications. A Kendall τ-b rank correlation computed the linear association between program rankings based on reputation alone and those derived from outcomes measures to validate whether reputation was a reasonable surrogate for globally judging program quality. Results: For the 218 programs with complete data eligible for analysis, the mean board pass rate was 72% with a standard deviation of 14%. A total of 60 programs were placed in the 75th percentile or above for the number of publications authored by program alumni. The correlational analysis reported a significant correlation of 0.428, indicating only a moderate association between programs ranked by outcomes measures and those ranked according to reputation. Seventeen programs that were ranked in the top 30 according to reputation were also ranked in the top 30 based on outcomes measures. Conclusions: This study suggests that reputation alone does not fully capture a representative snapshot of a program's quality. Rather, the use of multiple quantifiable indicators and attributes unique to programs ought to be given more consideration when assigning ranks to denote program quality. It is advised that the interpretation and subsequent use of program rankings be met with caution until further studies can rigorously demonstrate best practices for awarding program standings.

Original languageEnglish
JournalJournal of Surgical Education
DOIs
StateAccepted/In press - 2015

Fingerprint

Internship and Residency
reputation
ranking
Outcome Assessment (Health Care)
Publications
Practice Guidelines
Surveys and Questionnaires
surgery
alumni

Keywords

  • Outcomes
  • Program quality
  • Program rankings
  • Reputation
  • Surgery rankings
  • Xxx
  • Xxx
  • Xxx

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

Ranking Surgical Residency Programs : Reputation Survey or Outcomes Measures? / Wilson, Adam B.; Torbeck, Laura J.; Dunnington, Gary.

In: Journal of Surgical Education, 2015.

Research output: Contribution to journalArticle

@article{493f002c3b6b47a7949495a4dbb92df3,
title = "Ranking Surgical Residency Programs: Reputation Survey or Outcomes Measures?",
abstract = "Objective: The release of general surgery residency program rankings by Doximity and U.S. News & World Report accentuates the need to define and establish measurable standards of program quality. This study evaluated the extent to which program rankings based solely on peer nominations correlated with familiar program outcomes measures. Design: Publicly available data were collected for all 254 general surgery residency programs. To generate a rudimentary outcomes-based program ranking, surgery programs were rank-ordered according to an average percentile rank that was calculated using board pass rates and the prevalence of alumni publications. A Kendall τ-b rank correlation computed the linear association between program rankings based on reputation alone and those derived from outcomes measures to validate whether reputation was a reasonable surrogate for globally judging program quality. Results: For the 218 programs with complete data eligible for analysis, the mean board pass rate was 72{\%} with a standard deviation of 14{\%}. A total of 60 programs were placed in the 75th percentile or above for the number of publications authored by program alumni. The correlational analysis reported a significant correlation of 0.428, indicating only a moderate association between programs ranked by outcomes measures and those ranked according to reputation. Seventeen programs that were ranked in the top 30 according to reputation were also ranked in the top 30 based on outcomes measures. Conclusions: This study suggests that reputation alone does not fully capture a representative snapshot of a program's quality. Rather, the use of multiple quantifiable indicators and attributes unique to programs ought to be given more consideration when assigning ranks to denote program quality. It is advised that the interpretation and subsequent use of program rankings be met with caution until further studies can rigorously demonstrate best practices for awarding program standings.",
keywords = "Outcomes, Program quality, Program rankings, Reputation, Surgery rankings, Xxx, Xxx, Xxx",
author = "Wilson, {Adam B.} and Torbeck, {Laura J.} and Gary Dunnington",
year = "2015",
doi = "10.1016/j.jsurg.2015.03.021",
language = "English",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - Ranking Surgical Residency Programs

T2 - Reputation Survey or Outcomes Measures?

AU - Wilson, Adam B.

AU - Torbeck, Laura J.

AU - Dunnington, Gary

PY - 2015

Y1 - 2015

N2 - Objective: The release of general surgery residency program rankings by Doximity and U.S. News & World Report accentuates the need to define and establish measurable standards of program quality. This study evaluated the extent to which program rankings based solely on peer nominations correlated with familiar program outcomes measures. Design: Publicly available data were collected for all 254 general surgery residency programs. To generate a rudimentary outcomes-based program ranking, surgery programs were rank-ordered according to an average percentile rank that was calculated using board pass rates and the prevalence of alumni publications. A Kendall τ-b rank correlation computed the linear association between program rankings based on reputation alone and those derived from outcomes measures to validate whether reputation was a reasonable surrogate for globally judging program quality. Results: For the 218 programs with complete data eligible for analysis, the mean board pass rate was 72% with a standard deviation of 14%. A total of 60 programs were placed in the 75th percentile or above for the number of publications authored by program alumni. The correlational analysis reported a significant correlation of 0.428, indicating only a moderate association between programs ranked by outcomes measures and those ranked according to reputation. Seventeen programs that were ranked in the top 30 according to reputation were also ranked in the top 30 based on outcomes measures. Conclusions: This study suggests that reputation alone does not fully capture a representative snapshot of a program's quality. Rather, the use of multiple quantifiable indicators and attributes unique to programs ought to be given more consideration when assigning ranks to denote program quality. It is advised that the interpretation and subsequent use of program rankings be met with caution until further studies can rigorously demonstrate best practices for awarding program standings.

AB - Objective: The release of general surgery residency program rankings by Doximity and U.S. News & World Report accentuates the need to define and establish measurable standards of program quality. This study evaluated the extent to which program rankings based solely on peer nominations correlated with familiar program outcomes measures. Design: Publicly available data were collected for all 254 general surgery residency programs. To generate a rudimentary outcomes-based program ranking, surgery programs were rank-ordered according to an average percentile rank that was calculated using board pass rates and the prevalence of alumni publications. A Kendall τ-b rank correlation computed the linear association between program rankings based on reputation alone and those derived from outcomes measures to validate whether reputation was a reasonable surrogate for globally judging program quality. Results: For the 218 programs with complete data eligible for analysis, the mean board pass rate was 72% with a standard deviation of 14%. A total of 60 programs were placed in the 75th percentile or above for the number of publications authored by program alumni. The correlational analysis reported a significant correlation of 0.428, indicating only a moderate association between programs ranked by outcomes measures and those ranked according to reputation. Seventeen programs that were ranked in the top 30 according to reputation were also ranked in the top 30 based on outcomes measures. Conclusions: This study suggests that reputation alone does not fully capture a representative snapshot of a program's quality. Rather, the use of multiple quantifiable indicators and attributes unique to programs ought to be given more consideration when assigning ranks to denote program quality. It is advised that the interpretation and subsequent use of program rankings be met with caution until further studies can rigorously demonstrate best practices for awarding program standings.

KW - Outcomes

KW - Program quality

KW - Program rankings

KW - Reputation

KW - Surgery rankings

KW - Xxx

KW - Xxx

KW - Xxx

UR - http://www.scopus.com/inward/record.url?scp=84929224135&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84929224135&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2015.03.021

DO - 10.1016/j.jsurg.2015.03.021

M3 - Article

C2 - 25980829

AN - SCOPUS:84929224135

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

ER -