A template for reliable assessment of resident operative performance

Assessment intervals, numbers of cases and raters

Reed G. Williams, Steven Verhulst, Jerry A. Colliver, Hilary Sanfey, Xiaodong Chen, Gary Dunnington

Research output: Contribution to journalArticle

27 Citations (Scopus)

Abstract

Background: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. Methods: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). Results: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%). Conclusion: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.

Original languageEnglish (US)
Pages (from-to)517-527
Number of pages11
JournalSurgery
Volume152
Issue number4
DOIs
StatePublished - Oct 2012
Externally publishedYes

Fingerprint

Decision Support Techniques
Analysis of Variance
Surgeons

ASJC Scopus subject areas

  • Surgery

Cite this

A template for reliable assessment of resident operative performance : Assessment intervals, numbers of cases and raters. / Williams, Reed G.; Verhulst, Steven; Colliver, Jerry A.; Sanfey, Hilary; Chen, Xiaodong; Dunnington, Gary.

In: Surgery, Vol. 152, No. 4, 10.2012, p. 517-527.

Research output: Contribution to journalArticle

Williams, Reed G. ; Verhulst, Steven ; Colliver, Jerry A. ; Sanfey, Hilary ; Chen, Xiaodong ; Dunnington, Gary. / A template for reliable assessment of resident operative performance : Assessment intervals, numbers of cases and raters. In: Surgery. 2012 ; Vol. 152, No. 4. pp. 517-527.
@article{36be22c90502494da112a32fed0fc107,
title = "A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters",
abstract = "Background: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. Methods: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). Results: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63{\%} when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12{\%}). Procedure was a less important source (5{\%}). Conclusion: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.",
author = "Williams, {Reed G.} and Steven Verhulst and Colliver, {Jerry A.} and Hilary Sanfey and Xiaodong Chen and Gary Dunnington",
year = "2012",
month = "10",
doi = "10.1016/j.surg.2012.07.004",
language = "English (US)",
volume = "152",
pages = "517--527",
journal = "Surgery",
issn = "0039-6060",
publisher = "Mosby Inc.",
number = "4",

}

TY - JOUR

T1 - A template for reliable assessment of resident operative performance

T2 - Assessment intervals, numbers of cases and raters

AU - Williams, Reed G.

AU - Verhulst, Steven

AU - Colliver, Jerry A.

AU - Sanfey, Hilary

AU - Chen, Xiaodong

AU - Dunnington, Gary

PY - 2012/10

Y1 - 2012/10

N2 - Background: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. Methods: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). Results: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%). Conclusion: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.

AB - Background: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. Methods: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). Results: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%). Conclusion: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.

UR - http://www.scopus.com/inward/record.url?scp=84867065413&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867065413&partnerID=8YFLogxK

U2 - 10.1016/j.surg.2012.07.004

DO - 10.1016/j.surg.2012.07.004

M3 - Article

VL - 152

SP - 517

EP - 527

JO - Surgery

JF - Surgery

SN - 0039-6060

IS - 4

ER -