A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters

Reed G. Williams, Steven Verhulst, Jerry A. Colliver, Hilary Sanfey, Xiaodong Chen, Gary L. Dunnington

Research output: Contribution to journalArticle

30 Scopus citations

Abstract

Background: Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results. Methods: 21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80). Results: Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%). Conclusion: Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.

Original languageEnglish (US)
Pages (from-to)517-527
Number of pages11
JournalSurgery (United States)
Volume152
Issue number4
DOIs
StatePublished - Oct 1 2012
Externally publishedYes

ASJC Scopus subject areas

  • Surgery

Fingerprint Dive into the research topics of 'A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters'. Together they form a unique fingerprint.

  • Cite this