Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents

Jennine L. Larson, Reed G. Williams, Janet Ketchum, Margaret L. Boehler, Gary L. Dunnington

Research output: Contribution to journalArticle

134 Scopus citations


Background. Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Methods. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Results. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Conclusions. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

Original languageEnglish (US)
Pages (from-to)640-649
Number of pages10
Issue number4
StatePublished - Oct 1 2005
Externally publishedYes

ASJC Scopus subject areas

  • Surgery

Fingerprint Dive into the research topics of 'Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents'. Together they form a unique fingerprint.

  • Cite this