Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents

Jennine L. Larson, Reed G. Williams, Janet Ketchum, Margaret L. Boehler, Gary Dunnington

Research output: Contribution to journalArticle

108 Citations (Scopus)

Abstract

Background. Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Methods. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Results. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Conclusions. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

Original languageEnglish (US)
Pages (from-to)640-649
Number of pages10
JournalSurgery
Volume138
Issue number4
DOIs
StatePublished - Oct 2005
Externally publishedYes

Fingerprint

Reproducibility of Results
Internet
Decision Making
Parathyroidectomy
Segmental Mastectomy
Groin
Laparoscopic Cholecystectomy
Volunteers
Colon
Biopsy

ASJC Scopus subject areas

  • Surgery

Cite this

Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. / Larson, Jennine L.; Williams, Reed G.; Ketchum, Janet; Boehler, Margaret L.; Dunnington, Gary.

In: Surgery, Vol. 138, No. 4, 10.2005, p. 640-649.

Research output: Contribution to journalArticle

Larson, Jennine L. ; Williams, Reed G. ; Ketchum, Janet ; Boehler, Margaret L. ; Dunnington, Gary. / Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. In: Surgery. 2005 ; Vol. 138, No. 4. pp. 640-649.
@article{afe68d42f693475d8b334d881c863b75,
title = "Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents",
abstract = "Background. Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Methods. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Results. Return rates for the Internet-based form were full-time faculty (92{\%}), volunteer faculty (27{\%}), and overall (67{\%}). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25{\%} to 57{\%} of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Conclusions. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.",
author = "Larson, {Jennine L.} and Williams, {Reed G.} and Janet Ketchum and Boehler, {Margaret L.} and Gary Dunnington",
year = "2005",
month = "10",
doi = "10.1016/j.surg.2005.07.017",
language = "English (US)",
volume = "138",
pages = "640--649",
journal = "Surgery",
issn = "0039-6060",
publisher = "Mosby Inc.",
number = "4",

}

TY - JOUR

T1 - Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents

AU - Larson, Jennine L.

AU - Williams, Reed G.

AU - Ketchum, Janet

AU - Boehler, Margaret L.

AU - Dunnington, Gary

PY - 2005/10

Y1 - 2005/10

N2 - Background. Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Methods. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Results. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Conclusions. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

AB - Background. Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. Methods. Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. Results. Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. Conclusions. Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

UR - http://www.scopus.com/inward/record.url?scp=27544441243&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=27544441243&partnerID=8YFLogxK

U2 - 10.1016/j.surg.2005.07.017

DO - 10.1016/j.surg.2005.07.017

M3 - Article

VL - 138

SP - 640

EP - 649

JO - Surgery

JF - Surgery

SN - 0039-6060

IS - 4

ER -