Refining the Evaluation of Operating Room Performance

Michael J. Kim, Reed G. Williams, Margaret L. Boehler, Janet K. Ketchum, Gary Dunnington

Research output: Contribution to journalArticle

44 Citations (Scopus)

Abstract

Purpose: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95% confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95% CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures. Conclusions: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.

Original languageEnglish (US)
Pages (from-to)352-356
Number of pages5
JournalJournal of Surgical Education
Volume66
Issue number6
DOIs
StatePublished - Nov 2009
Externally publishedYes

Fingerprint

Operating Rooms
rating
evaluation
performance
resident
Confidence Intervals
Colectomy
Inguinal Hernia
Laparoscopic Cholecystectomy
confidence
Biopsy
Surgeons
examination

Keywords

  • clinical competence
  • education
  • educational measurement
  • evaluation
  • Medical Knowledge
  • Patient Care
  • performance ratings
  • Practice-Based Learning and Improvement

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

Refining the Evaluation of Operating Room Performance. / Kim, Michael J.; Williams, Reed G.; Boehler, Margaret L.; Ketchum, Janet K.; Dunnington, Gary.

In: Journal of Surgical Education, Vol. 66, No. 6, 11.2009, p. 352-356.

Research output: Contribution to journalArticle

Kim, Michael J. ; Williams, Reed G. ; Boehler, Margaret L. ; Ketchum, Janet K. ; Dunnington, Gary. / Refining the Evaluation of Operating Room Performance. In: Journal of Surgical Education. 2009 ; Vol. 66, No. 6. pp. 352-356.
@article{24da2a9342fe4048acf8ec86b440e954,
title = "Refining the Evaluation of Operating Room Performance",
abstract = "Purpose: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95{\%} confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95{\%} CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures. Conclusions: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.",
keywords = "clinical competence, education, educational measurement, evaluation, Medical Knowledge, Patient Care, performance ratings, Practice-Based Learning and Improvement",
author = "Kim, {Michael J.} and Williams, {Reed G.} and Boehler, {Margaret L.} and Ketchum, {Janet K.} and Gary Dunnington",
year = "2009",
month = "11",
doi = "10.1016/j.jsurg.2009.09.005",
language = "English (US)",
volume = "66",
pages = "352--356",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "6",

}

TY - JOUR

T1 - Refining the Evaluation of Operating Room Performance

AU - Kim, Michael J.

AU - Williams, Reed G.

AU - Boehler, Margaret L.

AU - Ketchum, Janet K.

AU - Dunnington, Gary

PY - 2009/11

Y1 - 2009/11

N2 - Purpose: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95% confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95% CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures. Conclusions: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.

AB - Purpose: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results. Methods: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. Results: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95% confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95% CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures. Conclusions: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.

KW - clinical competence

KW - education

KW - educational measurement

KW - evaluation

KW - Medical Knowledge

KW - Patient Care

KW - performance ratings

KW - Practice-Based Learning and Improvement

UR - http://www.scopus.com/inward/record.url?scp=75149180084&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=75149180084&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2009.09.005

DO - 10.1016/j.jsurg.2009.09.005

M3 - Article

C2 - 20142134

AN - SCOPUS:75149180084

VL - 66

SP - 352

EP - 356

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 6

ER -