A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment: A controlled prospective observational study

Reed G. Williams, Hilary Sanfey, Xiaodong Phoenix Chen, Gary Dunnington

Research output: Contribution to journalArticle

32 Citations (Scopus)

Abstract

Objective: This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment. Background: Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training. Methods: Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation. Results: A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances. Conclusions: OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.

Original languageEnglish (US)
Pages (from-to)177-187
Number of pages11
JournalAnnals of Surgery
Volume256
Issue number1
DOIs
StatePublished - Jul 2012
Externally publishedYes

Fingerprint

Observational Studies
Prospective Studies
Observation
Internship and Residency
Reproducibility of Results
Exercise
Surgeons

ASJC Scopus subject areas

  • Surgery

Cite this

A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment : A controlled prospective observational study. / Williams, Reed G.; Sanfey, Hilary; Chen, Xiaodong Phoenix; Dunnington, Gary.

In: Annals of Surgery, Vol. 256, No. 1, 07.2012, p. 177-187.

Research output: Contribution to journalArticle

@article{2d437bf9429c4c60a92980fd3168283d,
title = "A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment: A controlled prospective observational study",
abstract = "Objective: This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment. Background: Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training. Methods: Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation. Results: A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances. Conclusions: OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a {"}Good{"} performance whereas those for PGY 5 residents corresponded to a {"}Very Good{"} performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.",
author = "Williams, {Reed G.} and Hilary Sanfey and Chen, {Xiaodong Phoenix} and Gary Dunnington",
year = "2012",
month = "7",
doi = "10.1097/SLA.0b013e31825b6de4",
language = "English (US)",
volume = "256",
pages = "177--187",
journal = "Annals of Surgery",
issn = "0003-4932",
publisher = "Lippincott Williams and Wilkins",
number = "1",

}

TY - JOUR

T1 - A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment

T2 - A controlled prospective observational study

AU - Williams, Reed G.

AU - Sanfey, Hilary

AU - Chen, Xiaodong Phoenix

AU - Dunnington, Gary

PY - 2012/7

Y1 - 2012/7

N2 - Objective: This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment. Background: Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training. Methods: Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation. Results: A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances. Conclusions: OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.

AB - Objective: This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment. Background: Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training. Methods: Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation. Results: A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances. Conclusions: OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.

UR - http://www.scopus.com/inward/record.url?scp=84863321531&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863321531&partnerID=8YFLogxK

U2 - 10.1097/SLA.0b013e31825b6de4

DO - 10.1097/SLA.0b013e31825b6de4

M3 - Article

C2 - 22751518

AN - SCOPUS:84863321531

VL - 256

SP - 177

EP - 187

JO - Annals of Surgery

JF - Annals of Surgery

SN - 0003-4932

IS - 1

ER -