Assuring the reliability of resident performance appraisals

More items or more observations?

Reed G. Williams, Steven Verhulst, Jerry A. Colliver, Gary Dunnington

Research output: Contribution to journalArticle

35 Citations (Scopus)

Abstract

The tendency to add items to resident performance rating forms has accelerated due to new ACGME competency requirements. This study addresses the relative merits of adding items versus increasing number of observations. The specific questions addressed are (1) what is the reliability of single items used to assess resident performance, (2) what effect does adding items have on reliability, and (3) how many observations are required to obtain reliable resident performance ratings. Surgeon ratings of resident performance were collected for 3 years. The rating instrument had 3 single items representing clinical performance, professional behavior, and comparisons to other house staff. Reliability analyses were performed separately for each year, and variance components were pooled across years to compute overall reliability coefficients. Single-item resident performance rating scales were equivalent to multiple-item scales using conventional reliability standards. Increasing the number of rating items had little effect on reliability. Increasing the number of observations had a much larger effect. Program directors should focus on increasing the number of observations per resident to improve performance sampling and reliability of assessment. Increasing the number of rating items had little effect on reliability and is unlikely to assess new ACGME competencies adequately.

Original languageEnglish (US)
Pages (from-to)141-147
Number of pages7
JournalSurgery
Volume137
Issue number2
DOIs
StatePublished - Feb 2005
Externally publishedYes

Fingerprint

Internship and Residency
Surgeons

ASJC Scopus subject areas

  • Surgery

Cite this

Assuring the reliability of resident performance appraisals : More items or more observations? / Williams, Reed G.; Verhulst, Steven; Colliver, Jerry A.; Dunnington, Gary.

In: Surgery, Vol. 137, No. 2, 02.2005, p. 141-147.

Research output: Contribution to journalArticle

Williams, Reed G. ; Verhulst, Steven ; Colliver, Jerry A. ; Dunnington, Gary. / Assuring the reliability of resident performance appraisals : More items or more observations?. In: Surgery. 2005 ; Vol. 137, No. 2. pp. 141-147.
@article{49c60edb5cc64910aad09669f8c0313d,
title = "Assuring the reliability of resident performance appraisals: More items or more observations?",
abstract = "The tendency to add items to resident performance rating forms has accelerated due to new ACGME competency requirements. This study addresses the relative merits of adding items versus increasing number of observations. The specific questions addressed are (1) what is the reliability of single items used to assess resident performance, (2) what effect does adding items have on reliability, and (3) how many observations are required to obtain reliable resident performance ratings. Surgeon ratings of resident performance were collected for 3 years. The rating instrument had 3 single items representing clinical performance, professional behavior, and comparisons to other house staff. Reliability analyses were performed separately for each year, and variance components were pooled across years to compute overall reliability coefficients. Single-item resident performance rating scales were equivalent to multiple-item scales using conventional reliability standards. Increasing the number of rating items had little effect on reliability. Increasing the number of observations had a much larger effect. Program directors should focus on increasing the number of observations per resident to improve performance sampling and reliability of assessment. Increasing the number of rating items had little effect on reliability and is unlikely to assess new ACGME competencies adequately.",
author = "Williams, {Reed G.} and Steven Verhulst and Colliver, {Jerry A.} and Gary Dunnington",
year = "2005",
month = "2",
doi = "10.1016/j.surg.2004.06.011",
language = "English (US)",
volume = "137",
pages = "141--147",
journal = "Surgery",
issn = "0039-6060",
publisher = "Mosby Inc.",
number = "2",

}

TY - JOUR

T1 - Assuring the reliability of resident performance appraisals

T2 - More items or more observations?

AU - Williams, Reed G.

AU - Verhulst, Steven

AU - Colliver, Jerry A.

AU - Dunnington, Gary

PY - 2005/2

Y1 - 2005/2

N2 - The tendency to add items to resident performance rating forms has accelerated due to new ACGME competency requirements. This study addresses the relative merits of adding items versus increasing number of observations. The specific questions addressed are (1) what is the reliability of single items used to assess resident performance, (2) what effect does adding items have on reliability, and (3) how many observations are required to obtain reliable resident performance ratings. Surgeon ratings of resident performance were collected for 3 years. The rating instrument had 3 single items representing clinical performance, professional behavior, and comparisons to other house staff. Reliability analyses were performed separately for each year, and variance components were pooled across years to compute overall reliability coefficients. Single-item resident performance rating scales were equivalent to multiple-item scales using conventional reliability standards. Increasing the number of rating items had little effect on reliability. Increasing the number of observations had a much larger effect. Program directors should focus on increasing the number of observations per resident to improve performance sampling and reliability of assessment. Increasing the number of rating items had little effect on reliability and is unlikely to assess new ACGME competencies adequately.

AB - The tendency to add items to resident performance rating forms has accelerated due to new ACGME competency requirements. This study addresses the relative merits of adding items versus increasing number of observations. The specific questions addressed are (1) what is the reliability of single items used to assess resident performance, (2) what effect does adding items have on reliability, and (3) how many observations are required to obtain reliable resident performance ratings. Surgeon ratings of resident performance were collected for 3 years. The rating instrument had 3 single items representing clinical performance, professional behavior, and comparisons to other house staff. Reliability analyses were performed separately for each year, and variance components were pooled across years to compute overall reliability coefficients. Single-item resident performance rating scales were equivalent to multiple-item scales using conventional reliability standards. Increasing the number of rating items had little effect on reliability. Increasing the number of observations had a much larger effect. Program directors should focus on increasing the number of observations per resident to improve performance sampling and reliability of assessment. Increasing the number of rating items had little effect on reliability and is unlikely to assess new ACGME competencies adequately.

UR - http://www.scopus.com/inward/record.url?scp=12944324810&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=12944324810&partnerID=8YFLogxK

U2 - 10.1016/j.surg.2004.06.011

DO - 10.1016/j.surg.2004.06.011

M3 - Article

VL - 137

SP - 141

EP - 147

JO - Surgery

JF - Surgery

SN - 0039-6060

IS - 2

ER -