Examining rater and occasion influences in observational assessments obtained from within the clinical environment

Clarence D. Kreiter, Adam B. Wilson, Aloysius J. Humbert, Patricia A. Wade

Research output: Contribution to journalArticle

5 Scopus citations


Background:When ratings of student performance within the clerkship consist of a variable number of ratings per clinical teacher (rater), an important measurement question arises regarding how to combine such ratings to accurately summarize performance. As previous G studies have not estimated the independent influence of occasion and rater facets in observational ratings within the clinic, this study was designed to provide estimates of these two sources of error. Method: During 2 years of an emergency medicine clerkship at a large midwestern university, 592 students were evaluated an average of 15.9 times. Ratings were performed at the end of clinical shifts, and students often received multiple ratings from the same rater. A completely nested G study model (occasion: rater: person) was used to analyze sampled rating data. Results: The variance component (VC) related to occasion was small relative to the VC associated with rater. The D study clearly demonstrates that having a preceptor rate a student on multiple occasions does not substantially enhance the reliability of a clerkship performance summary score. Conclusions: Although further research is needed, it is clear that case-specific factors do not explain the low correlation between ratings and that having one or two raters repeatedly rate a student on different occasions/ cases is unlikely to yield a reliable mean score. This research suggests that it may be more efficient to have a preceptor rate a student just once. However, when multiple ratings from a single preceptor are available for a student, it is recommended that a mean of the preceptor's ratings be used to calculate the student's overall mean performance score.

Original languageEnglish (US)
Article number29279
JournalMedical Education Online
Issue number1
StatePublished - 2016


  • Clinical ratings
  • Clinical skills
  • Generalizability theory

ASJC Scopus subject areas

  • Education

Fingerprint Dive into the research topics of 'Examining rater and occasion influences in observational assessments obtained from within the clinical environment'. Together they form a unique fingerprint.

  • Cite this