Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

Valerie J. Lang, Norman B. Berman, Kirk Bronander, Heather Harrell, Susan Hingle, Amy Holthouser, Debra Leizman, Clifford D. Packer, Yoon Soo Park, T. Robert Vu, Rachel Yudkowsky, Sandra Monteiro, Georges Bordage

Research output: Contribution to journalArticle

Abstract

Purpose Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. Medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. Method From February 2012 through January 2013, 759 students (at eight U.S. Medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. Results Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P <.01). Conclusions These results provide strong evidence for response-process and relationship-To-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

Original languageEnglish (US)
Pages (from-to)259-266
Number of pages8
JournalAcademic Medicine
Volume94
Issue number2
DOIs
StatePublished - Feb 1 2019

ASJC Scopus subject areas

  • Education

Fingerprint Dive into the research topics of 'Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship'. Together they form a unique fingerprint.

  • Cite this

    Lang, V. J., Berman, N. B., Bronander, K., Harrell, H., Hingle, S., Holthouser, A., Leizman, D., Packer, C. D., Park, Y. S., Vu, T. R., Yudkowsky, R., Monteiro, S., & Bordage, G. (2019). Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship. Academic Medicine, 94(2), 259-266. https://doi.org/10.1097/ACM.0000000000002506