Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

Valerie J. Lang, Norman B. Berman, Kirk Bronander, Heather Harrell, Susan Hingle, Amy Holthouser, Debra Leizman, Clifford D. Packer, Yoon Soo Park, Toan Vu, Rachel Yudkowsky, Sandra Monteiro, Georges Bordage

Research output: Contribution to journalArticle

Abstract

PURPOSE: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. METHOD: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. RESULTS: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). CONCLUSIONS: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

Original languageEnglish (US)
Pages (from-to)259-266
Number of pages8
JournalAcademic medicine : journal of the Association of American Medical Colleges
Volume94
Issue number2
DOIs
StatePublished - Feb 1 2019

    Fingerprint

ASJC Scopus subject areas

  • Education

Cite this

Lang, V. J., Berman, N. B., Bronander, K., Harrell, H., Hingle, S., Holthouser, A., Leizman, D., Packer, C. D., Park, Y. S., Vu, T., Yudkowsky, R., Monteiro, S., & Bordage, G. (2019). Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship. Academic medicine : journal of the Association of American Medical Colleges, 94(2), 259-266. https://doi.org/10.1097/ACM.0000000000002506