Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

Valerie J. Lang, Norman B. Berman, Kirk Bronander, Heather Harrell, Susan Hingle, Amy Holthouser, Debra Leizman, Clifford D. Packer, Yoon Soo Park, Toan Vu, Rachel Yudkowsky, Sandra Monteiro, Georges Bordage

Research output: Contribution to journalArticle

Abstract

PURPOSE: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. METHOD: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. RESULTS: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). CONCLUSIONS: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

Original languageEnglish (US)
Pages (from-to)259-266
Number of pages8
JournalAcademic medicine : journal of the Association of American Medical Colleges
Volume94
Issue number2
DOIs
StatePublished - Feb 1 2019

Fingerprint

medicine
examination
evidence
medical examiner
school
decision making
student
medical student
experience
educator
ability
interaction

ASJC Scopus subject areas

  • Education

Cite this

Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship. / Lang, Valerie J.; Berman, Norman B.; Bronander, Kirk; Harrell, Heather; Hingle, Susan; Holthouser, Amy; Leizman, Debra; Packer, Clifford D.; Park, Yoon Soo; Vu, Toan; Yudkowsky, Rachel; Monteiro, Sandra; Bordage, Georges.

In: Academic medicine : journal of the Association of American Medical Colleges, Vol. 94, No. 2, 01.02.2019, p. 259-266.

Research output: Contribution to journalArticle

Lang, VJ, Berman, NB, Bronander, K, Harrell, H, Hingle, S, Holthouser, A, Leizman, D, Packer, CD, Park, YS, Vu, T, Yudkowsky, R, Monteiro, S & Bordage, G 2019, 'Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship', Academic medicine : journal of the Association of American Medical Colleges, vol. 94, no. 2, pp. 259-266. https://doi.org/10.1097/ACM.0000000000002506
Lang, Valerie J. ; Berman, Norman B. ; Bronander, Kirk ; Harrell, Heather ; Hingle, Susan ; Holthouser, Amy ; Leizman, Debra ; Packer, Clifford D. ; Park, Yoon Soo ; Vu, Toan ; Yudkowsky, Rachel ; Monteiro, Sandra ; Bordage, Georges. / Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship. In: Academic medicine : journal of the Association of American Medical Colleges. 2019 ; Vol. 94, No. 2. pp. 259-266.
@article{5dd72498f7b1417e957c460d029f1936,
title = "Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship",
abstract = "PURPOSE: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. METHOD: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. RESULTS: Five hundred fifteen (67.9{\%}) students consented to participate. Across KFE forms, mean scores ranged from 54.6{\%} to 60.3{\%} (standard deviation 8.4-9.6{\%}), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). CONCLUSIONS: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.",
author = "Lang, {Valerie J.} and Berman, {Norman B.} and Kirk Bronander and Heather Harrell and Susan Hingle and Amy Holthouser and Debra Leizman and Packer, {Clifford D.} and Park, {Yoon Soo} and Toan Vu and Rachel Yudkowsky and Sandra Monteiro and Georges Bordage",
year = "2019",
month = "2",
day = "1",
doi = "10.1097/ACM.0000000000002506",
language = "English (US)",
volume = "94",
pages = "259--266",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "2",

}

TY - JOUR

T1 - Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship

AU - Lang, Valerie J.

AU - Berman, Norman B.

AU - Bronander, Kirk

AU - Harrell, Heather

AU - Hingle, Susan

AU - Holthouser, Amy

AU - Leizman, Debra

AU - Packer, Clifford D.

AU - Park, Yoon Soo

AU - Vu, Toan

AU - Yudkowsky, Rachel

AU - Monteiro, Sandra

AU - Bordage, Georges

PY - 2019/2/1

Y1 - 2019/2/1

N2 - PURPOSE: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. METHOD: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. RESULTS: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). CONCLUSIONS: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

AB - PURPOSE: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables. METHOD: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores. RESULTS: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01). CONCLUSIONS: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

UR - http://www.scopus.com/inward/record.url?scp=85060808299&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85060808299&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000002506

DO - 10.1097/ACM.0000000000002506

M3 - Article

C2 - 30379661

AN - SCOPUS:85060808299

VL - 94

SP - 259

EP - 266

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 2

ER -