Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test

Adam B. Wilson, Gary R. Pike, Aloysius Humbert

Research output: Contribution to journalArticle

1 Scopus citations

Abstract

Background: To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods: This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results: Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions: In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.

Original languageEnglish (US)
Pages (from-to)439-446
Number of pages8
JournalMedical Science Educator
Volume25
Issue number4
DOIs
StatePublished - Dec 1 2015

Keywords

  • Clinical reasoning
  • Psychometrics
  • Script concordance test

ASJC Scopus subject areas

  • Medicine (miscellaneous)
  • Education

Fingerprint Dive into the research topics of 'Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test'. Together they form a unique fingerprint.

  • Cite this