Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test

Adam B. Wilson, Gary R. Pike, Aloysius Humbert

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Background: To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods: This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results: Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions: In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.

Original languageEnglish (US)
Pages (from-to)439-446
Number of pages8
JournalMedical Science Educator
Volume25
Issue number4
DOIs
StatePublished - Dec 1 2015

Fingerprint

Statistical Factor Analysis
factor analysis
Research Design
Medical Students
Research
medical student
data analysis

Keywords

  • Clinical reasoning
  • Psychometrics
  • Script concordance test

ASJC Scopus subject areas

  • Medicine (miscellaneous)
  • Education

Cite this

Testing for Multilevel Dimensionality : A Higher-Order Factor Analysis of a Script Concordance Test. / Wilson, Adam B.; Pike, Gary R.; Humbert, Aloysius.

In: Medical Science Educator, Vol. 25, No. 4, 01.12.2015, p. 439-446.

Research output: Contribution to journalArticle

@article{a38b51c66f7e4715969ee57d253fa20e,
title = "Testing for Multilevel Dimensionality: A Higher-Order Factor Analysis of a Script Concordance Test",
abstract = "Background: To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods: This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results: Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 {\%}. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions: In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.",
keywords = "Clinical reasoning, Psychometrics, Script concordance test",
author = "Wilson, {Adam B.} and Pike, {Gary R.} and Aloysius Humbert",
year = "2015",
month = "12",
day = "1",
doi = "10.1007/s40670-015-0178-7",
language = "English (US)",
volume = "25",
pages = "439--446",
journal = "Medical Science Educator",
issn = "2156-8650",
publisher = "Springer New York",
number = "4",

}

TY - JOUR

T1 - Testing for Multilevel Dimensionality

T2 - A Higher-Order Factor Analysis of a Script Concordance Test

AU - Wilson, Adam B.

AU - Pike, Gary R.

AU - Humbert, Aloysius

PY - 2015/12/1

Y1 - 2015/12/1

N2 - Background: To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods: This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results: Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions: In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.

AB - Background: To date, few script concordance test (SCT) research studies have empirically investigated whether SCTs measure a single test dimension of clinical reasoning. Prior analyses have been inconclusive and unsuccessful at identifying exam dimensions through simple first-order factor analyses. Therefore, the principle objective of this work was to explore the higher-order factor structure of a problem-solving SCT to determine whether the dimensionality of the test conformed to a multilevel construct arrangement. Methods: This retrospective data analysis utilized scores from medical students (n = 522) who took a SCT in their fourth year of undergraduate training. Higher-order factor analyses were conducted on six different SCT scoring methods. In addition, Schmid-Leiman solutions were conducted to evaluate the proportion of variance unique to a given level of the model. Results: Five of the six scoring methods yielded a factor structure with three first-order factors and one second-order factor. The total variance in the first-order factors explained by the second-order factor was notable at greater than 40 %. However, Schmid-Leiman solutions unveiled the deceptiveness of the higher-order models in that, across the various scoring methods, very few items contributed unique variance to the second- or first-order factors. Ideally, all items would have contributed unique variance to one or more levels of the model. Conclusions: In this study, SCT scores did not conform to a clear interpretable higher-order factor structure suggesting that SCTs may not measure the meaningful clinical reasoning constructs they are thought to measure. An explanation for these findings is provided, and recommendations for revising the SCT format and scoring procedures are proposed.

KW - Clinical reasoning

KW - Psychometrics

KW - Script concordance test

UR - http://www.scopus.com/inward/record.url?scp=85024830788&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85024830788&partnerID=8YFLogxK

U2 - 10.1007/s40670-015-0178-7

DO - 10.1007/s40670-015-0178-7

M3 - Article

AN - SCOPUS:85024830788

VL - 25

SP - 439

EP - 446

JO - Medical Science Educator

JF - Medical Science Educator

SN - 2156-8650

IS - 4

ER -