Measuring agreement between decision support reminders

The cloud vs. the local expert

Brian Dixon, Linas Simonaitis, Susan Perkins, Adam Wright, Blackford Middleton

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Background: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. Methods. Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. Results: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. Conclusions: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

Original languageEnglish
Article number31
JournalBMC Medical Informatics and Decision Making
Volume14
Issue number1
DOIs
StatePublished - Apr 10 2014

Fingerprint

Clinical Decision Support Systems
Preventive Medicine
Knowledge Management
Electronic Health Records
Terminology
Health Personnel
Primary Health Care
Guidelines
Health

Keywords

  • Clinical decision making
  • Computer-assisted knowledge management
  • Decision support systems
  • Preventive health services
  • Statistical data analysis

ASJC Scopus subject areas

  • Health Informatics
  • Health Policy

Cite this

Measuring agreement between decision support reminders : The cloud vs. the local expert. / Dixon, Brian; Simonaitis, Linas; Perkins, Susan; Wright, Adam; Middleton, Blackford.

In: BMC Medical Informatics and Decision Making, Vol. 14, No. 1, 31, 10.04.2014.

Research output: Contribution to journalArticle

@article{c640fb23614d4140a891536eee48479e,
title = "Measuring agreement between decision support reminders: The cloud vs. the local expert",
abstract = "Background: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. Methods. Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. Results: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95{\%} CI 0.24 - 0.42) to 0.99 (95{\%} CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. Conclusions: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.",
keywords = "Clinical decision making, Computer-assisted knowledge management, Decision support systems, Preventive health services, Statistical data analysis",
author = "Brian Dixon and Linas Simonaitis and Susan Perkins and Adam Wright and Blackford Middleton",
year = "2014",
month = "4",
day = "10",
doi = "10.1186/1472-6947-14-31",
language = "English",
volume = "14",
journal = "BMC Medical Informatics and Decision Making",
issn = "1472-6947",
publisher = "BioMed Central",
number = "1",

}

TY - JOUR

T1 - Measuring agreement between decision support reminders

T2 - The cloud vs. the local expert

AU - Dixon, Brian

AU - Simonaitis, Linas

AU - Perkins, Susan

AU - Wright, Adam

AU - Middleton, Blackford

PY - 2014/4/10

Y1 - 2014/4/10

N2 - Background: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. Methods. Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. Results: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. Conclusions: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

AB - Background: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. Methods. Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. Results: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. Conclusions: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

KW - Clinical decision making

KW - Computer-assisted knowledge management

KW - Decision support systems

KW - Preventive health services

KW - Statistical data analysis

UR - http://www.scopus.com/inward/record.url?scp=84899968208&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899968208&partnerID=8YFLogxK

U2 - 10.1186/1472-6947-14-31

DO - 10.1186/1472-6947-14-31

M3 - Article

VL - 14

JO - BMC Medical Informatics and Decision Making

JF - BMC Medical Informatics and Decision Making

SN - 1472-6947

IS - 1

M1 - 31

ER -