Measuring agreement between decision support reminders: The cloud vs. the local expert

Brian Edward Dixon, Linas Simonaitis, Susan M. Perkins, Adam Wright, Blackford Middleton

Research output: Contribution to journalArticle

7 Scopus citations


Background: A cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data. Methods. Electronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen's Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients. Results: The cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 - 0.42) to 0.99 (95% CI 0.97 - 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders. Conclusions: Preventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

Original languageEnglish (US)
Article number31
JournalBMC medical informatics and decision making
Issue number1
StatePublished - Apr 10 2014


  • Clinical decision making
  • Computer-assisted knowledge management
  • Decision support systems
  • Preventive health services
  • Statistical data analysis

ASJC Scopus subject areas

  • Health Informatics
  • Health Policy

Fingerprint Dive into the research topics of 'Measuring agreement between decision support reminders: The cloud vs. the local expert'. Together they form a unique fingerprint.

  • Cite this