Comparative study of heuristic evaluation and usability testing methods

Thankam Paul Thyvalikakath, Valerie Monaco, Himabindu Thambuganipalle, Titus Schleyer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Citations (Scopus)

Abstract

Usability methods, such as heuristic evaluation, cognitive walk-throughs and user testing, are increasingly used to evaluate and improve the design of clinical software applications. There is still some uncertainty, however, as to how those methods can be used to support the development process and evaluation in the most meaningful manner. In this study, we compared the results of a heuristic evaluation with those of formal user tests in order to determine which usability problems were detected by both methods. We conducted heuristic evaluation and usability testing on four major commercial dental computer-based patient records (CPRs), which together cover 80% of the market for chairside computer systems among general dentists. Both methods yielded strong evidence that the dental CPRs have significant usability problems. An average of 50% of empirically-determined usability problems were identified by the preceding heuristic evaluation. Some statements of heuristic violations were specific enough to precisely identify the actual usability problem that study participants encountered. Other violations were less specific, but still manifested themselves in usability problems and poor task outcomes. In this study, heuristic evaluation identified a significant portion of problems found during usability testing. While we make no assumptions about the generalizability of the results to other domains and software systems, heuristic evaluation may, under certain circumstances, be a useful tool to determine design problems early in the development cycle.

Original languageEnglish (US)
Title of host publicationStudies in Health Technology and Informatics
Pages322-327
Number of pages6
Volume143
DOIs
StatePublished - 2009
Externally publishedYes
Event2009 Conference on Advances in Information Technology and Communication in Health: Revolutionizing Health Care with Informatics - From Research to Practice, ITCH 2009 - Victoria, BC, Canada
Duration: Feb 19 2009Feb 22 2009

Other

Other2009 Conference on Advances in Information Technology and Communication in Health: Revolutionizing Health Care with Informatics - From Research to Practice, ITCH 2009
CountryCanada
CityVictoria, BC
Period2/19/092/22/09

Fingerprint

Testing
Application programs
Computer systems
Tooth
Software Design
Computer Systems
Heuristics
Dentists
Uncertainty
Software

Keywords

  • Dental CPR
  • Heuristic evaluation
  • Human-computer interaction
  • Usability testing

ASJC Scopus subject areas

  • Biomedical Engineering
  • Health Informatics
  • Health Information Management

Cite this

Thyvalikakath, T. P., Monaco, V., Thambuganipalle, H., & Schleyer, T. (2009). Comparative study of heuristic evaluation and usability testing methods. In Studies in Health Technology and Informatics (Vol. 143, pp. 322-327) https://doi.org/10.3233/978-1-58603-979-0-322

Comparative study of heuristic evaluation and usability testing methods. / Thyvalikakath, Thankam Paul; Monaco, Valerie; Thambuganipalle, Himabindu; Schleyer, Titus.

Studies in Health Technology and Informatics. Vol. 143 2009. p. 322-327.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Thyvalikakath, TP, Monaco, V, Thambuganipalle, H & Schleyer, T 2009, Comparative study of heuristic evaluation and usability testing methods. in Studies in Health Technology and Informatics. vol. 143, pp. 322-327, 2009 Conference on Advances in Information Technology and Communication in Health: Revolutionizing Health Care with Informatics - From Research to Practice, ITCH 2009, Victoria, BC, Canada, 2/19/09. https://doi.org/10.3233/978-1-58603-979-0-322
Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Comparative study of heuristic evaluation and usability testing methods. In Studies in Health Technology and Informatics. Vol. 143. 2009. p. 322-327 https://doi.org/10.3233/978-1-58603-979-0-322
Thyvalikakath, Thankam Paul ; Monaco, Valerie ; Thambuganipalle, Himabindu ; Schleyer, Titus. / Comparative study of heuristic evaluation and usability testing methods. Studies in Health Technology and Informatics. Vol. 143 2009. pp. 322-327
@inproceedings{8e2f9b9bc680455bb66078b67cd3fd75,
title = "Comparative study of heuristic evaluation and usability testing methods",
abstract = "Usability methods, such as heuristic evaluation, cognitive walk-throughs and user testing, are increasingly used to evaluate and improve the design of clinical software applications. There is still some uncertainty, however, as to how those methods can be used to support the development process and evaluation in the most meaningful manner. In this study, we compared the results of a heuristic evaluation with those of formal user tests in order to determine which usability problems were detected by both methods. We conducted heuristic evaluation and usability testing on four major commercial dental computer-based patient records (CPRs), which together cover 80{\%} of the market for chairside computer systems among general dentists. Both methods yielded strong evidence that the dental CPRs have significant usability problems. An average of 50{\%} of empirically-determined usability problems were identified by the preceding heuristic evaluation. Some statements of heuristic violations were specific enough to precisely identify the actual usability problem that study participants encountered. Other violations were less specific, but still manifested themselves in usability problems and poor task outcomes. In this study, heuristic evaluation identified a significant portion of problems found during usability testing. While we make no assumptions about the generalizability of the results to other domains and software systems, heuristic evaluation may, under certain circumstances, be a useful tool to determine design problems early in the development cycle.",
keywords = "Dental CPR, Heuristic evaluation, Human-computer interaction, Usability testing",
author = "Thyvalikakath, {Thankam Paul} and Valerie Monaco and Himabindu Thambuganipalle and Titus Schleyer",
year = "2009",
doi = "10.3233/978-1-58603-979-0-322",
language = "English (US)",
isbn = "9781586039790",
volume = "143",
pages = "322--327",
booktitle = "Studies in Health Technology and Informatics",

}

TY - GEN

T1 - Comparative study of heuristic evaluation and usability testing methods

AU - Thyvalikakath, Thankam Paul

AU - Monaco, Valerie

AU - Thambuganipalle, Himabindu

AU - Schleyer, Titus

PY - 2009

Y1 - 2009

N2 - Usability methods, such as heuristic evaluation, cognitive walk-throughs and user testing, are increasingly used to evaluate and improve the design of clinical software applications. There is still some uncertainty, however, as to how those methods can be used to support the development process and evaluation in the most meaningful manner. In this study, we compared the results of a heuristic evaluation with those of formal user tests in order to determine which usability problems were detected by both methods. We conducted heuristic evaluation and usability testing on four major commercial dental computer-based patient records (CPRs), which together cover 80% of the market for chairside computer systems among general dentists. Both methods yielded strong evidence that the dental CPRs have significant usability problems. An average of 50% of empirically-determined usability problems were identified by the preceding heuristic evaluation. Some statements of heuristic violations were specific enough to precisely identify the actual usability problem that study participants encountered. Other violations were less specific, but still manifested themselves in usability problems and poor task outcomes. In this study, heuristic evaluation identified a significant portion of problems found during usability testing. While we make no assumptions about the generalizability of the results to other domains and software systems, heuristic evaluation may, under certain circumstances, be a useful tool to determine design problems early in the development cycle.

AB - Usability methods, such as heuristic evaluation, cognitive walk-throughs and user testing, are increasingly used to evaluate and improve the design of clinical software applications. There is still some uncertainty, however, as to how those methods can be used to support the development process and evaluation in the most meaningful manner. In this study, we compared the results of a heuristic evaluation with those of formal user tests in order to determine which usability problems were detected by both methods. We conducted heuristic evaluation and usability testing on four major commercial dental computer-based patient records (CPRs), which together cover 80% of the market for chairside computer systems among general dentists. Both methods yielded strong evidence that the dental CPRs have significant usability problems. An average of 50% of empirically-determined usability problems were identified by the preceding heuristic evaluation. Some statements of heuristic violations were specific enough to precisely identify the actual usability problem that study participants encountered. Other violations were less specific, but still manifested themselves in usability problems and poor task outcomes. In this study, heuristic evaluation identified a significant portion of problems found during usability testing. While we make no assumptions about the generalizability of the results to other domains and software systems, heuristic evaluation may, under certain circumstances, be a useful tool to determine design problems early in the development cycle.

KW - Dental CPR

KW - Heuristic evaluation

KW - Human-computer interaction

KW - Usability testing

UR - http://www.scopus.com/inward/record.url?scp=67449091041&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=67449091041&partnerID=8YFLogxK

U2 - 10.3233/978-1-58603-979-0-322

DO - 10.3233/978-1-58603-979-0-322

M3 - Conference contribution

C2 - 19380955

AN - SCOPUS:67449091041

SN - 9781586039790

VL - 143

SP - 322

EP - 327

BT - Studies in Health Technology and Informatics

ER -