Can raters consistently evaluate the content of focus groups?

Morris Weinberger, Jeffrey A. Ferguson, Glenda Westmoreland, Lorrie A. Mamlin, Douglas S. Segar, George J. Eckert, James Y. Greene, Douglas Martin, William M. Tierney

Research output: Contribution to journalArticle

33 Citations (Scopus)

Abstract

Focus groups are increasingly being used to provide insights to researchers and policy makers. These data complement quantitative approaches to understanding the world. Unfortunately, quantitative and qualitative methodologies have often been viewed as antithetical, rather than complementary, strategies. While focus groups can clearly generate rich information that is unobtainable through other quantitative methods, it is important to determine the degree to which different raters can consistently extract information from transcripts. Thus, our goal was to quantify agreement in the interpretation of transcripts from patient and physician focus groups, using decision-making in ischemic heart disease as a model. We used data from focus groups with both patients and physicians that sought to identify factors affecting diagnostic and treatment decisions in ischemic heart disease. Three raters independently reviewed transcribed audiotapes from focus groups of patients with ischemic heart disease, as well as focus groups of physicians who care for these patients. We found that raters could not distinguish between major and minor factors reliably. More troubling, however, is that consistency regarding the apparently straightforward judgment as to the mere presence or absence of a factor was difficult to achieve. In particular, the three raters of each transcript failed to agree on between one third and one half of the factors. This reasonably high level of disagreement occurred despite the raters: (1) having generated the individual factors themselves based upon their reading a random sample of actual transcripts and (2) being trained in the use of rating forms (including standard definitions of themes). These data suggest that if a single rater evaluates focus group transcripts, as is commonly done, judgments may not be reproducible by other raters. Moreover; a single rater may nor extract all important information contained in the transcripts.

Original languageEnglish (US)
Pages (from-to)929-933
Number of pages5
JournalSocial Science and Medicine
Volume46
Issue number7
DOIs
StatePublished - Apr 1 1998
Externally publishedYes

Fingerprint

cardiovascular disease
Focus Groups
heart disease
Group
Myocardial Ischemia
physician
Physicians
decision making
Tape Recording
group decision
methodology
quantitative method
random sample
Focus groups
Raters
Administrative Personnel
diagnostic
rating
Reading
Decision Making

Keywords

  • Consistency of rating
  • Focus groups
  • Qualitative methods
  • Racial variation

ASJC Scopus subject areas

  • Economics and Econometrics
  • Public Health, Environmental and Occupational Health
  • Social Psychology
  • Development
  • Health(social science)

Cite this

Weinberger, M., Ferguson, J. A., Westmoreland, G., Mamlin, L. A., Segar, D. S., Eckert, G. J., ... Tierney, W. M. (1998). Can raters consistently evaluate the content of focus groups? Social Science and Medicine, 46(7), 929-933. https://doi.org/10.1016/S0277-9536(97)10028-4

Can raters consistently evaluate the content of focus groups? / Weinberger, Morris; Ferguson, Jeffrey A.; Westmoreland, Glenda; Mamlin, Lorrie A.; Segar, Douglas S.; Eckert, George J.; Greene, James Y.; Martin, Douglas; Tierney, William M.

In: Social Science and Medicine, Vol. 46, No. 7, 01.04.1998, p. 929-933.

Research output: Contribution to journalArticle

Weinberger, M, Ferguson, JA, Westmoreland, G, Mamlin, LA, Segar, DS, Eckert, GJ, Greene, JY, Martin, D & Tierney, WM 1998, 'Can raters consistently evaluate the content of focus groups?', Social Science and Medicine, vol. 46, no. 7, pp. 929-933. https://doi.org/10.1016/S0277-9536(97)10028-4
Weinberger M, Ferguson JA, Westmoreland G, Mamlin LA, Segar DS, Eckert GJ et al. Can raters consistently evaluate the content of focus groups? Social Science and Medicine. 1998 Apr 1;46(7):929-933. https://doi.org/10.1016/S0277-9536(97)10028-4
Weinberger, Morris ; Ferguson, Jeffrey A. ; Westmoreland, Glenda ; Mamlin, Lorrie A. ; Segar, Douglas S. ; Eckert, George J. ; Greene, James Y. ; Martin, Douglas ; Tierney, William M. / Can raters consistently evaluate the content of focus groups?. In: Social Science and Medicine. 1998 ; Vol. 46, No. 7. pp. 929-933.
@article{b448737666ff4ac6adf60591da026429,
title = "Can raters consistently evaluate the content of focus groups?",
abstract = "Focus groups are increasingly being used to provide insights to researchers and policy makers. These data complement quantitative approaches to understanding the world. Unfortunately, quantitative and qualitative methodologies have often been viewed as antithetical, rather than complementary, strategies. While focus groups can clearly generate rich information that is unobtainable through other quantitative methods, it is important to determine the degree to which different raters can consistently extract information from transcripts. Thus, our goal was to quantify agreement in the interpretation of transcripts from patient and physician focus groups, using decision-making in ischemic heart disease as a model. We used data from focus groups with both patients and physicians that sought to identify factors affecting diagnostic and treatment decisions in ischemic heart disease. Three raters independently reviewed transcribed audiotapes from focus groups of patients with ischemic heart disease, as well as focus groups of physicians who care for these patients. We found that raters could not distinguish between major and minor factors reliably. More troubling, however, is that consistency regarding the apparently straightforward judgment as to the mere presence or absence of a factor was difficult to achieve. In particular, the three raters of each transcript failed to agree on between one third and one half of the factors. This reasonably high level of disagreement occurred despite the raters: (1) having generated the individual factors themselves based upon their reading a random sample of actual transcripts and (2) being trained in the use of rating forms (including standard definitions of themes). These data suggest that if a single rater evaluates focus group transcripts, as is commonly done, judgments may not be reproducible by other raters. Moreover; a single rater may nor extract all important information contained in the transcripts.",
keywords = "Consistency of rating, Focus groups, Qualitative methods, Racial variation",
author = "Morris Weinberger and Ferguson, {Jeffrey A.} and Glenda Westmoreland and Mamlin, {Lorrie A.} and Segar, {Douglas S.} and Eckert, {George J.} and Greene, {James Y.} and Douglas Martin and Tierney, {William M.}",
year = "1998",
month = "4",
day = "1",
doi = "10.1016/S0277-9536(97)10028-4",
language = "English (US)",
volume = "46",
pages = "929--933",
journal = "Social Science and Medicine",
issn = "0277-9536",
publisher = "Elsevier Limited",
number = "7",

}

TY - JOUR

T1 - Can raters consistently evaluate the content of focus groups?

AU - Weinberger, Morris

AU - Ferguson, Jeffrey A.

AU - Westmoreland, Glenda

AU - Mamlin, Lorrie A.

AU - Segar, Douglas S.

AU - Eckert, George J.

AU - Greene, James Y.

AU - Martin, Douglas

AU - Tierney, William M.

PY - 1998/4/1

Y1 - 1998/4/1

N2 - Focus groups are increasingly being used to provide insights to researchers and policy makers. These data complement quantitative approaches to understanding the world. Unfortunately, quantitative and qualitative methodologies have often been viewed as antithetical, rather than complementary, strategies. While focus groups can clearly generate rich information that is unobtainable through other quantitative methods, it is important to determine the degree to which different raters can consistently extract information from transcripts. Thus, our goal was to quantify agreement in the interpretation of transcripts from patient and physician focus groups, using decision-making in ischemic heart disease as a model. We used data from focus groups with both patients and physicians that sought to identify factors affecting diagnostic and treatment decisions in ischemic heart disease. Three raters independently reviewed transcribed audiotapes from focus groups of patients with ischemic heart disease, as well as focus groups of physicians who care for these patients. We found that raters could not distinguish between major and minor factors reliably. More troubling, however, is that consistency regarding the apparently straightforward judgment as to the mere presence or absence of a factor was difficult to achieve. In particular, the three raters of each transcript failed to agree on between one third and one half of the factors. This reasonably high level of disagreement occurred despite the raters: (1) having generated the individual factors themselves based upon their reading a random sample of actual transcripts and (2) being trained in the use of rating forms (including standard definitions of themes). These data suggest that if a single rater evaluates focus group transcripts, as is commonly done, judgments may not be reproducible by other raters. Moreover; a single rater may nor extract all important information contained in the transcripts.

AB - Focus groups are increasingly being used to provide insights to researchers and policy makers. These data complement quantitative approaches to understanding the world. Unfortunately, quantitative and qualitative methodologies have often been viewed as antithetical, rather than complementary, strategies. While focus groups can clearly generate rich information that is unobtainable through other quantitative methods, it is important to determine the degree to which different raters can consistently extract information from transcripts. Thus, our goal was to quantify agreement in the interpretation of transcripts from patient and physician focus groups, using decision-making in ischemic heart disease as a model. We used data from focus groups with both patients and physicians that sought to identify factors affecting diagnostic and treatment decisions in ischemic heart disease. Three raters independently reviewed transcribed audiotapes from focus groups of patients with ischemic heart disease, as well as focus groups of physicians who care for these patients. We found that raters could not distinguish between major and minor factors reliably. More troubling, however, is that consistency regarding the apparently straightforward judgment as to the mere presence or absence of a factor was difficult to achieve. In particular, the three raters of each transcript failed to agree on between one third and one half of the factors. This reasonably high level of disagreement occurred despite the raters: (1) having generated the individual factors themselves based upon their reading a random sample of actual transcripts and (2) being trained in the use of rating forms (including standard definitions of themes). These data suggest that if a single rater evaluates focus group transcripts, as is commonly done, judgments may not be reproducible by other raters. Moreover; a single rater may nor extract all important information contained in the transcripts.

KW - Consistency of rating

KW - Focus groups

KW - Qualitative methods

KW - Racial variation

UR - http://www.scopus.com/inward/record.url?scp=0031939048&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0031939048&partnerID=8YFLogxK

U2 - 10.1016/S0277-9536(97)10028-4

DO - 10.1016/S0277-9536(97)10028-4

M3 - Article

C2 - 9541078

AN - SCOPUS:0031939048

VL - 46

SP - 929

EP - 933

JO - Social Science and Medicine

JF - Social Science and Medicine

SN - 0277-9536

IS - 7

ER -