Validated system for centralized grading of retinopathy of prematurity: Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study

Ebenezer Daniel, Graham E. Quinn, P. Lloyd Hildebrand, Anna Ells, G. Baker Hubbard, Antonio Capone, E. Revell Martin, Candace P. Ostroff, Eli Smith, Kelly C. Wade, Agnieshka Baumritter, Trang B. Duros, Lisa Erbring, Michael X. Repka, Jennifer A. Shepard, David Emmert, C. Mark Herring, Deborah VanderVeen, Suzanne Johnston, Carolyn WuJason Mantagos, Danielle Ledoux, Tamar Winter, Frank Weng, Theresa Mansfield, Don L. Bremer, Mary Lou McGregor, Catherine Olson Jordan, David L. Rogers, Rae R. Fellows, Suzanne Brandt, Brenda Mann, David K. Wallace, Sharon Freedman, Sarah K. Jones, Du Tran-Viet, Rhonda Young, Charles C. Barr, Rahul Bhola, Craig Douglas, Peggy Fishman, Michelle Bottorff, Brandi Hubbuch, Rachel Keith, Erick D. Bothun, Inge DeBecker, Jill Anderson, Ann Marie Holleschau, Nichole E. Miller, Darla N. Nyquist, R. Michael Siatkowski, Lucas Trigler, Marilyn Escobedo, Karen Corff, Michelle Huynh, Kelli Satnes, Monte D. Mills, Will Anninger, Gil Binenbaum, Karen A. Karp, Denise Pearson, Alice Gong, John Stokes, Clio Armitage Harper, Laurie Weaver, Carmen McHenry, Kathryn Conner, Rosalind Heemer, Elnora Cokley, Robert Hoffman, David Dries, Katie Jo Farnsworth, Deborah Harrison, Bonnie Carlstrom, Cyrie Ann Frye, David Morrison, Sean Donahue, Nancy Benegas, Sandy Owings, Sandra Phillips, Scott Ruark, Patrick Mitchell, April Ingram, Rosie Sorbie, Maureen Maguire, Mary Brightwell-Arnold, Kathleen McWilliams, Sandra Harris, Claressa Whearry, Krista Sepielli, Kerry Davis, G. Carl Gibson, Regina Hansen, Alex R. Kemper, Lisa Prosser, David C. Musch, Stephen P. Christiansen, Ditte J. Hess, Steven M. Kymes, Srini Vas R. Sadda, Ryan Spaulding, Eleanor B. Schron

Research output: Contribution to journalArticle

36 Citations (Scopus)

Abstract

IMPORTANCE: Measurable competence derived from comprehensive and advanced training in grading digital images is critical in studies using a reading center to evaluate retinal fundus images from infants at risk for retinopathy of prematurity (ROP). Details of certification for nonphysician trained readers (TRs) have not yet been described. OBJECTIVE: To describe a centralized system for grading ROP digital images by TRs in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. DESIGN, SETTING, AND PARTICIPANTS: Multicenter observational cohort study conducted from July 1, 2010, to June 30, 2014. The TRs were trained by experienced ROP specialists and certified to detect ROP morphology in digital retinal images under supervision of an ophthalmologist reading center director. An ROP reading center was developed with standard hardware, secure Internet access, and customized image viewing software with an electronic grading form. A detailed protocol for grading was developed. Based on results of TR gradings, a computerized algorithm determined whether referral-warranted ROP (RW-ROP; defined as presence of plus disease, zone I ROP, and stage 3 or worse ROP) was present in digital images from infants with birth weight less than 1251 g enrolled from May 25, 2011, through October 31, 2013. Independent double grading was done by the TRs with adjudication of discrepant fields performed by the reading center director. EXPOSURE Digital retinal images. MAIN OUTCOMES AND MEASURES: Intragrader and intergrader variability and monitoring for temporal drift. RESULTS: Four TRs underwent rigorous training and certification. A total of 5520 image sets were double graded, with 24.5%requiring adjudication for at least 1 component of RW-ROP. For individual RW-ROP components, the adjudication rate was 3.9%for plus disease, 12.4% for zone I ROP, and 16.9%for stage 3 or worse ROP. The weighted κ for intergrader agreement (n = 80 image sets) was 0.72 (95%CI, 0.52-0.93) for RW-ROP, 0.57 (95%CI, 0.37-0.77) for plus disease, 0.43 (95%CI, 0.24-0.63) for zone I ROP, and 0.67 (95%CI, 0.47-0.88) for stage 3 or worse ROP. The weighted κ for grade-regrade agreement was 0.77 (95%CI, 0.57-0.97) for RW-ROP, 0.87 (95%CI, 0.67-1.00) for plus disease, 0.70 (95%CI, 0.51-0.90) for zone I ROP, and 0.77 (95%CI, 0.57-0.97) for stage 3 or worse ROP. CONCLUSIONS AND RELEVANCE: These data suggest that the e-ROP system for training and certifying nonphysicians to grade ROP images under the supervision of a reading center director reliably detects potentially serious ROP with good intragrader and intergrader consistency and minimal temporal drift.

Original languageEnglish (US)
Pages (from-to)675-682
Number of pages8
JournalJAMA Ophthalmology
Volume133
Issue number6
DOIs
StatePublished - Jun 1 2015
Externally publishedYes

Fingerprint

Retinopathy of Prematurity
Telemedicine
Reading
Certification

ASJC Scopus subject areas

  • Ophthalmology

Cite this

Validated system for centralized grading of retinopathy of prematurity : Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. / Daniel, Ebenezer; Quinn, Graham E.; Hildebrand, P. Lloyd; Ells, Anna; Hubbard, G. Baker; Capone, Antonio; Martin, E. Revell; Ostroff, Candace P.; Smith, Eli; Wade, Kelly C.; Baumritter, Agnieshka; Duros, Trang B.; Erbring, Lisa; Repka, Michael X.; Shepard, Jennifer A.; Emmert, David; Herring, C. Mark; VanderVeen, Deborah; Johnston, Suzanne; Wu, Carolyn; Mantagos, Jason; Ledoux, Danielle; Winter, Tamar; Weng, Frank; Mansfield, Theresa; Bremer, Don L.; McGregor, Mary Lou; Jordan, Catherine Olson; Rogers, David L.; Fellows, Rae R.; Brandt, Suzanne; Mann, Brenda; Wallace, David K.; Freedman, Sharon; Jones, Sarah K.; Tran-Viet, Du; Young, Rhonda; Barr, Charles C.; Bhola, Rahul; Douglas, Craig; Fishman, Peggy; Bottorff, Michelle; Hubbuch, Brandi; Keith, Rachel; Bothun, Erick D.; DeBecker, Inge; Anderson, Jill; Holleschau, Ann Marie; Miller, Nichole E.; Nyquist, Darla N.; Siatkowski, R. Michael; Trigler, Lucas; Escobedo, Marilyn; Corff, Karen; Huynh, Michelle; Satnes, Kelli; Mills, Monte D.; Anninger, Will; Binenbaum, Gil; Karp, Karen A.; Pearson, Denise; Gong, Alice; Stokes, John; Harper, Clio Armitage; Weaver, Laurie; McHenry, Carmen; Conner, Kathryn; Heemer, Rosalind; Cokley, Elnora; Hoffman, Robert; Dries, David; Farnsworth, Katie Jo; Harrison, Deborah; Carlstrom, Bonnie; Frye, Cyrie Ann; Morrison, David; Donahue, Sean; Benegas, Nancy; Owings, Sandy; Phillips, Sandra; Ruark, Scott; Mitchell, Patrick; Ingram, April; Sorbie, Rosie; Maguire, Maureen; Brightwell-Arnold, Mary; McWilliams, Kathleen; Harris, Sandra; Whearry, Claressa; Sepielli, Krista; Davis, Kerry; Gibson, G. Carl; Hansen, Regina; Kemper, Alex R.; Prosser, Lisa; Musch, David C.; Christiansen, Stephen P.; Hess, Ditte J.; Kymes, Steven M.; Sadda, Srini Vas R.; Spaulding, Ryan; Schron, Eleanor B.

In: JAMA Ophthalmology, Vol. 133, No. 6, 01.06.2015, p. 675-682.

Research output: Contribution to journalArticle

Daniel, E, Quinn, GE, Hildebrand, PL, Ells, A, Hubbard, GB, Capone, A, Martin, ER, Ostroff, CP, Smith, E, Wade, KC, Baumritter, A, Duros, TB, Erbring, L, Repka, MX, Shepard, JA, Emmert, D, Herring, CM, VanderVeen, D, Johnston, S, Wu, C, Mantagos, J, Ledoux, D, Winter, T, Weng, F, Mansfield, T, Bremer, DL, McGregor, ML, Jordan, CO, Rogers, DL, Fellows, RR, Brandt, S, Mann, B, Wallace, DK, Freedman, S, Jones, SK, Tran-Viet, D, Young, R, Barr, CC, Bhola, R, Douglas, C, Fishman, P, Bottorff, M, Hubbuch, B, Keith, R, Bothun, ED, DeBecker, I, Anderson, J, Holleschau, AM, Miller, NE, Nyquist, DN, Siatkowski, RM, Trigler, L, Escobedo, M, Corff, K, Huynh, M, Satnes, K, Mills, MD, Anninger, W, Binenbaum, G, Karp, KA, Pearson, D, Gong, A, Stokes, J, Harper, CA, Weaver, L, McHenry, C, Conner, K, Heemer, R, Cokley, E, Hoffman, R, Dries, D, Farnsworth, KJ, Harrison, D, Carlstrom, B, Frye, CA, Morrison, D, Donahue, S, Benegas, N, Owings, S, Phillips, S, Ruark, S, Mitchell, P, Ingram, A, Sorbie, R, Maguire, M, Brightwell-Arnold, M, McWilliams, K, Harris, S, Whearry, C, Sepielli, K, Davis, K, Gibson, GC, Hansen, R, Kemper, AR, Prosser, L, Musch, DC, Christiansen, SP, Hess, DJ, Kymes, SM, Sadda, SVR, Spaulding, R & Schron, EB 2015, 'Validated system for centralized grading of retinopathy of prematurity: Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study', JAMA Ophthalmology, vol. 133, no. 6, pp. 675-682. https://doi.org/10.1001/jamaophthalmol.2015.0460
Daniel, Ebenezer ; Quinn, Graham E. ; Hildebrand, P. Lloyd ; Ells, Anna ; Hubbard, G. Baker ; Capone, Antonio ; Martin, E. Revell ; Ostroff, Candace P. ; Smith, Eli ; Wade, Kelly C. ; Baumritter, Agnieshka ; Duros, Trang B. ; Erbring, Lisa ; Repka, Michael X. ; Shepard, Jennifer A. ; Emmert, David ; Herring, C. Mark ; VanderVeen, Deborah ; Johnston, Suzanne ; Wu, Carolyn ; Mantagos, Jason ; Ledoux, Danielle ; Winter, Tamar ; Weng, Frank ; Mansfield, Theresa ; Bremer, Don L. ; McGregor, Mary Lou ; Jordan, Catherine Olson ; Rogers, David L. ; Fellows, Rae R. ; Brandt, Suzanne ; Mann, Brenda ; Wallace, David K. ; Freedman, Sharon ; Jones, Sarah K. ; Tran-Viet, Du ; Young, Rhonda ; Barr, Charles C. ; Bhola, Rahul ; Douglas, Craig ; Fishman, Peggy ; Bottorff, Michelle ; Hubbuch, Brandi ; Keith, Rachel ; Bothun, Erick D. ; DeBecker, Inge ; Anderson, Jill ; Holleschau, Ann Marie ; Miller, Nichole E. ; Nyquist, Darla N. ; Siatkowski, R. Michael ; Trigler, Lucas ; Escobedo, Marilyn ; Corff, Karen ; Huynh, Michelle ; Satnes, Kelli ; Mills, Monte D. ; Anninger, Will ; Binenbaum, Gil ; Karp, Karen A. ; Pearson, Denise ; Gong, Alice ; Stokes, John ; Harper, Clio Armitage ; Weaver, Laurie ; McHenry, Carmen ; Conner, Kathryn ; Heemer, Rosalind ; Cokley, Elnora ; Hoffman, Robert ; Dries, David ; Farnsworth, Katie Jo ; Harrison, Deborah ; Carlstrom, Bonnie ; Frye, Cyrie Ann ; Morrison, David ; Donahue, Sean ; Benegas, Nancy ; Owings, Sandy ; Phillips, Sandra ; Ruark, Scott ; Mitchell, Patrick ; Ingram, April ; Sorbie, Rosie ; Maguire, Maureen ; Brightwell-Arnold, Mary ; McWilliams, Kathleen ; Harris, Sandra ; Whearry, Claressa ; Sepielli, Krista ; Davis, Kerry ; Gibson, G. Carl ; Hansen, Regina ; Kemper, Alex R. ; Prosser, Lisa ; Musch, David C. ; Christiansen, Stephen P. ; Hess, Ditte J. ; Kymes, Steven M. ; Sadda, Srini Vas R. ; Spaulding, Ryan ; Schron, Eleanor B. / Validated system for centralized grading of retinopathy of prematurity : Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. In: JAMA Ophthalmology. 2015 ; Vol. 133, No. 6. pp. 675-682.
@article{a82d24b0ca124b13a903f02a93b2d392,
title = "Validated system for centralized grading of retinopathy of prematurity: Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study",
abstract = "IMPORTANCE: Measurable competence derived from comprehensive and advanced training in grading digital images is critical in studies using a reading center to evaluate retinal fundus images from infants at risk for retinopathy of prematurity (ROP). Details of certification for nonphysician trained readers (TRs) have not yet been described. OBJECTIVE: To describe a centralized system for grading ROP digital images by TRs in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. DESIGN, SETTING, AND PARTICIPANTS: Multicenter observational cohort study conducted from July 1, 2010, to June 30, 2014. The TRs were trained by experienced ROP specialists and certified to detect ROP morphology in digital retinal images under supervision of an ophthalmologist reading center director. An ROP reading center was developed with standard hardware, secure Internet access, and customized image viewing software with an electronic grading form. A detailed protocol for grading was developed. Based on results of TR gradings, a computerized algorithm determined whether referral-warranted ROP (RW-ROP; defined as presence of plus disease, zone I ROP, and stage 3 or worse ROP) was present in digital images from infants with birth weight less than 1251 g enrolled from May 25, 2011, through October 31, 2013. Independent double grading was done by the TRs with adjudication of discrepant fields performed by the reading center director. EXPOSURE Digital retinal images. MAIN OUTCOMES AND MEASURES: Intragrader and intergrader variability and monitoring for temporal drift. RESULTS: Four TRs underwent rigorous training and certification. A total of 5520 image sets were double graded, with 24.5{\%}requiring adjudication for at least 1 component of RW-ROP. For individual RW-ROP components, the adjudication rate was 3.9{\%}for plus disease, 12.4{\%} for zone I ROP, and 16.9{\%}for stage 3 or worse ROP. The weighted κ for intergrader agreement (n = 80 image sets) was 0.72 (95{\%}CI, 0.52-0.93) for RW-ROP, 0.57 (95{\%}CI, 0.37-0.77) for plus disease, 0.43 (95{\%}CI, 0.24-0.63) for zone I ROP, and 0.67 (95{\%}CI, 0.47-0.88) for stage 3 or worse ROP. The weighted κ for grade-regrade agreement was 0.77 (95{\%}CI, 0.57-0.97) for RW-ROP, 0.87 (95{\%}CI, 0.67-1.00) for plus disease, 0.70 (95{\%}CI, 0.51-0.90) for zone I ROP, and 0.77 (95{\%}CI, 0.57-0.97) for stage 3 or worse ROP. CONCLUSIONS AND RELEVANCE: These data suggest that the e-ROP system for training and certifying nonphysicians to grade ROP images under the supervision of a reading center director reliably detects potentially serious ROP with good intragrader and intergrader consistency and minimal temporal drift.",
author = "Ebenezer Daniel and Quinn, {Graham E.} and Hildebrand, {P. Lloyd} and Anna Ells and Hubbard, {G. Baker} and Antonio Capone and Martin, {E. Revell} and Ostroff, {Candace P.} and Eli Smith and Wade, {Kelly C.} and Agnieshka Baumritter and Duros, {Trang B.} and Lisa Erbring and Repka, {Michael X.} and Shepard, {Jennifer A.} and David Emmert and Herring, {C. Mark} and Deborah VanderVeen and Suzanne Johnston and Carolyn Wu and Jason Mantagos and Danielle Ledoux and Tamar Winter and Frank Weng and Theresa Mansfield and Bremer, {Don L.} and McGregor, {Mary Lou} and Jordan, {Catherine Olson} and Rogers, {David L.} and Fellows, {Rae R.} and Suzanne Brandt and Brenda Mann and Wallace, {David K.} and Sharon Freedman and Jones, {Sarah K.} and Du Tran-Viet and Rhonda Young and Barr, {Charles C.} and Rahul Bhola and Craig Douglas and Peggy Fishman and Michelle Bottorff and Brandi Hubbuch and Rachel Keith and Bothun, {Erick D.} and Inge DeBecker and Jill Anderson and Holleschau, {Ann Marie} and Miller, {Nichole E.} and Nyquist, {Darla N.} and Siatkowski, {R. Michael} and Lucas Trigler and Marilyn Escobedo and Karen Corff and Michelle Huynh and Kelli Satnes and Mills, {Monte D.} and Will Anninger and Gil Binenbaum and Karp, {Karen A.} and Denise Pearson and Alice Gong and John Stokes and Harper, {Clio Armitage} and Laurie Weaver and Carmen McHenry and Kathryn Conner and Rosalind Heemer and Elnora Cokley and Robert Hoffman and David Dries and Farnsworth, {Katie Jo} and Deborah Harrison and Bonnie Carlstrom and Frye, {Cyrie Ann} and David Morrison and Sean Donahue and Nancy Benegas and Sandy Owings and Sandra Phillips and Scott Ruark and Patrick Mitchell and April Ingram and Rosie Sorbie and Maureen Maguire and Mary Brightwell-Arnold and Kathleen McWilliams and Sandra Harris and Claressa Whearry and Krista Sepielli and Kerry Davis and Gibson, {G. Carl} and Regina Hansen and Kemper, {Alex R.} and Lisa Prosser and Musch, {David C.} and Christiansen, {Stephen P.} and Hess, {Ditte J.} and Kymes, {Steven M.} and Sadda, {Srini Vas R.} and Ryan Spaulding and Schron, {Eleanor B.}",
year = "2015",
month = "6",
day = "1",
doi = "10.1001/jamaophthalmol.2015.0460",
language = "English (US)",
volume = "133",
pages = "675--682",
journal = "JAMA Ophthalmology",
issn = "2168-6165",
publisher = "American Medical Association",
number = "6",

}

TY - JOUR

T1 - Validated system for centralized grading of retinopathy of prematurity

T2 - Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study

AU - Daniel, Ebenezer

AU - Quinn, Graham E.

AU - Hildebrand, P. Lloyd

AU - Ells, Anna

AU - Hubbard, G. Baker

AU - Capone, Antonio

AU - Martin, E. Revell

AU - Ostroff, Candace P.

AU - Smith, Eli

AU - Wade, Kelly C.

AU - Baumritter, Agnieshka

AU - Duros, Trang B.

AU - Erbring, Lisa

AU - Repka, Michael X.

AU - Shepard, Jennifer A.

AU - Emmert, David

AU - Herring, C. Mark

AU - VanderVeen, Deborah

AU - Johnston, Suzanne

AU - Wu, Carolyn

AU - Mantagos, Jason

AU - Ledoux, Danielle

AU - Winter, Tamar

AU - Weng, Frank

AU - Mansfield, Theresa

AU - Bremer, Don L.

AU - McGregor, Mary Lou

AU - Jordan, Catherine Olson

AU - Rogers, David L.

AU - Fellows, Rae R.

AU - Brandt, Suzanne

AU - Mann, Brenda

AU - Wallace, David K.

AU - Freedman, Sharon

AU - Jones, Sarah K.

AU - Tran-Viet, Du

AU - Young, Rhonda

AU - Barr, Charles C.

AU - Bhola, Rahul

AU - Douglas, Craig

AU - Fishman, Peggy

AU - Bottorff, Michelle

AU - Hubbuch, Brandi

AU - Keith, Rachel

AU - Bothun, Erick D.

AU - DeBecker, Inge

AU - Anderson, Jill

AU - Holleschau, Ann Marie

AU - Miller, Nichole E.

AU - Nyquist, Darla N.

AU - Siatkowski, R. Michael

AU - Trigler, Lucas

AU - Escobedo, Marilyn

AU - Corff, Karen

AU - Huynh, Michelle

AU - Satnes, Kelli

AU - Mills, Monte D.

AU - Anninger, Will

AU - Binenbaum, Gil

AU - Karp, Karen A.

AU - Pearson, Denise

AU - Gong, Alice

AU - Stokes, John

AU - Harper, Clio Armitage

AU - Weaver, Laurie

AU - McHenry, Carmen

AU - Conner, Kathryn

AU - Heemer, Rosalind

AU - Cokley, Elnora

AU - Hoffman, Robert

AU - Dries, David

AU - Farnsworth, Katie Jo

AU - Harrison, Deborah

AU - Carlstrom, Bonnie

AU - Frye, Cyrie Ann

AU - Morrison, David

AU - Donahue, Sean

AU - Benegas, Nancy

AU - Owings, Sandy

AU - Phillips, Sandra

AU - Ruark, Scott

AU - Mitchell, Patrick

AU - Ingram, April

AU - Sorbie, Rosie

AU - Maguire, Maureen

AU - Brightwell-Arnold, Mary

AU - McWilliams, Kathleen

AU - Harris, Sandra

AU - Whearry, Claressa

AU - Sepielli, Krista

AU - Davis, Kerry

AU - Gibson, G. Carl

AU - Hansen, Regina

AU - Kemper, Alex R.

AU - Prosser, Lisa

AU - Musch, David C.

AU - Christiansen, Stephen P.

AU - Hess, Ditte J.

AU - Kymes, Steven M.

AU - Sadda, Srini Vas R.

AU - Spaulding, Ryan

AU - Schron, Eleanor B.

PY - 2015/6/1

Y1 - 2015/6/1

N2 - IMPORTANCE: Measurable competence derived from comprehensive and advanced training in grading digital images is critical in studies using a reading center to evaluate retinal fundus images from infants at risk for retinopathy of prematurity (ROP). Details of certification for nonphysician trained readers (TRs) have not yet been described. OBJECTIVE: To describe a centralized system for grading ROP digital images by TRs in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. DESIGN, SETTING, AND PARTICIPANTS: Multicenter observational cohort study conducted from July 1, 2010, to June 30, 2014. The TRs were trained by experienced ROP specialists and certified to detect ROP morphology in digital retinal images under supervision of an ophthalmologist reading center director. An ROP reading center was developed with standard hardware, secure Internet access, and customized image viewing software with an electronic grading form. A detailed protocol for grading was developed. Based on results of TR gradings, a computerized algorithm determined whether referral-warranted ROP (RW-ROP; defined as presence of plus disease, zone I ROP, and stage 3 or worse ROP) was present in digital images from infants with birth weight less than 1251 g enrolled from May 25, 2011, through October 31, 2013. Independent double grading was done by the TRs with adjudication of discrepant fields performed by the reading center director. EXPOSURE Digital retinal images. MAIN OUTCOMES AND MEASURES: Intragrader and intergrader variability and monitoring for temporal drift. RESULTS: Four TRs underwent rigorous training and certification. A total of 5520 image sets were double graded, with 24.5%requiring adjudication for at least 1 component of RW-ROP. For individual RW-ROP components, the adjudication rate was 3.9%for plus disease, 12.4% for zone I ROP, and 16.9%for stage 3 or worse ROP. The weighted κ for intergrader agreement (n = 80 image sets) was 0.72 (95%CI, 0.52-0.93) for RW-ROP, 0.57 (95%CI, 0.37-0.77) for plus disease, 0.43 (95%CI, 0.24-0.63) for zone I ROP, and 0.67 (95%CI, 0.47-0.88) for stage 3 or worse ROP. The weighted κ for grade-regrade agreement was 0.77 (95%CI, 0.57-0.97) for RW-ROP, 0.87 (95%CI, 0.67-1.00) for plus disease, 0.70 (95%CI, 0.51-0.90) for zone I ROP, and 0.77 (95%CI, 0.57-0.97) for stage 3 or worse ROP. CONCLUSIONS AND RELEVANCE: These data suggest that the e-ROP system for training and certifying nonphysicians to grade ROP images under the supervision of a reading center director reliably detects potentially serious ROP with good intragrader and intergrader consistency and minimal temporal drift.

AB - IMPORTANCE: Measurable competence derived from comprehensive and advanced training in grading digital images is critical in studies using a reading center to evaluate retinal fundus images from infants at risk for retinopathy of prematurity (ROP). Details of certification for nonphysician trained readers (TRs) have not yet been described. OBJECTIVE: To describe a centralized system for grading ROP digital images by TRs in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity (e-ROP) Study. DESIGN, SETTING, AND PARTICIPANTS: Multicenter observational cohort study conducted from July 1, 2010, to June 30, 2014. The TRs were trained by experienced ROP specialists and certified to detect ROP morphology in digital retinal images under supervision of an ophthalmologist reading center director. An ROP reading center was developed with standard hardware, secure Internet access, and customized image viewing software with an electronic grading form. A detailed protocol for grading was developed. Based on results of TR gradings, a computerized algorithm determined whether referral-warranted ROP (RW-ROP; defined as presence of plus disease, zone I ROP, and stage 3 or worse ROP) was present in digital images from infants with birth weight less than 1251 g enrolled from May 25, 2011, through October 31, 2013. Independent double grading was done by the TRs with adjudication of discrepant fields performed by the reading center director. EXPOSURE Digital retinal images. MAIN OUTCOMES AND MEASURES: Intragrader and intergrader variability and monitoring for temporal drift. RESULTS: Four TRs underwent rigorous training and certification. A total of 5520 image sets were double graded, with 24.5%requiring adjudication for at least 1 component of RW-ROP. For individual RW-ROP components, the adjudication rate was 3.9%for plus disease, 12.4% for zone I ROP, and 16.9%for stage 3 or worse ROP. The weighted κ for intergrader agreement (n = 80 image sets) was 0.72 (95%CI, 0.52-0.93) for RW-ROP, 0.57 (95%CI, 0.37-0.77) for plus disease, 0.43 (95%CI, 0.24-0.63) for zone I ROP, and 0.67 (95%CI, 0.47-0.88) for stage 3 or worse ROP. The weighted κ for grade-regrade agreement was 0.77 (95%CI, 0.57-0.97) for RW-ROP, 0.87 (95%CI, 0.67-1.00) for plus disease, 0.70 (95%CI, 0.51-0.90) for zone I ROP, and 0.77 (95%CI, 0.57-0.97) for stage 3 or worse ROP. CONCLUSIONS AND RELEVANCE: These data suggest that the e-ROP system for training and certifying nonphysicians to grade ROP images under the supervision of a reading center director reliably detects potentially serious ROP with good intragrader and intergrader consistency and minimal temporal drift.

UR - http://www.scopus.com/inward/record.url?scp=84931062683&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84931062683&partnerID=8YFLogxK

U2 - 10.1001/jamaophthalmol.2015.0460

DO - 10.1001/jamaophthalmol.2015.0460

M3 - Article

C2 - 25811772

AN - SCOPUS:84931062683

VL - 133

SP - 675

EP - 682

JO - JAMA Ophthalmology

JF - JAMA Ophthalmology

SN - 2168-6165

IS - 6

ER -