A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring

Chengyuan Lin, Daniel Andersen, Voicu Popescu, Edgar Rojas-Munoz, Maria Eugenia Cabrera, Brian Mullis, Ben Zarzaur, Kathryn Anderson, Sherri Marley, Juan Wachs

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This application paper presents the work of a multidisciplinary group of designing, implementing, and testing an Augmented Reality (AR) surgical telementoring system. The system acquires the surgical field with an overhead camera, the video feed is transmitted to the remote mentor, where it is displayed on a touch-based interaction table, the mentor annotates the video feed, the annotations are sent back to the mentee, where they are displayed into the mentee's field of view using an optical see-through AR head-mounted display (HMD). The annotations are reprojected from the mentor's second-person view of the surgical field to the mentee's first-person view. The mentee sees the annotations with depth perception, and the annotations remain anchored to the surgical field as the mentee moves their head. Average annotation display accuracy is 1.22cm. The system was tested in the context of a user study where surgery residents ($n = 20$) were asked to perform a lower-leg fasciotomy on cadaver models. Participants who benefited from telementoring using our system received a higher Individual Performance Score, and they reported higher usability and self confidence levels.

Original languageEnglish (US)
Title of host publicationAdjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3-8
Number of pages6
ISBN (Electronic)9781538675922
DOIs
StatePublished - Apr 25 2019
Externally publishedYes
Event17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 - Munich, Germany
Duration: Oct 16 2018Oct 20 2018

Publication series

NameAdjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

Conference

Conference17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018
CountryGermany
CityMunich
Period10/16/1810/20/18

Fingerprint

Augmented reality
Display devices
Depth perception
Surgery
Cameras
Testing

Keywords

  • Human computer interaction
  • Human-centered computing
  • Interaction paradigms
  • Mixed / augmented reality

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Artificial Intelligence

Cite this

Lin, C., Andersen, D., Popescu, V., Rojas-Munoz, E., Cabrera, M. E., Mullis, B., ... Wachs, J. (2019). A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring. In Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 (pp. 3-8). [8699305] (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISMAR-Adjunct.2018.00021

A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring. / Lin, Chengyuan; Andersen, Daniel; Popescu, Voicu; Rojas-Munoz, Edgar; Cabrera, Maria Eugenia; Mullis, Brian; Zarzaur, Ben; Anderson, Kathryn; Marley, Sherri; Wachs, Juan.

Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 3-8 8699305 (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lin, C, Andersen, D, Popescu, V, Rojas-Munoz, E, Cabrera, ME, Mullis, B, Zarzaur, B, Anderson, K, Marley, S & Wachs, J 2019, A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring. in Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018., 8699305, Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018, Institute of Electrical and Electronics Engineers Inc., pp. 3-8, 17th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018, Munich, Germany, 10/16/18. https://doi.org/10.1109/ISMAR-Adjunct.2018.00021
Lin C, Andersen D, Popescu V, Rojas-Munoz E, Cabrera ME, Mullis B et al. A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring. In Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 3-8. 8699305. (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018). https://doi.org/10.1109/ISMAR-Adjunct.2018.00021
Lin, Chengyuan ; Andersen, Daniel ; Popescu, Voicu ; Rojas-Munoz, Edgar ; Cabrera, Maria Eugenia ; Mullis, Brian ; Zarzaur, Ben ; Anderson, Kathryn ; Marley, Sherri ; Wachs, Juan. / A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring. Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 3-8 (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018).
@inproceedings{d690ba9f010d4066a001e58ab5bc20d2,
title = "A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring",
abstract = "This application paper presents the work of a multidisciplinary group of designing, implementing, and testing an Augmented Reality (AR) surgical telementoring system. The system acquires the surgical field with an overhead camera, the video feed is transmitted to the remote mentor, where it is displayed on a touch-based interaction table, the mentor annotates the video feed, the annotations are sent back to the mentee, where they are displayed into the mentee's field of view using an optical see-through AR head-mounted display (HMD). The annotations are reprojected from the mentor's second-person view of the surgical field to the mentee's first-person view. The mentee sees the annotations with depth perception, and the annotations remain anchored to the surgical field as the mentee moves their head. Average annotation display accuracy is 1.22cm. The system was tested in the context of a user study where surgery residents ($n = 20$) were asked to perform a lower-leg fasciotomy on cadaver models. Participants who benefited from telementoring using our system received a higher Individual Performance Score, and they reported higher usability and self confidence levels.",
keywords = "Human computer interaction, Human-centered computing, Interaction paradigms, Mixed / augmented reality",
author = "Chengyuan Lin and Daniel Andersen and Voicu Popescu and Edgar Rojas-Munoz and Cabrera, {Maria Eugenia} and Brian Mullis and Ben Zarzaur and Kathryn Anderson and Sherri Marley and Juan Wachs",
year = "2019",
month = "4",
day = "25",
doi = "10.1109/ISMAR-Adjunct.2018.00021",
language = "English (US)",
series = "Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "3--8",
booktitle = "Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018",

}

TY - GEN

T1 - A First-Person Mentee Second-Person Mentor AR Interface for Surgical Telementoring

AU - Lin, Chengyuan

AU - Andersen, Daniel

AU - Popescu, Voicu

AU - Rojas-Munoz, Edgar

AU - Cabrera, Maria Eugenia

AU - Mullis, Brian

AU - Zarzaur, Ben

AU - Anderson, Kathryn

AU - Marley, Sherri

AU - Wachs, Juan

PY - 2019/4/25

Y1 - 2019/4/25

N2 - This application paper presents the work of a multidisciplinary group of designing, implementing, and testing an Augmented Reality (AR) surgical telementoring system. The system acquires the surgical field with an overhead camera, the video feed is transmitted to the remote mentor, where it is displayed on a touch-based interaction table, the mentor annotates the video feed, the annotations are sent back to the mentee, where they are displayed into the mentee's field of view using an optical see-through AR head-mounted display (HMD). The annotations are reprojected from the mentor's second-person view of the surgical field to the mentee's first-person view. The mentee sees the annotations with depth perception, and the annotations remain anchored to the surgical field as the mentee moves their head. Average annotation display accuracy is 1.22cm. The system was tested in the context of a user study where surgery residents ($n = 20$) were asked to perform a lower-leg fasciotomy on cadaver models. Participants who benefited from telementoring using our system received a higher Individual Performance Score, and they reported higher usability and self confidence levels.

AB - This application paper presents the work of a multidisciplinary group of designing, implementing, and testing an Augmented Reality (AR) surgical telementoring system. The system acquires the surgical field with an overhead camera, the video feed is transmitted to the remote mentor, where it is displayed on a touch-based interaction table, the mentor annotates the video feed, the annotations are sent back to the mentee, where they are displayed into the mentee's field of view using an optical see-through AR head-mounted display (HMD). The annotations are reprojected from the mentor's second-person view of the surgical field to the mentee's first-person view. The mentee sees the annotations with depth perception, and the annotations remain anchored to the surgical field as the mentee moves their head. Average annotation display accuracy is 1.22cm. The system was tested in the context of a user study where surgery residents ($n = 20$) were asked to perform a lower-leg fasciotomy on cadaver models. Participants who benefited from telementoring using our system received a higher Individual Performance Score, and they reported higher usability and self confidence levels.

KW - Human computer interaction

KW - Human-centered computing

KW - Interaction paradigms

KW - Mixed / augmented reality

UR - http://www.scopus.com/inward/record.url?scp=85065551360&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065551360&partnerID=8YFLogxK

U2 - 10.1109/ISMAR-Adjunct.2018.00021

DO - 10.1109/ISMAR-Adjunct.2018.00021

M3 - Conference contribution

AN - SCOPUS:85065551360

T3 - Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

SP - 3

EP - 8

BT - Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -