Specification of cross-modal source information in isolated kinematic displays of speech

Lorin Lachs, David Pisoni

Research output: Contribution to journalArticle

21 Citations (Scopus)

Abstract

Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.

Original languageEnglish
Pages (from-to)507-518
Number of pages12
JournalJournal of the Acoustical Society of America
Volume116
Issue number1
DOIs
StatePublished - Jul 2004

Fingerprint

specifications
kinematics
information theory
acoustic properties
replicas
stimuli
Kinematics
Hearing
Word Recognition

ASJC Scopus subject areas

  • Acoustics and Ultrasonics

Cite this

Specification of cross-modal source information in isolated kinematic displays of speech. / Lachs, Lorin; Pisoni, David.

In: Journal of the Acoustical Society of America, Vol. 116, No. 1, 07.2004, p. 507-518.

Research output: Contribution to journalArticle

@article{4d4c39ccaa7646dd8bbaca07d196ac74,
title = "Specification of cross-modal source information in isolated kinematic displays of speech",
abstract = "Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.",
author = "Lorin Lachs and David Pisoni",
year = "2004",
month = "7",
doi = "10.1121/1.1757454",
language = "English",
volume = "116",
pages = "507--518",
journal = "Journal of the Acoustical Society of America",
issn = "0001-4966",
publisher = "Acoustical Society of America",
number = "1",

}

TY - JOUR

T1 - Specification of cross-modal source information in isolated kinematic displays of speech

AU - Lachs, Lorin

AU - Pisoni, David

PY - 2004/7

Y1 - 2004/7

N2 - Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.

AB - Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.

UR - http://www.scopus.com/inward/record.url?scp=3142688301&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=3142688301&partnerID=8YFLogxK

U2 - 10.1121/1.1757454

DO - 10.1121/1.1757454

M3 - Article

VL - 116

SP - 507

EP - 518

JO - Journal of the Acoustical Society of America

JF - Journal of the Acoustical Society of America

SN - 0001-4966

IS - 1

ER -