Multimodal perceptual organization of speech: Evidence from tone analogs of spoken utterances

Robert E. Remez, Jennifer M. Fellowes, David B. Pisoni, Winston D. Goh, Philip E. Rubin

Research output: Contribution to journalArticle

19 Scopus citations

Abstract

Theoretical and practical motives alike have prompted recent investigations of multimodal speech perception. Theoretically, multimodal studies have extended the conceptualization of perceptual organization beyond the familiar modality-bound accounts deriving from Gestalt psychology. Practically, such investigations have been driven by a need to understand the proficiency of multimodal speech perception using an electrocochlear prosthesis for hearing. In each domain, studies have shown that perceptual organization of speech can occur even when the perceiver's auditory experience departs from natural speech qualities. Accordingly, our research examined auditor-visual multimodal integration of videotaped faces and selected acoustic constituents of speech signals, each realized as a single sinewave tone accompanying a video image of an articulating face. The single tone reproduced the frequency and amplitude of the phonatory cycle or of one of the lower three oral formants. Our results showed a distinct advantage for the condition pairing the video image of the face with a sinewave replicating the second formant, despite its unnatural timbre and its presentation in acoustic isolation from the rest of the speech signal. Perceptual coherence of multimodal speech in these circumstances is established when the two modalities concurrently specify the same underlying phonetic attributes.

Original languageEnglish (US)
Pages (from-to)65-73
Number of pages9
JournalSpeech Communication
Volume26
Issue number1-2
DOIs
StatePublished - Oct 1998

    Fingerprint

Keywords

  • Auditory-visual speech perception
  • Intersensory integration
  • Multimodal speech perception
  • Perceptual organization
  • Sinewave speech
  • Speechreading

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering
  • Experimental and Cognitive Psychology
  • Linguistics and Language

Cite this