Investigations on End- to-End Audiovisual Fusion

Michael Wand, Jurgen Schmidhuber, Ngoc Thang Vu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Scopus citations


Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.
Original languageEnglish (US)
Title of host publicationICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages5
ISBN (Print)9781538646588
StatePublished - Sep 10 2018
Externally publishedYes


Dive into the research topics of 'Investigations on End- to-End Audiovisual Fusion'. Together they form a unique fingerprint.

Cite this