Motion Dynamics Improve Speaker-Independent Lipreading

Matteo Riva; Michael Wand; Jurgen Schmidhuber

doi:10.1109/ICASSP40776.2020.9053535

Motion Dynamics Improve Speaker-Independent Lipreading

Matteo Riva, Michael Wand, Jurgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Scopus citations

Abstract

We present a novel lipreading system that improves on the task of speaker-independent word recognition by decoupling motion and content dynamics. We achieve this by implementing a deep learning architecture that uses two distinct pipelines to process motion and content and subsequently merges them, implementing an end-to-end trainable system that performs fusion of independently learned representations. We obtain a average relative word accuracy improvement of ≈6.8% on unseen speakers and of ≈3.3% on known speakers, with respect to a baseline which uses a standard architecture.

Original language	English (US)
Title of host publication	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	4407-4411
Number of pages	5
ISBN (Print)	9781509066315
DOIs	https://doi.org/10.1109/ICASSP40776.2020.9053535
State	Published - May 1 2020
Externally published	Yes

Access to Document

10.1109/ICASSP40776.2020.9053535

Cite this

@inproceedings{54a518dc2e4540629fd5e842463b5b48,

title = "Motion Dynamics Improve Speaker-Independent Lipreading",

abstract = "We present a novel lipreading system that improves on the task of speaker-independent word recognition by decoupling motion and content dynamics. We achieve this by implementing a deep learning architecture that uses two distinct pipelines to process motion and content and subsequently merges them, implementing an end-to-end trainable system that performs fusion of independently learned representations. We obtain a average relative word accuracy improvement of ≈6.8% on unseen speakers and of ≈3.3% on known speakers, with respect to a baseline which uses a standard architecture.",

author = "Matteo Riva and Michael Wand and Jurgen Schmidhuber",

note = "Generated from Scopus record by KAUST IRTS on 2022-09-14",

year = "2020",

month = may,

day = "1",

doi = "10.1109/ICASSP40776.2020.9053535",

language = "English (US)",

isbn = "9781509066315",

pages = "4407--4411",

booktitle = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

}

TY - GEN

T1 - Motion Dynamics Improve Speaker-Independent Lipreading

AU - Riva, Matteo

AU - Wand, Michael

AU - Schmidhuber, Jurgen

N1 - Generated from Scopus record by KAUST IRTS on 2022-09-14

PY - 2020/5/1

Y1 - 2020/5/1

N2 - We present a novel lipreading system that improves on the task of speaker-independent word recognition by decoupling motion and content dynamics. We achieve this by implementing a deep learning architecture that uses two distinct pipelines to process motion and content and subsequently merges them, implementing an end-to-end trainable system that performs fusion of independently learned representations. We obtain a average relative word accuracy improvement of ≈6.8% on unseen speakers and of ≈3.3% on known speakers, with respect to a baseline which uses a standard architecture.

AB - We present a novel lipreading system that improves on the task of speaker-independent word recognition by decoupling motion and content dynamics. We achieve this by implementing a deep learning architecture that uses two distinct pipelines to process motion and content and subsequently merges them, implementing an end-to-end trainable system that performs fusion of independently learned representations. We obtain a average relative word accuracy improvement of ≈6.8% on unseen speakers and of ≈3.3% on known speakers, with respect to a baseline which uses a standard architecture.

UR - https://ieeexplore.ieee.org/document/9053535/

UR - http://www.scopus.com/inward/record.url?scp=85089224269&partnerID=8YFLogxK

U2 - 10.1109/ICASSP40776.2020.9053535

DO - 10.1109/ICASSP40776.2020.9053535

M3 - Conference contribution

SN - 9781509066315

SP - 4407

EP - 4411

BT - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Motion Dynamics Improve Speaker-Independent Lipreading

Abstract

Access to Document

Other files and links

Fingerprint

Cite this