Yi-Ting Yeh

Yi-Ting Yeh

Master student in Language Technologies Institute at Carnegie Mellon University. Research interests cover Machine Learning, Natural Language Processing and dialogue systems.

Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling

Published in In 7th Dialog System Technology Challenge (DSTC7) in the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), 2019

Yi-Ting Yeh, Tzu-Chuan Lin, Hsiao-Hua Cheng, Yi-Hsuan Deng, Shang-Yu Su, and Yun-Nung Chen.

Full Paper: arXiv

Code: GitHub

Visual question answering and visual dialogue tasks have been increasingly studied in the multimodal field towards more practical real-world scenarios.

A more challenging task, audio visual scene-aware dialogue (AVSD), is proposed to further advance the technologies that connect audio, vision, and language, which introduces temporal video information and dialogue interactions between a questioner and an answerer.

This paper proposes an intuitive mechanism that fuses features and attention in multiple stages in order to well integrate multimodal features, and the results demonstrate its capability in the experiments.

Also, we apply several state-of-the-art models in other tasks to the AVSD task, and further analyze their generalization across different tasks.