Skip to Content
Applied SciencesApplied Sciences
  • Article
  • Open Access

10 October 2024

11 Pages

A Modality-Enhanced Multi-Channel Attention Network for Multi-Modal Dialogue Summarization

,
and
1
School of Cyber Science and Technology, Beihang University, Beijing 100191, China
2
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China
3
State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China
*
Author to whom correspondence should be addressed.

Abstract

Integrating multi-modal data in natural language processing has opened new pathways for the enhancement of dialogue summarization. However, existing models often struggle to effectively synthesize textual, auditory, and visual inputs. This paper introduces a Modality-Enhanced Multi-Channel Attention Network (MEMA), a novel approach designed to optimize the integration and interaction of diverse modalities for dialogue summarization. MEMA leverages symmetrical embedding strategies to balance the integrity and distinctiveness of each modality, ensuring a harmonious interaction within the unified architecture. By maintaining symmetry in the processing flow, MEMA enhances the contextual richness and coherence of the generated summaries. Our model demonstrates superior performance on the Multi-modal Dialogue Summarization (MDS) dataset, particularly in generating contextually enriched abstract summaries. The results underscore MEMA’s potential to transform dialogue summarization by providing a more symmetrical and integrated understanding of multi-modal interactions, bridging the gap in multi-modal data processing, and setting a new standard for future summarization tasks.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.