Skip to main content
Date
2020-11-06
Visual analytics (Bild: TIB/Andrea Seifert)

ICMR 2020: Contribution "Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency" awarded

The paper "Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency" received the Best Paper Award at the ACM International Conference on Multimedia Retrieval (ICMR) 2020, which took place digitally this year. The authors of the paper are Eric Müller-Budack (Visual Analytics Research Group of the TIB – Leibniz Information Centre for Science and Technology), Jonas Theiner, Maximilian Idahl (both L3S Research Center), Sebastian Diering (Leibniz Universität Hannover) and Ralph Ewerth (Visual Analytics Research Group and L3S Research Center).

The paper presents a multimodal approach to quantify the entity coherence between image and text in real-world news. Named entity linking is applied to extract persons, locations, and events from news texts. Several measures are suggested to calculate these entities' cross-modal similarity with the news photo, using state-of-the-art computer vision approaches. In contrast to previous work, the system automatically gathers example data from the Web and is applicable to real-world news. The feasibility is demonstrated on two novel datasets that cover different languages, topics, and domains. Quantifying the cross-modal consistency of entity representations can assist human assessors in evaluating the overall multimodal message. In some cases, such measures might give hints to detect fake news, which is an increasingly important topic in today’s society.

Reference: E. Müller-Budack, J. Theiner, S. Diering, M. Idahl, R. Ewerth: Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency. In: ACM International Conference on Multimedia Retrieval (ICMR), Dublin, Ireland (online event), 2020, 16-25. DOI: https://dl.acm.org/doi/pdf/10.1145/3372278.3390670 (PDF)