Publication
Title
Multi-modal label retrieval for the visual arts : the case of Iconclass
Author
Abstract
Iconclass is an iconographic classification system from the domain of cultural heritage which is used to annotate subjects represented in the visual arts. In this work, we investigate the feasibility of automatically assigning Iconclass codes to visual artworks using a cross-modal retrieval set-up. We explore the text and image branches of the cross-modal network. In addition, we describe a multi-modal architecture that can jointly capitalize on multiple feature sources: textual features, coming from the titles for these artworks (in multiple languages) and visual features, extracted from photographic reproductions of the artworks. We utilize Iconclass definitions in English as matching labels. We evaluate our approach on a publicly available dataset of artworks (containing English and Dutch titles). Our results demonstrate that, in isolation, textual features strongly outperform visual features, although visual features can still offer a useful complement to purely linguistic features. Moreover, we show the cross-lingual (Dutch-English) strategy to be on par with the monolingual approach (English-English), which opens important perspectives for applications of this approach beyond resource-rich languages.
Language
English
Source (book)
Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) - Volume 1
Publication
SCITEPRESS , 2021
ISBN
978-989-758-484-8
DOI
10.5220/0010390606220629
Volume/pages
p. 622-629
ISI
000661445300068
Full text (Publisher's DOI)
Full text (open access)
UAntwerpen
Faculty/Department
Research group
Publication type
Subject
Art 
Affiliation
Publications with a UAntwerp address
External links
Web of Science
Record
Identifier
Creation 18.03.2021
Last edited 02.10.2024
To cite this reference