Next Article in Journal
Quantitative Analyses of the Left Ventricle Volume and Cardiac Function in Normal and Infarcted Yucatan Minipigs
Next Article in Special Issue
Camera Color Correction for Cultural Heritage Preservation Based on Clustered Data
Previous Article in Journal
How Can a Deep Learning Algorithm Improve Fracture Detection on X-rays in the Emergency Room?
Previous Article in Special Issue
Restoration and Enhancement of Historical Stereo Photos
Article

Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis

1
Department of Electronics Information and Bioengineering, Politecnico di Milano, 20133 Milano, Italy
2
Department of ICT and Engineering, NTNU—Norwegian University of Science and Technology, 6009 Ålesund, Norway
*
Author to whom correspondence should be addressed.
Academic Editors: Giovanna Castellano, Gennaro Vessio and Fabio Bellavia
J. Imaging 2021, 7(7), 106; https://doi.org/10.3390/jimaging7070106
Received: 15 May 2021 / Revised: 16 June 2021 / Accepted: 24 June 2021 / Published: 29 June 2021
(This article belongs to the Special Issue Fine Art Pattern Extraction and Recognition)
Iconography studies the visual content of artworks by considering the themes portrayed in them and their representation. Computer Vision has been used to identify iconographic subjects in paintings and Convolutional Neural Networks enabled the effective classification of characters in Christian art paintings. However, it still has to be demonstrated if the classification results obtained by CNNs rely on the same iconographic properties that human experts exploit when studying iconography and if the architecture of a classifier trained on whole artwork images can be exploited to support the much harder task of object detection. A suitable approach for exposing the process of classification by neural models relies on Class Activation Maps, which emphasize the areas of an image contributing the most to the classification. This work compares state-of-the-art algorithms (CAM, Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++) in terms of their capacity of identifying the iconographic attributes that determine the classification of characters in Christian art paintings. Quantitative and qualitative analyses show that Grad-CAM, Grad-CAM++, and Smooth Grad-CAM++ have similar performances while CAM has lower efficacy. Smooth Grad-CAM++ isolates multiple disconnected image regions that identify small iconographic symbols well. Grad-CAM produces wider and more contiguous areas that cover large iconographic symbols better. The salient image areas computed by the CAM algorithms have been used to estimate object-level bounding boxes and a quantitative analysis shows that the boxes estimated with Grad-CAM reach 55% average IoU, 61% GT-known localization and 31% mAP. The obtained results are a step towards the computer-aided study of the variations of iconographic elements positioning and mutual relations in artworks and open the way to the automatic creation of bounding boxes for training detectors of iconographic symbols in Christian art images. View Full-Text
Keywords: convolutional neural network; class activation maps; explainability; iconography; artwork analysis convolutional neural network; class activation maps; explainability; iconography; artwork analysis
Show Figures

Figure 1

MDPI and ACS Style

Pinciroli Vago, N.O.; Milani, F.; Fraternali, P.; da Silva Torres, R. Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis. J. Imaging 2021, 7, 106. https://doi.org/10.3390/jimaging7070106

AMA Style

Pinciroli Vago NO, Milani F, Fraternali P, da Silva Torres R. Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis. Journal of Imaging. 2021; 7(7):106. https://doi.org/10.3390/jimaging7070106

Chicago/Turabian Style

Pinciroli Vago, Nicolò O., Federico Milani, Piero Fraternali, and Ricardo da Silva Torres. 2021. "Comparing CAM Algorithms for the Identification of Salient Image Features in Iconography Artwork Analysis" Journal of Imaging 7, no. 7: 106. https://doi.org/10.3390/jimaging7070106

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop