A Video Mapping Performance as an Innovative Tool to Bring to Life and Narrate a Pictorial Cycle

Abstract

Video mapping is defined as a particular form of augmented reality capable of transforming any surface, flat or irregular, into a dynamic surface capable of enriching human sensory perception. Video mapping projections can become a medium to link the historical facts and the location by means of the valorisation of the monument and narration of its story through images and sounds. This paper aims to show how video mapping, beyond its purely technological aspect, can be linked to cultural heritage and represents a tool capable of becoming a mediator of culture, tradition, and legends. It is used to pass on and tell the legend of the foundation of the present Cathedral of Maria Santissima della Madia in Monopoli through the animation of the pictorial cycle by Nicolò Maria Signorile preserved in the church.

1. Introduction

In the last decades, technological evolution has led us towards an individual and intimate vision using tablets and smartphones, but it has also enhanced the possibility of creating collective visions by transforming the facades of monuments into screens.

Spatial Augmented Reality (SAR) technology, better known as video mapping, consists of enriching human sensory perception through the addition (or subtraction) of information that allows physical reality to be perceived in a completely different way [1].

At the basis of this technology is the idea of creating an illusion, i.e., a form of hybridisation and dialogue that is established between real and virtual elements by means of a projection.

Unlike other uses of augmented reality, Augmented Spatial Reality has the advantage of allowing the user to interact with reality without the use of Head Mounted Displays or smartphones that could distract attention from the purpose of the realisation and lead it towards the medium used to enjoy it; through the projection on the surface to be animated, it is possible to guarantee a greater participation of the public with the surrounding space in which it is located.

Video mapping is therefore defined as a particular form of augmented reality capable of transforming any surface, flat or irregular, into a dynamic surface capable of enriching human sensory perception by using a computer and a system of video projectors.

It is a game of light and shadow that, through spectacular visual effects, gives the object an ‘animated skin’ in continuous movement in order to create a balance between spectacle, narration and content [2].

Over the years, video mapping has become a technique with infinite expressive potential, used in numerous events, festivals, and exhibitions but also in the commercial sector to communicate the launch of new products in an innovative way. Thus, technology allows for building of a creative and narrative path, and virtuality helps to integrate and enrich reality. Moreover, video mapping also means to impress, and therefore a precise integration of images and sounds is necessary.

However, we always must consider the coherence between the projected area and the story being told. If traditional visual storytelling has been able to evolve into a real multimedia entertainment that comes to life thanks to light effects, attention must be paid to the correct valorisation of the historical–cultural context through new ways of fruition to support the dissemination of culture and awareness of the urban environment.

Most video mapping projections are highly focused on special effects and rarely on the visual narrative, with the goal of communicating and disseminating historical knowledge and enhancing culture for a wider audience.

Video mapping became an artistic form of expression that began to be used widely in the cultural sphere in order to upgrade the urban environment, disseminate narratives and traditions and enhance historical contexts.

At the same time, the hybridisation of this technology with traditional museum systems and continuous experimentation has led to a gradual shift from “collection museums” to “storytelling and connection museums” [3], where storytelling is adopted as a communication technique capable of impacting the educational, touristic and economic spheres through the museum’s role as cultural mediator. In this new vision, digital storytelling, in all its countless forms, becomes a supreme and innovative form of communication [4,5].

The intrinsic connection that is created between the light show and the spectator is one of the primary objectives of Spatial Augmented Reality; the context, the cultural heritage and the person must become one, supported by light effects and narration.

The high potential of video mapping has made it an extremely versatile tool, whose applicability can be useful in different fields, such as artistic and cultural events, marketing and advertising. Video mapping projections can also be considered an alternative method of education.

In architecture, some possible applications can easily highlight architectural elements and illustrate the implementation process in order to demonstrate the relationship between the external and internal spaces.

In museums, video mapping allows enhancing knowledge of the exposed object, promoting an immediate approach to the acquisition of information and enhancing cultural and historical learning.

In addition, video mapping projections can become a medium to link the historical facts and the location by means of the valorisation of the monument and narration of its story through images and sounds.

Recent developments have led to the integration between SAR technology and some forms of interactivity, which use different types of sensors able to detect movements of the user’s body or hands, such as the Kinect infrared system, the Leap Motion device or an Arduino board equipped with specific motion sensors [6,7].

This paper aims to show how video mapping, beyond its purely technological aspect, can be linked to cultural heritage and represents a tool capable of becoming a mediator of culture, tradition and legends. In this specific case, it is used to pass on and tell the legend of the foundation of the present Cathedral of Maria Santissima della Madia in Monopoli through the animation of the pictorial cycle by Nicolò Maria Signorile preserved in the church.

2. Related Work

In recent years, numerous SAR applications have been implemented to improve the realization of cultural heritage and restore and enhance the physical memories that have become lost.

The strategic choice of multimedia content and its projection modalities on the surface of the artwork ensures that the viewer is emotionally involved through an audio-visual language that is historically and artistically coherent with the surface on which it is projected.

At the reopening of the Basilica of Santa Maria Antiqua in Rome [8], an exhibition with a series of video installations attempted to resurrect the ancient decorations of the Basilica, by underlining, through digital reconstructions, the paintings of the different surface layers of the wall.

Also in Rome, in 2014 for the celebration of the 2000th anniversary of the death of the great emperor Augustus, the Ara Pacis could be admired again in colour through light projections of the altar. The projection technique makes it possible to modify and modulate profiles and colours in real time in order to render the original colour by projecting directly onto the marble surfaces of the altar and to recreate the total and realistic effect of the original polychrome [9].

Other important video mapping performances took place during the editions of the GLOWFestival, organized to promote, through digital technologies, the revaluation of the cultural heritage in Ostuni, Italy [10].

In Afghanistan, where in recent years the two giants of Bamiyan’s Buddhas, dating back to 1500 and 1800 years ago, have been destroyed, in June 2015 the major Buddha returned to life for one night, thanks to a 3D laser projection of the statue inside the niche in which it was located. This case is exemplary in the demonstration that new technologies as video mapping can be a great resource in the field of artistic and cultural heritage and in their public restitution. Even if projecting images and colours on the surface of an ancient vestige is not equivalent to regaining that asset in its artistic integrity, it facilitates the interpretation of what is no longer visible [11].

As a demonstration of how this technology can be applied to countless surfaces, a video mapping performance in Rome in December 2020 brought Bernini’s Fountain of the Four Rivers in Piazza Navona to life, where the combination of light, colour, sound, and water animation greatly impressed the audience [12].

The Exstasis project, created by the Odd Agency, proves that video mapping does not apply exclusively the external walls of buildings. The project is a 360° immersive projection inside the Church of Santa Caterina d’Alessandria in Palermo (28 and 29 December 2019). Exstasis is a projection show in which the audience, outside the real architectural space, is immersed in an experience of collective transcendence in contact with nature, where technology acts as a glue between the different individuals amazed by the combination of three-dimensional visual effects, light and music [13].

In addition, the possibility of creating forms of hybridisation with further augmented reality systems led to the creation of the Duoreal system for the augmented reality use of the North Frieze of the Parthenon. After the projection of video mapping on the monument, it was possible to activate in-depth multimedia contents using a smartphone and the recognition of specific markers [14].

In July 2019, during the Festas de Lisboa celebrations, a video mapping projection was used for the opening of Vieira da Silva’s temporary exhibition and celebrated the artist’s work by immersing viewers in his artworks [15].

In Paris in April 2018, the Atelier des Lumières was the first centre dedicated to digital art through the creation of “immersive and monumental exhibitions”, in which 140 video projectors and 50 loudspeakers were used to project more than 3000 images of works of art in a 2000 m² space [16].

The NMSPCAM (Alto Minho Cultural Heritage Supported by New Media) project proposed the development of an audio-visual production for a video mapping projection of the city of Viana do Castelo to allow people to learn about the long and rich history of the city, which is mostly unknown even to the local community [17].

In [18], a combination of two technologies was used for the Basilica of Saint Catherina of Alexandria in Galatina (Italy), where one can find one of the most extensive and best preserved late Gothic fresco cycles in Europe. Spatial Augmented Reality was used to tell the story of the Basilica through images and sounds, and Augmented Reality was used to facilitate the reading and interpretation of the most important frescoes located along its internal walls. The aim was to create a unique itinerary that started from the facade of the Basilica and continued into the interior spaces.

A recent example of architectural mapping is the Video Mapping performance projected on the facade of the Church of Otranto, “Mosaic 3D of Life”, where music, immersive fruition and digital art were connected in a journey that illustrated the story of the famous mosaic floor inside the church [19].

The idea for this paper takes its cue from a project carried out by the art collective Skullmapping in 2018 at Brussels Airport on copies of paintings by the artist Peter Paul Rubens. In the painting ‘The Feast of Venus’, the Cupid was animated, coming out of the painting and having fun jumping from one work to another. The idea was to demonstrate how video mapping can achieve spectacular results even in small-scale applications [20].

3. Materials and Methods

The choice of video mapping technology was dictated by the desire to convey a message capable of enhancing a certain type of visual art, such as the paintings in question, but also by the tradition and the historical and cultural context of a community that has found its roots in the legendary tale that led the Madonna della Madia to be elected patroness of the city [21].

The strategy adopted was to imagine an ideal situation in which the main event of the legend is linked to the animation of Mercury, a character depicted in the first painting, who decides to escape from the painting to tell his dream. In this innovative way of telling the story, Mercury accompanies the viewer to discover the story depicted in all four paintings in order to spread a message that educates and entertains at the same time.

It is from a sacred image on which the foundations of the town of Monopoli were set. In the Middle Ages, the townspeople were linked to the legend of a raft landing in the town port, carrying a Byzantine icon depicting the Madonna della Madia. The legend, handed down from generation to generation, has survived relatively unchanged to the present day, thanks to the oral but also visual evidence of paintings, illustrations and writings that guarantee the alleged ‘truthfulness’ of the story that tells us about the foundation of the Basilica of Maria Santissima della Madia [22].

The old temple, which had become too cramped for the large number of inhabitants of the city and also was in poor condition, was demolished to build a new one. The funds, however, were not sufficient to complete the construction of the truss vault, which remained incomplete for about ten years.

At dawn on 16 December 1117, a prodigious event took place: in a dream, the Virgin Mary appeared to Mercurio, the sacristan of the Cathedral, and told him that the beams to complete the roof had arrived on a raft at the city harbour. In the middle of the night the man ran to warn the bishop, who did not believe him. When Mercurio went to the harbour, he saw the huge pile of beams arranged to form a raft, on which was also placed a Byzantine icon of the Madonna of the Madia. This timber made it possible to complete the construction of the church.

The four canvases of the painting cycle made by the artist Nicolò Maria Signorile in 1742 and kept in the Basilica of Maria Santissima della Madia in Monopoli reconstruct four salient scenes of the legend and offer glimpses into eighteenth-century history; the canvases are shown in Figure 1 and underline the importance of Marian devotion in the city.

A preliminary phase of this study involved the collection of documentation necessary to fully understand the methodology to be adopted. The first step was to retrieve the historical documentation for the drafting of the plot and the subsequent creation of the storytelling. It was necessary to acquire an in-depth knowledge of the legend in order to extract the salient moments to be included in the narrative. The strategy adopted was to imagine the escape of a character from the painting in which he was ‘stuck’ and to use him to tell the story depicted in the paintings and to construct the different scenes of the narrative reported in a storyboard.

Figure 2 shows the preliminary phase leading to the definition of the storytelling.

In addition to the choice of images and animations accompanied by sound to achieve a high level of audience involvement and entertainment, the message to be conveyed in these minutes of projection is of fundamental importance.

Therefore, a playful form of communication aimed at learning and knowledge was chosen, based on the concept of edutainment, a way of designing educational applications adopted in the development of serious games and in many applications developed in the field of digital cultural heritage [23,24].

In order to safeguard and protect the original paintings, smaller prints were made. The first phase involved taking a picture of the four paintings affixed on the wall. Then, Adobe Photoshop was used to straighten the picture and correct the optical aberrations; the corrections were minimal considering that the photographic shot was taken using a tripod and positioning the camera at a height such that it was placed centrally and orthogonally to the projection surface.

For the creation of the 3D scene and the modelling of the characters to be animated, the 3D graphics software Blender was used. The models were then imported into Cinema 4D, and the animations were developed within the 3D scene.

The two characters (Mercurio and the Angel) are shown in Figure 3.

The Camera Calibration tag made it possible to animate the characters according to the real position of the pictures. Adobe After Effects was used for the final rendering and the various animations were accompanied by a narrator’s voice.

Finally, the video content was projected on and adapted to the four paintings with the help of Mad Mapper software. In particular, the academic procedure consisted of the calculations of the values of Ansi Lumen, Aspect Ratio and Throw Ratio, which matched the characteristics of the following projector: Optoma ML1050ST+, 1280 × 800, 16:10 (WXGA). This was placed at the height of 1.70 m and at a distance of 3.56 m from the projection surface, centrally and orthogonally to it.

The sound design ensured a greater incisiveness of the actions, making the understanding of the contents richer and clearer.

Some examples of character animation are shown in Figure 4.

In the development of the application (pictures and video are available as Supplementary Materials Figures S1–S6 and Video S1), four macro areas were identified:

• Introduction: a journey back in time represented by the clock with the lancets turning anticlockwise and the current facade of the church, which is deconstructed to make room for the beginning of the story;
• Dream: a beam of light illuminates Mercurio as he sleeps, and the Angel announces the arrival of the raft at the port of Monopoli;
• 3D animation: Mercurio, helped by the Angel, comes out of the painting, and guides the spectator to discover the story depicted in the other paintings;
• Final: in the last painting, a beam of light lingers on the altarpiece, and the narrator tells the story of the construction of the facade.

These macro areas are shown in Figure 5.

4. Results—User Experience

The video mapping performance was submitted to a sample of users to check whether audio and video contents allowed a clear understanding of the legend’s narrative and to collect feedback regarding possible improvements.

A questionnaire was administered to viewers to evaluate the user experience during the video mapping performance. Having found in the literature a lack of questionnaires specifically designed to evaluate a video mapping performance, a specific one was generated for this type of application [25,26]. The questionnaire consisted of eighteen questions with a seven-point Likert scale and an open-ended question (“Other considerations”: not mandatory).

• I liked the experience because it had elements of novelty.
• I liked the type of experience.
• I felt mentally immersed.
• The experience was engaging.
• I think the technique used allows for telling a story in a very effective way.
• The story is engaging.
• The images appeared sharp.
• The animations seemed meaningful and consistent with the story.
• Various measures were used to convey emotions.
• My senses were completely engrossed in the projection.
• The video component of the projection made a strong impression on me.
• The audio component of the projection made a strong impression on me.
• The experience was exciting.
• I was curious to know the rest of the projection.
• At the end of the projection, I felt as if I had returned from a trip.
• Daily thoughts and worries disappeared during the projection.
• I was so focused on the projection that I neglected any external distractions.
• I lost track of time, and the only thing I could think about was the projection.

The test was administered to 50 users, 68% female and 32% male, belonging to an age range between 22 and 63 years.

The visual impact of the contents was clear, and the 3D animation of the characters was particularly appreciated; in general, the narrative was understandable and effective, especially considering that almost nobody knew the legend represented in the pictorial cycle. There was a general enthusiasm both of those who had already seen a video mapping projection (38%) and of those who had never experienced this technology (62%). This enthusiasm is evident in the high level of positive responses (6 = agree and 7 = absolutely agree) to questions asking how much the experience was enjoyed and how engaging it was.

The 18 items of the questionnaire were grouped into four factors, representing Enjoyment (items 1, 2 and 13), Immersion (items 3, 10, 15, 16, 17 and 18), narrative Effectiveness of the video mapping techniques (items 5, 7, 8, 9, 11 and 12) and user Engagement (items 4, 6 and 14). The score of each factor, between zero and six, was calculated for each participant as the average of the scores of the component items, decreased by one.

Table 1. Mean values, standard deviations and coefficients of variation of scores from the questionnaire.

	Mean	Standard Deviation	Coefficient of Variation
Enjoyment	5.040	1.047	0.208
Immersion	4.660	1.140	0.245
Effectiveness	4.957	0.834	0.168
Engagement	5.073	0.917	0.181

shows the mean values, standard deviations and coefficients of variation (i.e., the ratios between standard deviations and mean values) of the scores computed for these factors on a scale from zero to six. In particular, the coefficients of variation indicate whether users’ opinions agree or disagree.

The density ridgeline plots in Figure 6 represent the density estimates obtained for each of the four factors.

The average scores are high for all four factors, as confirmed by the density plot humps, and do not show particularly marked differences. The slightly higher coefficient of variation for Immersion indicates a slightly higher variability of scores for this factor. This is confirmed by the hump of the density plot, which for Immersion appears wider but is still oriented towards fairly high scores.

The final open-ended question provided us with more detailed insight into the performance appreciation; it was pointed out that at times the viewer concentrated more on the images than on listening to the narrator’s voice.

5. Conclusions and Future Work

The objective of this paper was to realise a Video Mapping performance for the Nicolò Maria Signorile painting cycle of the Monopoli Cathedral and to evaluate whether this technology could be a valid tool to be used in the cultural sphere to facilitate the dissemination of traditions and legends and for the valorisation of historical contexts.

We tried to contribute to the dissemination of a specific message that led to the structuring of a storytelling and therefore of a storyboard that told the legend of the creation of the painting cycle in an innovative way.

The project was inspired by the project of the Dutch collective Skullmapping. Technological development, in-depth historiographic research and an accurate study of the bibliographic sources were carried out.

Some difficulties were encountered, especially due to the rendering of the projection because of the particularly dark colours of the paintings and the definition of a specific methodology for testing the performance.

The evaluation questionnaire submitted to the spectators allowed for collecting feedback and useful suggestions for future implementation.