Museums have long been a source of education, information and recreation to the general public. In the United States, museums contribute over $
50 billion to the U.S. economy [1
] with 66% of leisure travelers willing to explore local art or cultural environments [2
]. Therefore, it becomes important to continuously learn what keeps visitors motivated to go to museums, what makes them curious about what they see and what makes them satisfied with their visits. Since the launch of online video portals in 2005, many museums have encouraged the use of platforms such as YouTube in delivering museum experiences to online viewers in an attempt to reach larger audiences, increase awareness and loyalty, and most importantly, bring new forms of public engagement [3
Studies specific to understanding museum visitors have spurred interest among curators and researchers since the 19th century, and museum professionals are always looking for new ways to meet these expectations. Phelan [5
] provided an excellent overview of self-determination theory in explaining how museums meet one’s psychological needs for competence, autonomy and relatedness [6
]. If these needs are met, then one may argue that visitors will be motivated to learn. To support one’s need for competence, museum professionals should make the design of their exhibits fun and easy for visitors to navigate freely [7
]. Most museums offer free-choice learning, where navigation is guided by a visitor’s needs and interests [8
]. While this is important to satisfy one’s need for autonomy, curators or exhibit designers should consider how to make use of a wide range of design options in presenting exhibits to visitors. The need for relatedness can be satisfied through visiting the museum with friends or family members, or by offering the visitor opportunities for social interaction with others [10
There is a large body of literature that explores the role of architectural design and exhibits on visitors’ engagement and emotions using observations, surveys, focus groups or interviews. Many of these studies focus on museum layouts and the position of exhibits within galleries. Known as space syntax, scholars offered tools to identify spaces within the museum that gave exhibits higher levels of co-visibility, co-awareness and co-presence, thereby, increasing the probability of engagement [11
]. Numerous other studies look at the role of narrations in creating a fertile area around exhibits and offer new dimensions to their meanings. However, these studies may be too restrictive in understanding the meaning of space [14
]. In a well-cited study, Smith and Smith [15
] looked at visitors coming alone and visitors coming in groups. They documented that the interaction between visitors generated a social event while experiencing an art exhibit. Others went in-depth in examining the role of architecture, especially those designed by signature architects such as Gehry, I.M. Pei or Hadid [16
]. Further studies focus on the impact of adding sound to spaces and exhibits [17
] or enhancing visitors’ experiences through the use of virtual or mixed reality technologies [18
]. Most recently, studies show how the use of QR codes in delivering information can maximize visitors’ engagement [19
To capture a higher resolution of data on visitors’ preferences, Rashed et al. [20
] used a LIDAR pole human tracking system installed at shoulder level at a museum. This technology enabled the researchers to create heat maps representing four different visiting styles: The Ant visitor spent more time around all exhibits, the Fish visitor moved mostly around empty areas, the Grasshopper visited certain exhibits while skipping others, and the Butterfly visited most exhibits but not with an equal amount of time. Such information helps curators know which exhibits were preferred over others. It also helps curators understand how to arrange exhibits to avoid congestion.
Tseng et al. [21
] used a different technology to capture visitors’ emotional reactions to exhibits. They developed an eyewear system and a microrecorder and asked six participants to wear the glasses during their visit. They were able to capture the visitors’ eye movements through wearable glasses with an attached camera and connected the eye movements to the exhibit being viewed by the visitor. Once data were extracted, participants were asked to explain the moments that caused their eyes to blink within the context of five experiences such as feel, sense, think, act and relate.
While these studies offer valuable insights to museum architectural designers and curators, the findings may only be applicable to in-person visits. To date, there has been insufficient literature that examines the behavior and motivations of visitors of museums online [4
]. The purpose of this qualitative study is to analyze the impact of online videos of different museums on emotions using facial expression detection technology. The results of this study will help museum curators understand the impact different exhibits have on emotions and will help them understand what key architectural features attract and engage visitors. The results of this study will also help architects and exhibit designers understand the role their design has on users before executing the designs. Therefore, the study questions are: (1) To what extent does viewing museums via online videos impact a viewers’ emotions? (2) Can these emotions be captured accurately using facial expression detection technology? And (3) Using what is learnt from literature review, how similar are emotional reactions to in-person visits?
2. Literature Review
Museum visitors do not have infinite time to visit each exhibit. They visit museums of their free will and, therefore, there is an expectation that they will see, move, navigate and experience exhibits in their own way. This literature review is organized around three themes that address visitors’ engagement, reactions, and expectations. First, it presents a review of major design principles related to how exhibits and architectural features impact visitors’ engagement. Second, it summaries previous work that evaluated the effectiveness of these principles. Third, it outlines previous work that measured specific visitors’ emotions and how they impacted their satisfaction level with their visits.
Alt and Shaw [22
] argued that it is important to classify museum exhibits in terms of their effectiveness on visitors. they generated and tested 53 attributes on participants who just visited a museum. They later examined these attributes on 45 exhibits and a total of 1980 subjects and were able to reduce the list to 23 attributes. Examples of these attributes included an overall evaluation of the exhibit such as “It presents information I’m already familiar with,” the attractiveness of the exhibit such as “Your attention is distracted from it by other displays,” an evaluation of the subject matter such as “It is a difficult subject to understand,” an emotional reaction such as “It is entertaining,” the visual effect of the exhibit such as “It looks like something of science fiction,” and the clarity of the exhibit such as “It is confusing.”
Another major contribution to museum visitor studies was Perry’s [23
] development of six guiding principles based on visitors’ psychological needs. Perry found visitors expected to see objects that triggered their curiosity, increased their level of confidence in their ability to learn something new, challenged them when figuring something out, engaged their sensory experience, and encouraged their communication with others through exchanges of ideas. Perry [24
] tested these principles and found that they increased visitor engagement with exhibits and helped museums better understand how to design exhibits that suit visitors’ psychological needs.
Interestingly, Forrest [25
] proposed borrowing the concept of atmospherics in understanding the role of environmental psychology on visitors’ experiences. More specifically, the author borrowed the stimulus-organism-response (S-O-R) model to explain how the sensory inputs such as lighting, layout or color (stimulus), although depended on personal factors or past experiences, triggered emotional reactions such as pleasure or arousal (organism). This in turn produced behavioral outcomes such as approach or avoidance (response). This study shed further light on explaining the possibility that emotions do matter in understanding whether the visitor was curious or engaged, or would visit the exhibit again.
Building on prior works, Skydsgaard et al. [26
] argued that the use of design principles may enhance visitors’ reflection and discussion. They examined four principles of design in museums developed earlier by the Steno Museum. These were curiosity (such as new knowledge or connecting new information to existing knowledge), challenge (such as exhibits that evoke emotions such as taboos), narratives (such as using narrative devices that stimulate reflection) and participation (such as physical interactivity that requires bodily interaction and dialogic interactivity where visitors share ideas with others). The research team developed a questionnaire and distributed it to pupils from four schools. Ninety-eight participants gave feedback through the questionnaires. Although their study showed that elements of curiosity and challenge were what attracted visitors the most, these four guiding principles were helpful in creating the desired experience among visitors to keep them engaged.
Museums have recently changed the way they deliver experience to visitors. Over the last decade, there has been more focus on presenting objects to visitors and a shift from product orientation to consumer orientation [27
]. Therefore, it becomes reasonable to understand the motivations behind visits to the museum, what impacts visitors’ emotions, what makes them satisfied with the visit and what makes them return [28
]. Visitors who expressed positive emotions from their visits were more likely to be satisfied [29
]. Learning more about the specifics of these emotions could be helpful in enhancing the experience [30
] since people are more influenced with the emotional values associated with objects—an area that still needs to be teased out [31
]. Characterized by their intensity [32
], emotions were usually formed as a response to consumption of an experience (e.g., seeing an exhibit) and may be measured by core emotions such as joy, anger, disgust and fear [33
] or by bi-polar state of emotions such as pleasant/ unpleasant or calm/excited [34
Due to lack of research on the role of emotions as shaped by experiencing objects or exhibits, Del Chiappa et al. [27
] analyzed the behavior of 410 people who visited the National Museum of Archeology in Sardinia. The questionnaires assessed their sociodemographic data, their emotions and their overall satisfaction. Visitors also participated in focus groups to assess their emotions further. Findings confirmed that emotions were directly linked to visitors’ overall level of satisfaction with the museum. The authors argued that considering emotions in future studies examining visitors’ behavior was important to understand what increased their level of satisfaction.
Although studies around emotions were limited, other strands of museum visitor research selected to focus on specific elements that were believed to impact visitors’ emotions. One such example was integrating sound into exhibits to captivate the sensory experience of visitors. In historical museums, listening to sounds from the past to represent a certain episode was found to be more immersive as it lifted visitors out of reality and magically transported them to a different time period [17
]. Working with the Danish Museum, Hjortkjær (2019) tracked what visitors remembered the most from their visit to the historical display through user surveys. Almost every fourth respondent referred to the different sound installations as the highlight of their experience. This study, however, captured one of many elements that could play a role in one’s own emotions.
Another strand of research looked at the power of social conversations on visitors’ engagement [36
] and how exhibits create such interactions. Achiam et al. [14
] recorded the vocalization of 12 visitors who visited two dioramas representing the ice age and a discovery room with objects that had no labels. The authors transcribed the dialogues between participants verbatim to capture their thought process as they interacted with the exhibits rather than capturing their assessments using post-hoc questionnaires. They discussed the findings within the concept of perceived affordances and how the unlabeled objects in the discovery room were successful in constructing different narratives depending on the visitors’ prior knowledge and experience.
Although numerous studies showed that visiting museums is largely social and most visitors go with friends or family members, one recent study surveyed 100 users of online platforms and semi-interviewed 10 of these users [4
]. Their findings showed that viewers were motivated to visit museums through online video platforms because they were well-designed, immersed and interestingly, identified themselves with the author of the video. The educational part of the online museum visits also allowed viewers to escape reality by exploring something different and manage their moods. Although they were not accompanied by friends or family, most respondents expressed their feeling of belonging to a community who share similar interests by being able to use the comment section [4
Although there is insufficient literature that examines the impact of visiting museums online on the viewer’s emotions, this study will test what is known from visiting museums using a facial expression analysis software.
In this pilot study, the authors reviewed over 100 online video tours of different museums captured by visitors. The authors qualified 14 museums because the videos were professionally taken, known for their unique architectural features, captured enough architectural details and offered high resolution close ups of the exhibits. All 14 video clips had a minimum of 13,000 views and a maximum of 2 million views between 2016 and 2020. Although the authors acknowledge that these videos were not immersive since they did not capture important details such as smell, the video clips had similar qualities recommended by Gladysheva, Verboom and Arora [4
Other criterion included in the selection of the videos was to focus on two types of museum: art museums and dark museums. Art museums were selected because this category is not restricted to a geographical boundary. This allows the researchers to test art museums from different countries to focus on the space and the display of the exhibit. Paintings like those of Van Gogh or Monet are good examples of art pieces that are common across different art museums. Dark museums were selected because they were designed to trigger emotions.
The 14 selected museums representing art museums and dark tourism museums are as follows: the Tate Modern Museum in London, the Ulucanlar Prison Museum in Ankara, the Museum of Modern Art in New York, the Eastern State Penitentiary in Philadelphia, the Eli and Edythe Broad Art Museum in East Lansing, Alcatraz Penitentiary in San Francisco, the Ukrainian National Chernobyl Museum in Kiev, the Istanbul Art Museum in Istanbul, the Guggenheim Museum in New York, the Guggenheim Museum in Bilbao, the University of Michigan Museum of Art in Ann Arbor, the Toledo Modern Art Museum in Toledo, the Sinop Prison Museum in Sinop and the 9/11 Memorial and Museum in New York. Table 1
summarizes the types of these museums and indicates whether they were visited by the participants prior to the study.
After qualifying museums for this study, video clips were downloaded into an appropriate format. These short clips were then identified as stimuli
inside a software called iMotions. iMotions is an advanced software that has the capability of analyzing 7 core emotions and 20 facial expression measures. Recent research tested the reliability of this software in detecting emotions by comparing its performance to other similar facial expression detection software [37
]. In their research, Kulke et al. [38
] compared the recognition software with EMG measurements of the zygomaticus mayor and corrugator superclii muscles—the muscles responsible for emotions like happiness and anger. Results showed that the values correlated highly.
The iMotions software computes different emotions based on analyzing the facial expression of the participant. When the participant views a stimulus (in this case, the video clip), the software captures the main facial features of the participant while looking at the clip through a high-resolution camera attached to the computer. The distance between the eyebrows, the distance between the nose and the upper lip, and the position of the chin are all examples of how the software computes the different emotions. If the angles of the lips are pulled up in certain ways along with other facial features because of a reaction from the zygomaticus major muscle that is responsible for the smile, then that is an indication of joy. The algorithms built into the software to register each of these emotions were computed based on scientific research on movements of facial landmarks and other types of muscle movement. The software registers these different emotions numerically and graphically (see Figure 1
). The graphic output is similar to that of a heartbeat rhythm strip displayed on an ECG monitor. Each emotion has a strip of waves. Changes in facial expressions as a response to a stimulus in the video clip will cause the wave to peak (short, sharp peaks or long, curvy peaks). Within the context of museums, these peaks may correspond to an architectural space that suddenly opens to the view of the participant or an artwork with a high level of saliency (see Figure 2
In this research, each museum clip was identified in the software as a stimulus. The investigators of this study were the two participants. The researchers purposefully used self-experimentation technique to be able to verify and explain each resulting emotion and reaction. Researchers within similar fields advocated for the use of this approach because when the experimenter becomes the subject of the study, it helps test new concepts and formulate new ideas [39
]. A recent review of self-experimentation technique showed how this method had led to notable, including Nobel Prize, discoveries [40
The decision to proceed with self-experimentation for this research was for the following reasons: First, the total viewing time requires a time commitment of about 4 h from each participant to watch and analyze each video. Second, the authors spent additional time to review the accuracy of all 2035 emotional peaks resulting from the stimuli. Third, this self-experimentation also allowed the authors to discuss, verify and code the results produced in iMotions software while following inductive and thematic analysis as proposed by Strauss and Corbin [41
]. Were the emotional changes a response to seeing an architectural space? Were they the result of seeing an exhibit with certain features? Or were they the result of the subjective experience of the participant?
The software created a timeline that registered peaks of different emotions while timestamping the exact second in the video clip that created the change in emotion. The timeline for the two participants for each stimulus (video clip) was identical. This was helpful because it allowed the researcher to compare how similar or different emotional reactions were between the two participants. It was also helpful for the purpose of this research to select an existing video clip because the path remained constant for the participants as opposed to having a self-guided walk through.
In this study, there were 302 identical emotional reactions, and 1733 unique ones. The authors applied open coding techniques with thematic groupings and redundancy checks on all 2035 emotional peaks, and this was repeated until the authors agreed on conclusions [41
]. The researchers continued to analyze and code the emotions until a theoretical saturation was completed. The three main themes generated were the architecture of the space, the exhibits, and the subjective background of the visitor.
After compiling the emotional reactions from watching the different stimuli, the authors created a table with rows and columns (Table 2
). Rows represent the three major categories: architectural space, exhibits, and subjective background of the visitor while the columns represent eight different emotions (anger, joy, disgust, contempt, fear, sadness, surprise and engagement). Further analysis and use of axial coding techniques was undertaken to explore connections and linkages between codes. This step enabled the researchers to identify subcategories that were eventually connected to each of the three main themes. These new subcategories represented something very specific that demonstrated a pattern across all 14 museums.
The aim of this study was to explore whether viewing museum spaces via online video portals would affect virtual visitors’ experiences the same way it would in person. Using a facial expression detection software, it was possible to expand on the review of literature and elaborate on how spaces and exhibits triggered human emotions in museums. Literature on visitors’ studies demonstrated strong links between visitors’ engagement and exhibits that used sound [17
]; narration [26
]; sensory stimulation like light and color [25
]; attractiveness or complexity [22
]; and curiosity and challenge [24
]. Results of this pilot study showed that these concepts were also applicable to digital museums. Moreover, the findings extended our knowledge on how the architecture of space had more impact on one’s own feelings of sadness, surprise and engagement while the exhibits impacted other emotions, such as fear and joy. Anger was shown to be triggered based on one’s own background. Not surprisingly, dark museums had a higher impact on emotions like anger and sadness.
When exploring the impact of space on the participant’s experience, new themes emerged that enabled the researchers to create an inventory of different architectural elements that were most impactful. To summarize, long and narrow spaces, curved hallways, and complex spatial layouts represented spaces that triggered a sense of fear, perhaps due to their claustrophobic effects. Second, in prison museums, the visitation space had an equal impact on emotions. Giving attention to such spaces may offer opportunities of higher engagement. Third, it was an expected finding that architectural lighting played a significant role in shaping one’s emotions. Dark spaces evoked mystery and fear, while bright spaces gave further emphasis to the elements being exhibited. Curators may benefit from integrating architectural lighting around certain exhibits. If the layout of the museum is complex, then lighting may serve as a landmark to help visitors navigate and distinguish spaces or galleries with different themed exhibits. Visitors can easily be overwhelmed with the complexity of the museum layout and the number and positions of exhibits within the space. If the museum is housed in an existing space, such as in the case of a prison museum, a careful play of lighting might mitigate those experiences. However, maintaining the integrity of the original finishes of the spaces will enhance one’s experience. In sequential galleries, moving from one gallery space to another might make one nervous or curious about what they will see next. Therefore, the width of the opening, the careful positioning of exhibits within sequential gallery spaces and the use of contrasting wall colors will all keep one curious, engaged, and in some instances, full of joy.
The findings also recognize that every museum type has unique exhibits. This pilot study showed that the display of newspapers demonstrating historical events had a significant impact on emotions. Consistent with the findings of Hjortkjær [17
], amplifying spaces with sound can also intensify these emotions. Another instance was displaying one’s belongings from the past. Whether it was old furniture or personal clothes, these types of exhibits always pique interest. In the prison type museum, fully furnished prison cells with inmates’ belongings had more impact on emotions than empty ones. This finding was not surprising, and a simple image search about these museums on Google, TripAdvisor or Instagram will show how most photos shared by visitors are about belongings. Exhibiting specific objects from the past had a powerful way of narrating those events and reaching out to visitors. When it comes to the exhibit itself, whether it is an artwork or a 3D exhibit, its scale, color, texture and ratio with the space were all important factors in drawing visitors into the space. Museum visitors always connect what they have seen to their past experiences. Therefore, each visitor’s experience and interpretation of the exhibition will be subjective.
6. Conclusions and Limitations
In conclusion, the findings validated the use of iMotions Software in capturing the correct emotions within the context of museums. After coding all the emotional peaks, the researchers were able to understand the qualities of the architectural spaces and exhibits that may have the most significant impact on a virtual visitor’s experience. The results of this study can help create impactful museum videos and help architects, interior designers, curators, and exhibition designers understand the qualities that keep museum visitors engaged and curious. Additionally, the use of this software makes it possible for architects to test computer generated designs of buildings and exhibits prior to execution. Using facial expression recognition software may also give instant feedback to curators before putting up an exhibit.
This study has some limitations. The authors acknowledge that this study was self-experimentation that was based on the investigators’ observations. However, because timing was not an issue, it did allow the authors to qualitatively analyze all the emotional reactions following Strauss and Corbin’s approach in inductive and thematic analysis [41
]. The second limitation was not being able to capture senses like smell and touch through post-video analysis. The authors acknowledge that visiting museums online will never replace the experience of visiting museums in person. The quality of the colors of the artwork is also dependent on the technical specifications of the viewer’s screens. As Mintz (1998) puts it in her article Media and Museums: A Museum Perspective
: “…On a video screen, a painting has no texture; something is always lost when three dimensions are collapsed into two. Distinctions of scale disappear; a tiny Renoir and a gigantic David are the same size. The electronic palette cannot perfectly match colors in the real world [43
].” Additionally, since most visitors visit museums with family and friends, viewing museums through recorded videos limit such interactions to texts via the comments section. Nevertheless, opportunities for future studies could be extended to include the impact of social interaction during virtual visits. Lastly, the authors also acknowledge that pre-recorded videos do not extend the same autonomy that museum visitors have during their free exploration [8
]. Since many museums are shifting their attention to integrate technologies that enhance visitors’ experiences, technologies like virtual reality or eye-tracking systems can easily address these limitations and can easily be analyzed using facial expression software to further support their studies.