Naturalistic Faces and Faces in Paintings: An Overview

Definition

Faces are the most important social signal in our society. Nevertheless, there is a problem with faces: they are all made up of the same features in the same general order (the eyes are above the nose, which is above the mouth). To process faces one uses a special kind of processing, which is holistic, considering the integration of the face’s features and their relative distances. One may distinguish the recognition of known faces and the processing of unfamiliar faces. Face processing abilities may be lost due to either a lesion or developmental reasons, i.e., prosopagnosia. To further explore these reasons, one could consider pictorial representations of faces—such as faces in paintings. These are particularly interesting because different art styles differ in how realistic/distorted they are relative to real faces, which allows for exploring people’s sensitivity to face-likeness. In a way, individuals are not sensitive to face-likeness. In face matching part–whole tasks, performance does not differ across art styles. Still, individuals are not fully impervious to distortion: early markers of face processing (N170 component) are sensitive to face-likeness, with more realistic (vs. distorted) art styles eliciting responses more in line with those of real faces.

1. Introduction: How Are Faces Processed?

Facial recognition is an essential component of human social interaction. Because of this importance, and unlike common objects that are typically identified at the category level (e.g., “cat”), faces are identified at the level of the individual (e.g., “Mary”) [1].

Humans can differentiate the identity of individuals rapidly and accurately from relatively small differences in facial structure [2]. To explain the effectiveness of human face recognition, a range of evidence suggests that rather than processing facial features independently, information is integrated (holistic processing [3,4]).

Thus, a key characteristic of facial processing is its holistic nature [5,6,7], with minimal engagement in part-based analysis. Instead, there exists a robust integration among facial components. Most faces possess identical features (eyes, nose, and mouth) and consistent gross configurational information (eyes positioned above the nose, nose above the mouth), and different individuals exhibit comparable facial characteristics (e.g., eye color). Nevertheless, most adults can distinguish and recognize thousands of individual faces [8]. This suggests that details regarding specific facial features are not reliable for identifying an individual face. Holistic processing is deemed essential for distinguishing visually similar objects, such as faces, by using subtle variations in the configuration or relationships among different visual features.

Three experimental paradigms are widely acknowledged as standard metrics for holistic processing in the literature on face recognition [9,10,11]. These include the inversion effect, the composite task, and the part–whole task. Inversion effects, where upside-down stimuli are harder to recognize (i.e., recognized with lower accuracy) than upright stimuli [12], are considered an indirect measure of holistic face processing because they demonstrate that faces are processed differently from other objects. While inversion impacts both faces and objects, it has a disproportionately larger impact on faces, suggesting that holistic processing plays a more significant role in face recognition. However, this effect does not directly quantify the nature of holistic processing, as it does not differentiate between differences in face and object processing. There is evidence suggesting that people are less change-blind to faces than to objects. Research employing the “flicker paradigm,” which involves alternating images with minor changes, demonstrates that alterations in faces are identified more rapidly and accurately than alterations in other things, particularly when the faces are upright and displayed alongside various objects [13]. This advantage for faces appears to be associated with their biological and social importance, and it diminishes when faces are reversed or displayed in isolation.

Other tasks are more direct in quantifying holistic processing. In the part–whole task, the perception of a feature (e.g., eyes) is enhanced when it is presented within a whole object (a face) rather than in isolation, demonstrating the facilitative effect of the whole on part processing [3]. This effect has been interpreted to indicate that face parts are encoded not in isolation but rather as integral elements of a larger visual unit which is a (upright) face, improving performance for parts when integrated in the whole face.

A definitive illustration of holistic face processing is represented by the composite effect, which denotes that the perception of a significant portion of a face (e.g., the upper half) is affected by an irrelevant portion (e.g., the lower half). The composite effect is typically implemented using a same–different paradigm, where two faces are presented in quick succession, and the participant must determine whether the target half (e.g., the upper half) is identical in the sequentially displayed faces. Individuals often encounter difficulties when focusing their attention on a single facial feature, which apparently reflects (automatic) attention to all parts.

Do these three tasks measure the same process (or the same aspects of a single process)? Rezlescu and collaborators [10] investigated the interrelations among composite, inversion, and part–whole effects. The three holistic processing measures exhibited minimal overlap, with a significant and moderate correlation identified solely between the inversion and part–whole effects. Consequently, the three tasks represent distinct holistic mechanisms. Ventura and collaborators [14] found that variations in the same composite task (sequential vs. simultaneous presentation) may capture different aspects of holistic processing, necessitating a thorough investigation of these differences. Furthermore, various composite tasks may engage different aspects or subprocesses of holistic processing, including early and experience-dependent holistic processes.

An additional consideration thus pertains to the distinction between early holistic processes and experience-dependent holistic processes. Indeed, the significance of experience in the establishment of holistic processing has been challenged. Zhao and collaborators [15] discovered holistic processing resembling facial recognition for non-facial, novel stimuli without the influence of perceptual expertise. In the absence of training, line patterns exhibiting prominent Gestalt features (such as connectedness, closure, and continuity among components) were processed holistically, similar to faces.

2. The Role of Holistic Processing in Face Processing

Surprisingly, the proposed role of holistic processing in face processing was not tested until 2010, with mixed results. About 15 years ago this association was tested using the composite task and facial recognition. Konar et al. [16] and Verhallen et al. [17] discovered no correlation between holistic processing and face recognition abilities assessed by the Cambridge Face Memory Test (CFMT [18]). Richler and collaborators [19] identified a modest yet significant correlation between holistic processing and CFMT performance.

Rezlescu and collaborators [10] investigated whether the composite task, inversion, and part–whole effects are predicative of face processing, particularly face perception as assessed by the Cambridge Face Perception Test (CFPT [18]). The measures differed greatly in their associations with face perception ability, with correlations spanning from moderate (r = 0.42 for the inversion effect) to weak (r = 0.25 for the part–whole effect) and negligible (r = 0.04 for the composite effect). These results indicate that these tasks do not strongly predict real-world face recognition skills.

Ventura et al. [14] found no evidence that holistic processing, as measured by three composite tasks, predicts superior face processing, thereby questioning the significance of holistic processing in forecasting differences in face processing or in the understanding of holistic processing itself.

The absence of face processing prediction by holistic processing may indicate that holistic processing is highly saturated in most individuals due to substantial experience with faces [20]. This indicates that holistic processing may be a hallmark of a specific category of information involved in facial computations; however, variations within the normal range do not inherently correlate with the efficiency of facial processing. A specific level of experience with faces may suffice for elevated levels of holistic processing.

3. Variations in Face Recognition Ability

Historically, persistent difficulties with face recognition were considered exceedingly uncommon. In the past two decades, there has been an increasing recognition that developmental prosopagnosia is significantly more prevalent than previously assumed. Approximately 2% of the general population report enduring face recognition difficulties that significantly impair their daily lives [21,22].

The growing awareness of these difficulties has stimulated the creation of tools for identifying and assessing face recognition impairments. The CFMT [18] is a recognized standardized objective assessment of face recognition capacity, designed to identify cases of developmental prosopagnosia.

A second instrument designed to facilitate the identification of developmental prosopagnosia is the Twenty Item Prosopagnosia Index (PI20 [23]). This self-report questionnaire was developed to furnish standardized self-reported evidence of face recognition impairments, supplementing diagnostic data acquired from objective computer-based evaluations like the CFMT. The PI20, originally written in English, has been translated into multiple languages, for example, Portuguese [24]. Adults have insight into their face recognition ability, as can be ascertained from the moderate-to-large size of the relationship between questionnaire (PI20) and behavioral measures of face recognition (see, e.g., [24,25]).

Acquired prosopagnosia (AP) and developmental prosopagnosia (DP) are both disorders characterized by significant impairments in face identity recognition. AP arises after cerebral damage, frequently due to stroke, traumatic brain injury, or neurodegenerative disorders, resulting in an abrupt inability to identify individuals by their facial features [26,27,28]. DP is characterized by a significant inability to recognize faces without any identifiable brain damage, cognitive or socio-emotional deficits, or limitations in low-level visual acuity that could explain the recognition impairments [18,26,29]. In contrast to AP, DP is thought to result from the abnormal development of the neural circuitry that supports the face processing system.

Super-recognizers, in contrast to those with prosopagnosia, are individuals with an extraordinary ability to recognize faces, often exceeding the average person’s capacity. They can identify faces they have seen only briefly or have not seen in years. This exceptional ability is believed to be a natural talent, not something that can be learned [30].

4. Expertise with Faces: General or Limited to Familiar Stimuli?

Our ability to recognize faces is not as good as we tend to believe. We are not as good at accurately perceiving unfamiliar faces. We are deceived, for instance, by images captured under varying lighting conditions by disparate cameras [31]. The recognition of familiar faces is significantly more resilient and capable of accommodating such variations [32]. Even the recognition of familiar faces is not infallible. Errors documented in diary studies where individuals note instances of failing to recognize others [33] or in formal studies assessing the identification of photographs of famous individuals [34] indicate that lapses can occur even with well-known faces, although these errors are typically rectified promptly.

Bruce and Young [2] propose that face processing is accomplished through distinct functional components. Following a perceptual analysis of the face, it is suggested that facial expression recognition, facial speech movements (which contribute to speech perception), directed visual processing (to ascertain attributes such as age and sex from facial appearance), and face recognition are largely independent and occur concurrently. In face recognition, a more detailed description of the process is provided, encompassing a sequence that begins with the recognition of a familiar face, progresses to accessing identity-specific semantic information, and culminates in the retrieval of the individual’s name.

Young and Burton [32] contrasted their findings on the importance of familiarity in face identity recognition with the notion that humans possess expertise in face identity recognition more generally (cf. [35]). They advocate for a distinct definition of expertise, positing that it necessitates experience, leads to precise responses, and operates automatically. Their discussion is grounded in the aforementioned studies indicating that individuals are not as proficient as one might assume when making judgments about unfamiliar faces, whether in experimental settings or real-life scenarios. This led Young and Burton [32] to conclude that we have limited expertise with unfamiliar face perception and thus can only truly be considered experts with familiar faces.

The question of what happens when we learn a face under diverse visual conditions and its association with semantic information is important. Nonetheless, addressing such questions will not diminish the fact that experience also affects the processing of unfamiliar examples (e.g., [36,37]). In general, although investigating familiarity effects is a valuable endeavor, it may underestimate the expertise we apply when recognizing the identities of unfamiliar faces [35].

Gauthier and colleagues (e.g., [35]) suggest that faces are special because we have become experts at making intra-categorical discriminations. As mentioned in the initial introduction, faces require intra-categorical discrimination (between one face and another), whereas the visual recognition of most other things requires inter-categorical discrimination (e.g., between a cup and a saucer). We have become experts at making these intra-categorical discriminations because of our vast experience with thousands and thousands of examples.

This intra-categorical discrimination also occurs for other objects for which people develop expertise. For example, expertise in discriminating three-dimensional objects, “Greebles” [37], involves a shift from part-based to holistic processing and the activation of the fusiform area for faces (FFA [38]), which also occurs with other types of intra-categorical discrimination (e.g., as seen in car experts [39]). The FFA in the brain is strongly associated with face processing and shows a high degree of selectivity for faces over other objects. While the FFA is also activated by other objects in expert individuals (e.g., cars in car experts [40]), the initial activation and response are often stronger and more specific to faces. This suggests a specialized neural architecture for face recognition, indicating a level of specialization beyond general object recognition.

In the middle fusiform gyrus, faces and words engage highly specialized sites, the fusiform face area (FFA) and the visual word form area (VWFA), respectively, the VWFA typically (but not always) being lateralized to the left and the FFA to the right hemisphere [41]).

Studies with evoked potentials indicate that the magnitude of an early component, referred to as the N170, is significantly larger when participants are viewing faces than when they are viewing objects (e.g., [42]). In a study by Tanaka and Curran [43], bird experts show an N170 component for birds, and dog experts show an N170 component for dogs. It thus seems that expertise with faces extends to other objects for which we are experts in making discriminations (Greebles, bird and dog experts, car experts, etc.).

5. Seeing Beyond the Face: Research with Faces in Paintings

But can we study face perception using stimuli other than real faces (e.g., in photographs)? Importantly, the answer could be ‘yes’: as long as the visual stimulus shares enough resemblance with actual faces (in their shape, features, relationship, etc.), face processing mechanisms could be activated for non-face stimuli on account of this similarity (or “face-likeness”).

In line with this, drawings of faces—from realistic portraits to schematic representations—have been shown to bring about neuronal activation overlapping with the activation that real faces prompt [44], as well as comparable behavioral responses [45]. The same is found for Arcimboldo faces, which are faces resulting from the combination of non-face objects (e.g., a face resulting from over-imposed fruits [46,47]). In fact, this type of facial representation is sensitive to inversion, such that the corresponding activation is diminished when Arcimboldo faces are inverted [48]. This suggests that they are not only recognized as faces, but they also mobilize holistic processing mechanisms usually observed for regular faces. These conclusions extend to faces with some degradation, such that inversion effects are also observed for distorted faces (e.g., Mooney faces [49]). All in all, there is converging evidence that non-face objects that represent faces are still picked up as faces, both in neuronal and behavioral terms.

Still, human face recognition capabilities are quite sensitive to stimulus degradation (or even decreases in the amount of face information accessible), namely resulting from lighting conditions or viewpoint presentation [31]. Representations of faces are arguably also subject to these impacts, and there may be a point past which the visual processing system does not interpret them as faces. Faces in paintings constitute a particularly useful category of face representations in this regard. Faces in paintings usually follow a set of rules regarding brush stroke, color implementation, thematic focus, etc.—which together comprise an art style. Some art styles are more oriented towards preserving the characteristic of actual faces, or replicating them in the paintings, than others. Additionally, and within one same art style, there may be substantial differences. Thus, paintings can depict faces in ways that are aligned with the features of faces that individuals naturally come across in their environment (i.e., more realistic faces) or instead represent faces in ways that depart more from actual faces (e.g., via the introduction of distortion to their features).

Studying faces in paintings can therefore advance our understanding of two primary questions of interest in the face perception literature: (1) To what extent are representations of faces similar to actual faces (e.g., in photographs), in terms of the mechanisms and resources involved? (2) If these mechanisms overlap, what features of the pictorial representation of the face (e.g., face-likeness vs. distortion) establish whether face processing mechanisms are recruited or not? These questions allow us to be informed about how real faces are processed, by identifying when and why a certain face-like stimulus instills face-like responses.

6. Faces in Paintings Are Processed Like Real Faces Across Art Styles

In a sense, individuals have been shown to be insensitive to face-likeness, treating representations of faces as real faces regardless of their distortion. That is, representations of faces are treated as faces in an all-or-nothing fashion and processed as faces if they pass a minimum threshold that allows them to be processed like a face (after which being more or less face-like bears no impact on how they are processed).

Ventura and collaborators [50] conducted a part–whole face recognition task with faces in paintings to test whether they activate the holistic processing level. In addition to faces in photographs, we included paintings from four art styles: cubism, post-impressionism, expressionism, and renaissance. These range from the least to most realistic in the order described (or from most to least distorted) in that the former comprise geometrical shapes and distorted colors included to generate emotional responses, whereas the latter seek to represent faces with realistic facial expressions and features.

In each trial, participants first saw a face, followed by two stimuli presented simultaneously, which could either be features (e.g., two mouths) or full faces. One was a lure, and the other corresponded to the face (or one of its features), and participants had to indicate which of the two stimuli they had just seen. After completing the part–whole task, participants performed an abstract art memory task (AAMT [51,52]) and measures of interest in art (VAIAK [53]).

Recall that part–whole recognition tasks are direct markers of holistic processing [3], a central mechanism in face processing [5,6,7], as it is commonly found that part identification is improved when included into a face (with all of its features vs. presented in isolation)—for whole over part trials. Interestingly, the effect of trial type did not interact with the art style of the stimulus, and performance was better for whole trials across art styles, in line with the results obtained for photographs (argued to reflect holistic face processing; see the Introduction: Holistic Face Processing Section). Thus, not only are faces in paintings processed holistically across art styles, but they are also processed to similar extents. Furthermore, interest in art and general recognition abilities correlated negatively (as interest probably presupposes focusing more on isolated features) and positively (as holistic processing is associated with better recognition abilities) with holistic processing.

This work therefore suggests that faces in paintings are perceived as faces, regardless of whether they closely mirror their characteristics or not.

7. Still, in Some Respects, People Are Sensitive to Face-Likeness/Distortion

The results above are striking to the extent that they show that even representations of faces that preserve minimal resemblances to real faces are treated as such. Consider the case of cubism paintings—though the faces in these paintings are akin to faces in photographs in that they share a set of features (e.g., eyes), their pictorial representation is often simplified (e.g., turned into geometrical shapes) and their relative positioning heavily modified (e.g., eyes not aligned and/or overlapping the nose). Thus, it is prudent to consider whether individuals are truly fully insensitive to the extent that face representations are distorted or realistic.

Ventura and collaborators [54], directly tapped into the neuronal activation patterns that underline face processing. Though behavioral performance mostly aligns with the idea that faces in all art styles are processed the same, Ventura and collaborators [54] explored early neuronal responses associated with face perception which precede them. Concretely, event-related potentials associated with early face detection responses—the P1 (i.e., identification of a face as such based on low-level properties [55,56,57]) and N170 (i.e., higher-level integration that activates holistic processing [47,55,58,59] components were measured.

Participants were instructed to pay close attention to a set of stimuli presented on the screen and had to press a key when the stimulus was a butterfly. The butterflies were intermixed with other stimuli, which differed across blocks. One block comprised visual non-face shapes (Ziggerins), and all other blocks contained faces—faces in photographs and in paintings (renaissance, post-impressionism, expressionism, cubism) alike. EEG activity was measured at 64 active electrodes.

The P1 amplitude did not differ for most art styles (except for cubism), indicating that faces in paintings are generally quickly identified as faces even when they are somewhat distorted. The following N170 signal, however, displayed a gradient in line with the decrease in realism: faces in renaissance evoked responses similar to those of photographs of faces, and these responses were larger in amplitude than those elicited by the other art styles, which progressively diminished from more to less realistic art styles. This suggests that neuronal responses elicit face-congruent responses as a function of the stimuli’s face-likeness. Bullot and Reber [60] propose a “psycho-historical” research program for the integrative science of art, arguing that understanding art appreciation requires considering both psychological and historical factors. They critique traditional psychological approaches to art appreciation for neglecting the historical context in which artworks are created and viewed. Ventura and collaborators [54] results do not solve the question of how much today’s lens can tell us what those contemporary to the painting comprehended (cf. [60]). However, it tells us about the biology of perception between natural scenes and visual art and across a time course.

8. Bringing These Findings Together

For the most part, it appears that initial face identification responses emerge for faces in paintings. This is observed behaviorally (e.g., in part–whole tasks [50]) and also when measuring neuronal activity (i.e., P1 potential [54]). As face processing stages unfold, however, it appears that realism/distortion comes into play. Evoked potentials associated with higher-level representation integration (i.e., N170 response [54]) exhibit gradation as a function of art style—treating more realistic faces as akin to faces but distorted ones as non-face objects. All in all, the threshold for labeling a face-like stimulus as a face may be low (e.g., reliant on the identification of features or broad configurations), and as face processing unfolds, the response becomes more stringent (as they move from lower-level to integrated representations). Future research could address the temporal dynamics of setting apart faces in paintings from faces in photographs, exploring the limits of when the former are treated as the latter, and concretely regarding the holistic processing that is a hallmark of faces (and other object categories of expertise).

9. What More Can We Learn from Faces in Paintings?

Though illustrative of the potential contribution that exploring the processing of faces in paintings may have for our understanding of naturalistic face perception in general, they constitute merely an example of what could be studied in this regard. That is, further adapting paradigms used in the context of face perception could allow us to better map the extent to which faces in paintings are similar to real faces, contributing to our knowledge about what is “special” about the latter stimulus category (if anything).

For example, faces in paintings could be used to characterize specific aspects of holistic processing, namely the role that attending to the relative positioning of the features (termed relational/configural processing) plays in it. Individuals are generally good at capturing changes to the configuration of a face when they are presented as faces that normally appear in the environment (but not when inverted) [61,62,63]. In line with this, one could assess the extent to which faces in paintings are recognized as faces by exploring whether the awareness of changes in face configuration (e.g., spacing of features) is affected by the stimuli’s orientation differently for more and less realistic faces in paintings, for example.

There are future directions worth pursuing at the neuronal/implementational level. It would be interesting to explore whether the functional areas in the brain associated with the processing of natural faces are equally recruited for faces in paintings. In this regard, simple behavioral tasks could be particularly informative. Face processing displays a right hemisphere advantage, in that recognition performance improves when faces are presented in the left half of the visual hemifield [64], in tandem with increased activation in the opposite hemisphere [42]. Previous researchers had already explored lateralization in faces in paintings [65,66] (also focusing on art appreciation [67]), but here it is proposed, using faces in paintings across art styles, along the above-mentioned realism–distortion continuum, to explore the extent to which faces in paintings also benefit from being presented in the left half of the visual field and, more importantly, whether this is modulated by the extent to which these pictorial representations are realistic.

Finally, it would be important to dedicate more attention to the idea of “face-likeness” and to test whether other face-like depictions are similarly perceived. Sculptures would constitute an interesting case study. On the one hand, sculptures are similar to paintings in that their characteristics also vary substantially as a function of art style, in a realism-to-distortion continuum. On the other hand, sculptures are more similar to real faces in that they are tridimensional and thus contain visual aspects that are factored in by face perception systems (e.g., depth) that may be absent, be less informative, or be more difficult to parse in paintings.

10. Conclusions

Given their importance as social stimuli, it is only natural that dedicated mechanisms optimized for perceived faces have emerged, namely because face stimuli display regularities (i.e., similar global structure) that would make face recognition challenging otherwise. In the present work, the literature on said mechanisms was reviewed—such as holistic processing (i.e., the effortless integration of face features and their relative distances in one same percept)—and explored aspects that affect their deployment (from stimulus familiarity to the presence of lesions). Additionally the case for the importance of considering pictorial stimuli that share visual resemblances with naturalistic faces—such as faces in paintings was built. While considering these stimuli may help uncover boundary conditions pertaining to when and how the visual system recognizes a face as such (e.g., based on its face-likeness vs. distortion), this research endeavor is still in its early stages, and its potential can only be unlocked with the development of more research on the topic.