Sign Language Anonymization: Face Swapping Versus Avatars

Abstract

The visual nature of Sign Language datasets raises privacy concerns that hinder data sharing, which is essential for advancing deep learning (DL) models in Sign Language recognition and translation. This study evaluated two anonymization techniques, realistic avatar synthesis and face swapping (FS), designed to anonymize the identities of signers, while preserving the semantic integrity of signed content. A novel metric, Identity Anonymization with Expressivity Preservation (IAEP), is introduced to assess the balance between effective anonymization and the preservation of facial expressivity crucial for Sign Language communication. In addition, the quality evaluation included the LPIPS and FID metrics, which measure perceptual similarity and visual quality. A survey with deaf participants further complemented the analysis, providing valuable insight into the practical usability and comprehension of anonymized videos. The results show that while face swapping achieved acceptable anonymization and preserved semantic clarity, avatar-based anonymization struggled with comprehension. These findings highlight the need for further research efforts on securing privacy while preserving Sign Language understandability, both for dataset accessibility and the anonymous participation of deaf people in digital content.

1. Introduction

Sign Language is the primary means of communication for the deaf community, which comprises about 5% of the population. Being a visual language, it requires special attention to body movements and facial expressions. Providing a system that facilitates and aids communication between the deaf and hearing communities is urgent. In the last decade, with the advent of Deep Learning (DL), research efforts have increased related to translation between Sign Language and spoken languages, but much larger datasets are required for training DL models. However, there is some reluctance to contribute to data collection [1] due to privacy concerns about identity exposure in Sign Language datasets.

In this work, we analyze different techniques to address the challenge of Sign Language data anonymization, aiming to change the appearance of the subjects shown in the videos, while maintaining the original content being signed.

Anonymization of signers allows those contributing to a dataset to conceal their identity, without compromising the meaning of the information they convey. The process of visually anonymizing Sign Language datasets has resulted in a boost in openness to participate in data sharing, largely attributed to the elimination of signer-specific appearance [1]. Alternative approaches such as pixelation [2], blackening [3], and grayscale filters [1] have been tried unsuccessfully because, during sign execution, they affect the facial and body information that is strictly necessary to provide the correct meaning of the message.

The objective of this study was to evaluate the impact of anonymization techniques, specifically through the use of realistic avatars and Face Swapping (FS), on the quality and intelligibility of Sign Language content. This evaluation was carried out using quality metrics, as well as validation by a volunteer group of individuals through a survey, ensuring a comprehensive assessment of how these anonymization methods affected both the preservation of the original message and the usability of the data.

Our work makes the following contributions to the field:

• We compare two formats of video anonymization: one using realistic avatars, and another using face swapping techniques.
• We propose a new measure to evaluate the anonymization of the facial expression preservation of the generated Sign Language videos.
• An extensive survey for a group of deaf individuals was carried out to evaluate the anonymization and comprehension of the synthetic videos.

The remainder of this paper is organized as follows: In Section 2, we provide a comprehensive review of related work. Section 3 shows the methodology used for anonymizing Sign Language videos. The experiment is explained in Section 4, with the results in Section 5. Finally, the conclusions and future work are given in Section 6.

2. Related Work

Although tasks related to Sign Language Recognition [4,5,6,7] and Sign Language Translation [8,9] have been explored for years, the scarcity of consistent and adequate datasets [1] creates a major challenge for their advancement.

Providing variations in appearance and style ensures the diversity in scenarios and situations that will be needed later for the object of study of Sign Language, for which it is essential to have a high number of signers in a dataset. This raises privacy concerns due to possible misuse of data, which has a negative impact on the collection of Sign Language datasets [10]. Anonymization of such video datasets, which have significant research value, would facilitate the contribution of participating signers [1]. Furthermore, it could help increase the anonymous participation of deaf signers, who share their videos on social networks that sometimes include sensitive subjects [11,12]. To preserve the privacy of participants and ensure anonymization of Sign Language datasets, different approaches have been addressed in the state of the art [13] and have been supported by recent studies [14]. It should be noted that work has even been carried out [15] to identify an original signer from his or her gestures.

Early approaches used pixelation [2] or basic graphic filters that masked the face during the sign, but this was not optimal because it resulted in the omission of facial expression, which is essential for the understanding of the sign. Therefore, Sign Language anonymization works followed the following strategies:

• Avatar-based techniques
• Photorealistic retargeting

Both strategies are normally supported by tools such as OpenPose [16] or Mediapipe [17], which estimate body landmarks of the signer. Thus, in a frame-by-frame manner, movements are captured and can be used in the anonymized version. It should also be noted that these techniques have been used for Sign Language Production (SLP) [18], so the state of the art is quite advanced, given that the production of signs was an objective pursued before anonymization.

2.1. Using Avatars for Anonymization

The idea is to reproduce all the signer signs using a representation of the human. These representations or avatars can be seen as cartoonized animations [19], or virtual 3D human animations [20,21]. Once the manual, corporal, and facial expressions have been collected, it is possible to re-enact signed messages. The use of realistic avatars is a time-intensive task [13], but current game engines such as Unreal (https://www.unrealengine.com/, accessed on 1 June 2025) are reducing the processing time and generating realistic avatars.

The work of Tze et al. [19] anonymized Sign Language videos by converting them, through skeleton retargeting, into animated cartoons that mimic the original signer’s gestures, while hiding their identity. This method is designed to be lightweight and capable of running efficiently on a CPU, delivering performance close to real time. For skeleton extraction, it uses MediaPipe Holistic modules. The authors pointed out that their method occasionally fails to accurately transfer facial expressions, mouths, and the spatial alignment of the hands and face.

Virtual 3D humans have been explored by Krishna et al. [22] in SignPose, comparing 3D pose estimation models, including 3D body and hand motion from monocular videos [23,24,25] and those that convert a 2D pose to 3D, using OpenPose as an intermediate model and a quaternion-based architecture to move the 2D keypoints to 3D. Results showed that this intermediate option generated better results. However, the work had a major shortcoming: it did not capture facial expressions. SGNify [26] introduced linguistic priors and constraints for 3D hand pose that help in resolving ambiguities within isolated signs. Unlike Signpose, which requires all OpenPose key points to be detected above the pelvis, which is impractical in noisy LS videos, SGNify incorporates Sign Language knowledge in the form of linguistic constraints. Their authors evaluated SGNify through a perceptual study including 20 experts in American Sing Language, who rated their ease in recognizing the presented sign. Future work included the need to better capture facial expressions, tongue and eye movements, mouth morphemes, and eyebrows. Finally, Baltatzis et al. [27] proposed a diffusion-based 3D SLP model trained on a large-scale dataset of 4D signing avatars and corresponding text transcripts. The proposed method uses an anatomically informed graph neural network defined on the body skeleton.

2.2. Photorealistic Retargeting

To achieve an anonymized signer capable of communicating in the same manner as the original signer, various techniques have been investigated. Research in Sign Language Production (SLP) has utilized Generative Adversarial Networks (GANs) to generate photo-realistic Sign Language video sequences [28], as well as transformer-based architectures [29,30].

Focusing specifically on anonymization methods, AnonySign [31] is a notable example. This approach generates photo-realistic retargeted videos using a Variational Autoencoder (VAE) [32] that incorporates pose and style encoders to synthesize human appearance, effectively replacing the original signer’s identity with a synthetic one.

In [33,34], SignGAN was introduced to produce continuous, photo-realistic Sign Language videos directly from spoken language. The system employed a transformer architecture with a Mixture Density Network (MDN) [35] to translate spoken language into skeletal pose, followed by a pose-conditioned human synthesis model to generate the photo-realistic videos. A keypoint-based loss function was implemented to enhance the quality of hand images in the synthesized outputs.

Xia et al. [14] proposed an unsupervised image animation method featuring a high-resolution decoder and loss functions specifically designed for facial and hand regions, to transform the signer’s identity. However, the authors noted that this method struggled to adapt to videos captured in uncontrolled environments. To address these challenges, they later introduced diffusion models [36] for zero-shot text-guided Sign Language video anonymization [37,38]. These models integrate ControlNet [39] to eliminate the need for pose estimation, along with a specialized module for capturing facial expressions.

The authors claimed their method improves upon existing approaches, such as Cartoonized Anonymization [19], by relying on skeleton estimation to ensure accurate anonymization. Furthermore, they highlighted that, while photorealistic methods such as AnonySign [31], SLA [14], and Neural Sign Reenactor [40] can achieve realistic results, they require training on large-scale Sign Language datasets or precise skeleton estimation, making them less effective in real-world scenarios. Despite these advancements, their proposed method has limitations, particularly when the face is occluded by one or both hands or when rapid movements result in motion blur. Additionally, artifacts commonly associated with Stable Diffusion Models occasionally appear, impacting the overall video quality.

2.3. Conclusions

Progress has been made in anonymization from two points of view, SLP, where signs must be generated from text, and anonymization itself, replacing the original signer with another signer through cartoons, 3D avatars, or photorealistic assemblies. Progress has been very significant in recent years, but there is no established methodology to check whether anonymization has actually been achieved and, the most important question, whether the message has been retained. In this sense, this paper proposes a series of metrics to answer these questions and provides a comparison of objective quality metrics and subjective answers from deaf people watching the same set of videos.

3. Methodology

This section details the proposed methodology for the anonymization of Sign Language datasets.

3.1. Collection of Original Video

To conduct our experiments, we first had to select a series of videos in Spanish Sign Language (LSE), as the objective was to carry out a survey targeting the deaf population in our country. For this purpose, we utilized videos previously recorded by our research group under controlled laboratory conditions, as detailed in [41]. Specifically, a Nikon D3400 camera (Nikon Corporation, Tokyo, Japan) was used to capture high quality RGB video signals (resolution 1920 × 1080 @ 60 FPS). Five out of 40 signers gave explicit permission to appear in an online survey where their faces would be distorted: two men and three women (this highlights the deaf community’s privacy concerns). For each signer, 6 videos were chosen from a set of 18 different LSE phrases. This set was used for the face-swapping tests.

Additionally, one of the collaborator signers was asked to record an additional set of videos specifically for this work, where instead of signing a full phrase, he signed pairs or trios of signs that differed only in facial expressions. This second set, consisting of 16 videos (5 pairs plus 2 trios), was specifically used to evaluate the preservation of facial expressiveness after the face-swapping process.

Finally, for the generation of avatar videos, 9 LSE videos from the first set were selected, and the same collaborating signer was asked to record an additional set of 5 videos with particularly expressive signing to evaluate the transfer of expressiveness to the avatar. This resulted in a total of 14 videos, which were generated using both male and female avatars. All the videos, original and generated, can be watched at GitHub (https://github.com/Deepknowledge-US/TAL-IA/tree/main/Synthetic_video_for_Sign_Language_Anonymization, accessed on 30 Sepptember 2024). Section 4.1.2 outlines the survey design.

3.2. Video Synthesis with Avatars

To generate videos anonymizing the signers through avatars, we utilized the Dollars Mocap tool [42] (Dollars Mocap, Shanghai, China), an AI-powered motion capture solution designed to streamline the creation of realistic animations for characters in various digital applications.

Figure 1 illustrates the Dollars tool animation workflow. The process starts with MediaPipe (Google LLC, Mountain View, CA, USA) capturing a continuous stream of image frames from the original video file and detecting body, hand, and face landmarks. The tool include tracking algorithms to ensure smooth and consistent key points across frames, enabling accurate motion capture. The captured data are processed to calculate skeleton rotation angles, followed by smoothing and pose adjustments to reduce noise and enhance stability. The refined data are then exported or streamed to third-party software for avatar generation.

Figure 2 shows an example of the Dollars tool interface. As can be seen, the tool uses the extraction of keypoints with mediapipe for subsequent generation of the avatar.

The final avatar in the video was created using Unreal Engine’s (Epic Games, Cary, NC, USA) online Metahuman 5.0 application (https://metahuman.unrealengine.com/, accessed on 3 June 2025), which utilized BVH-exported coordinates from EGAO software. These coordinates were integrated with the Metahuman skeleton, allowing the animation data to be linked and further refined. Figure 3 shows the animation workflow, where, through UE5 blueprints, the animation data (represented by the yellow box) are linked to the Metahuman skeleton (represented by the red box). At this stage, adjustments can be made to the animation through the blueprints, such as the one in Figure 4, allowing modification of aspects such as the movement range.

Additionally, if the inference of the points is too far from the correct one, the animation can be manually corrected. While manual corrections are possible for cases where the point inference deviates significantly, such as self-intersections, this process is time-intensive and may introduce distortions that affect the entire animation, so it is reserved for critical issues only, such as self-intersections, as can be seen in Figure 5.

3.3. Video Synthesis with Face Swapping

In this section, the process developed to anonymize the identity of the signers in the dataset through face swapping techniques is explained.

Face swapping [43] is a technique that replaces one person’s face in an image or video with another person’s face, using image processing and computer vision tools. This typically involves detecting and mapping facial landmarks in both images, aligning the facial features, and applying blending techniques to ensure seamless integration, while preserving consistency in lighting, textures, and facial expressions.

A major challenge in face swapping is achieving results that are both visually convincing and scalable. To address this, we opted for a near real-time solution using the DeepFaceLive tool [44], built on the DeepFaceLab framework [45]. This tool enables face swapping from a webcam feed or a video using pre-trained facial models from famous actors, and it also allows the training of custom models, which we utilized to incorporate our own faces into the synthesis of new videos. To achieve this swapping, this tool uses a state-of-the-art face detector, then it aligns the face using 2DFAN [46] or PRNet [47] and calculates a similarity transformation matrix through a facial landmark template. Finally, the new face can be swapped using face segmentation [48].

DeepFaceLive provides user-friendly customization options, enabling us to fine-tune the face-swapping process to meet the specific requirements of Sign Language video synthesis. This ensured that the generated results were both accurate and ethically sound.

We performed face swapping using Live with one of the official pretrained models from DeepFaceLab. These models rely on a U-Net–based segmentation network known as TernausNet [48], which uses a VGG11 encoder pretrained on ImageNet for accurate facial region extraction [45]. The pretrained models were originally trained on tens of thousands of images from the FFHQ dataset, which includes a wide variety of facial poses, lighting conditions, and expressions. No additional training or fine-tuning was applied in our case, and we used the models as provided (https://www.deepfakevfx.com/guides/deepfacelab-2-0-guide/, accessed on 3 June 2025).

Given the real-time constraints of our application, we employed the Quick96 model, which offers a favorable balance between performance and computational efficiency. This model is commonly used in DeepFaceLive, due to its low-resolution architecture that is optimized for live inference, making it suitable for generating anonymized face-swapped videos without sacrificing speed.

4. Experiments on Video Synthesis

4.1. Evaluation Metrics

In this section, we detail all metrics that were used to evaluate the anonymization precision of the dataset. The anonymization quality was tested with both objective quality metrics and subjective human evaluations.

4.1.1. Quality Metrics

We next describe the computer-vision-based metrics used to evaluate the quality of anonymization achieved through Face Swapping. The “Learned Perceptual Image Patch Similarity” (LPIPS) [49] and “Frechet Inception Distance” (FID) [50] metrics, widely utilized in the literature for avatar generation [51] and dataset anonymization [14,31,52] were applied. Additionally, we introduced a novel quality metric specifically designed to evaluate the effectiveness of anonymization in Sign Language videos.

The LPIPS metric evaluates the perceptual similarity between images by leveraging features extracted from a pre-trained deep neural network, such as VGG or AlexNet. These networks capture high-level representations that align more closely with human perception of similarities and differences, surpassing simple pixel-to-pixel comparisons. Higher LPIPS values indicate greater perceptual differences between images.

FID is a widely adopted metric for assessing the quality of images generated by models like Generative Adversarial Networks (GANs), making it particularly suitable for evaluating face swapping. FID compares the feature distributions of generated and real images, utilizing features extracted by a pre-trained network, typically Inception v3. This metric calculates the Frechet distance (Wasserstein-2 distance) between two multivariate Gaussian distributions. Lower FID values indicate greater consistency between the generated content and the original videos.

LPIPS and FID primarily evaluate the overall visual quality and similarity between the original and generated videos, but they do not measure whether the identity of the original individual has been effectively obscured or guarantee proper anonymization in synthetic videos. To address this gap, we propose a new metric, IAEP (Identity Anonymization with Expressivity Preservation), designed to evaluate changes in apparent identity while preserving facial expressivity. IAEP combines measurements from two embedding spaces: one optimized for face verification, and another for encoding facial expressions.

Face verification models, which have surpassed human performance [53], can provide reliable metrics for measuring anonymization success. However, encoding facial expressions presents a greater challenge. Most off-the-shelf models have been trained to classify basic emotional expressions (e.g., neutral, angry, surprise, fear, happiness, disgust, sadness) [54], whereas facial expressivity in Sign Language conveys subtle grammatical information through movements of the eyebrows, mouth, and head [55]. For such nuanced expressivity, the Facial Action Coding System (FACS) [56] offers a more appropriate framework, as it describes changes in facial muscles through Action Units (AUs).

In this context, AU intensity estimation quantifies the strength of specific facial muscle movements. To implement IAEP, we propose a method that integrates AU-based expressivity measurements with identity verification to assess anonymization success, while ensuring the preservation of crucial facial expressions in Sign Language communication. The following implementation details this approach:

$$IAEP({\mathbf{V}}_{\mathbf{o}},{\mathbf{V}}_{\mathbf{s}})={\displaystyle \frac{1}{\left|\mathcal{F}\right|}}\sum _{f\in \mathcal{F}}\mathcal{D}(ID\left(V_o\left(f\right)\right),ID\left(V_s\left(f\right)\right))-\mathcal{D}(AU\left(V_o\left(f\right)\right),AU\left(V_s\left(f\right)\right))$$

where

• $\mathcal{F}$: the set of frames selected in the original video ${\mathbf{V}}_{\mathbf{o}}$ (the swapped frames in ${\mathbf{V}}_{\mathbf{s}}$, are the same),
• $\mathcal{D}(ID\left(V_o\left(f\right)\right),ID\left(V_s\left(f\right)\right))$: a distance function between the embedding vectors coming out from the Face Verification model ($ID$) corresponding to aligned frames, f, ${\mathbf{V}}_{\mathbf{o}},{\mathbf{V}}_{\mathbf{s}}$, and
• $\mathcal{D}(AU\left(V_o\left(f\right)\right),AU\left(V_s\left(f\right)\right))$, the same for the AU intensity embedding vectors.

Using a normalized metric $\mathcal{D}$, such as the Cosine distance from the scipy.spatial.distance function (normalized [0, 2]), $IAEP$ values fall within the range [−2, 2]. However, the specific range depends on the choice of $ID$ and $AU$ embedding spaces, and, in practice, they typically lie between [0, 1], as demonstrated in the examples below. This range is more interpretable compared to FID and LPIPS. Higher $IAEP$ values indicate better anonymization and preservation of facial expressivity, whereas lower values reflect poor anonymization and/or ineffective transfer of expressions to the swapped face. Unlike IAEP, LPIPS and FID primarily measure visual similarity. While small FID and LPIPS values suggest a high visual consistency between original and swapped faces, they do not provide specific insights into changes in identity or expressivity.

The Face Verification model used to analyze the original–swapped video pairs was FaceNet [57], which generates a 512-dimensional embedding vector, as implemented in the Deepface Python library [58]. For Action Unit (AU) intensity estimation, we employed the LibreFace Python library [59], which was trained to estimate the intensity of 12 key AUs (1, 2, 4, 5, 6, 9, 12, 15, 17, 20, 25, 26). These AUs represent most of the eyebrow and mouth movements critical for conveying the grammatical nuances essential in Sign Language communication (Visit https://imotions.com/blog/learning/research-fundamentals/facial-action-coding-system/ (accessed on 1 May 2025) for a video description of each AU).

The set of frames selected for calculating $IAEP$, $\mathcal{F}$, should capture the variability in expressiveness within the video, as the primary goal is to evaluate both anonymization effectiveness and the preservation of expressive moments. Using all frames for comparison risks averaging out critical changes in expression that occur over a few frames. To address this, we implemented a selection process based on intra-video AU changes, $\mathcal{D}(AU\left(V_o\left(f_i\right)\right),AU\left(V_o\left(f_j\right)\right))$, starting with a high threshold and iteratively reducing it until at least N frames are selected for inter-video comparison. This was implemented with a recursive algorithm explained in the following pseudocode (Algorithms 1 and 2):

Algorithm 1: Recursively select frames from original video

Algorithm 2: Select frames based on Action Unit (AU) changes

We conducted experiments by varying N from 5 to 20 and observed no significant changes within this range. When N was too small, the results tended to be noisy. As N increased beyond 20 and approached the total number of frames in the video, the $IAEP$ metric became increasingly dominated by the Face Verification component. This was because more frames with similar and neutral Action Units (AUs) were averaged, which reduced the influence of the Expressivity Preservation term—especially in cases where the swapped face failed to replicate extreme facial expressions. In the experiments presented in Section 5, N was fixed at 5. Note that more frames may be selected if their AU differences exceed the current threshold.

Figure 6 provides a graphical example of IAEP calculation, comparing an original video (top) with its swapped counterpart (bottom). The figure includes intra-ID and intra-AU scores as references for the distances between embeddings from the same video, as well as inter-ID and inter-AU scores for the corresponding selected frames, $\mathcal{F}$, across the two videos.

A detailed analysis of these metrics highlighted several key properties of the IAEP measurement:

• Robustness of ID-Embedding: The largest intra-ID scores provide insights into the robustness of the ID-embedding against changes in expression and pose. Smaller intra-ID values indicate a more robust ID model.
• Anonymization Quality: For a robust ID-embedding, the difference between inter-ID and intra-ID scores quantifies the separability of identities, offering an objective assessment of anonymization effectiveness.
• Robustness of AU-Embedding: inter-AU and intra-AU scores provide insights into the robustness of the AU-embedding to variations in identity and pose. In general, inter-AU scores are expected to be smaller than intra-AU scores.
• Expressivity Transfer: For a robust AU-embedding and effective transfer of expressivity, the distribution of intra-AU scores in the original video should closely match that of the swapped video, and the inter-AU scores should remain minimal.

These properties collectively ensure that IAEP provides a comprehensive evaluation of both identity anonymization and the preservation of expressive nuances.

The example in Figure 6 illustrates these properties effectively. The ID verification model reveals significant differences between inter-ID and intra-ID scores. While this reflects good separability between identities, a greater invariance to pose and expression changes would be ideal. For instance, values of 0.32 and 0.37 between the first and second selected frames are relatively high for a state-of-the-art verifier like FaceNet-512, which typically uses an ID verification threshold around 0.3.

Similarly, the AU intensity estimation model demonstrated that intra-AU values in the original video are generally higher than in the swapped video. These differences become more pronounced when expressive features, such as raised eyebrows or mouth deformations, are not accurately transferred. As with the ID verification model, greater invariance in AU intensity estimation to pose changes would be beneficial. For example, the low values observed between the 1st and 2nd frames and the 4th and 5th frames in the swapped video are inconsistent with the rest of the values, highlighting areas for improvement in the robustness of AU estimation.

4.1.2. Subjective Evaluation

In addition to the quality metrics, we emphasized the importance of incorporating subjective evaluation. To achieve this, we designed a survey aimed at a group of deaf participants, to address three specific scientific questions. The survey was divided into three sections, with participants watching videos of deaf individuals signing particular phrases or words for evaluation.

• Part 1: Anonymization through Face Swapping: The first part aimed to assess whether a face swapping technique successfully anonymized the identity of the signer. Familiarity of the participant with the signing person might bias their response, so we designed two questions. The participants are first asked if they recognize the signer, which helps gauge how familiar they are with the person signing. The second question directly asks whether they believe the face swapping has successfully anonymized the person’s identity.
• Part 2: Preservation of Meaning with Facial Expression: The second part focuses on whether face swapping has preserved the meaning of signs where facial expressions are crucial. Participants watch videos of words that vary in meaning depending on facial expression and are then asked a single question: they must choose the correct meaning from several options.
• Part 3: Comprehensibility of Avatars: In the third part, instead of using face swapping for anonymization, avatars are used. Here, the goal is to assess whether the avatar can generate an understandable signed sentence. Participants were asked three questions: Whether they understood the phrase signed by the avatar. After watching the original video with the phrase, participants were asked if the recognized phrase matched the original one. Finally, participants were asked what they believe could be improved, specifically whether the manual components of the sign or the facial expressiveness could be enhanced for better clarity.

To design the survey questions, we sought help from our contacts in the deaf community and from interpreters who collaborate with us. We needed to select sentences with sufficient manual and facial expressiveness so that the survey results would be generalizable. For Part 1, we also needed to find famous individuals with enough videos available online to feed the face swapping algorithm and generate videos of volunteer signers for the survey design. In the end, we had five signers—three women and two men—each of whom recorded several different sentences for Part 1. Each of these sentences underwent a face swapping process, generating 30 sentences to present randomly to the respondents. It is important to highlight that the face swapping was carried out while preserving the gender of the signer, to avoid undesirable effects of generating faces with gender ambiguity.

For Part 2, we needed to find signs that had different meanings depending on the facial expression. We recorded seven groups of videos for this section, each containing two or three challenging signs that could only be distinguished by facial expression. All these videos were recorded by the same highly expressive deaf signer to avoid bias resulting from different respondents watching videos of different expressive individuals. Each of these videos then underwent a face-swapping process, resulting in a final total of 16 videos to be shown randomly.

Finally, for Part 3, we generated 14 sentences with two avatars (one male and one female). All the created videos are available at the previous GitHub repository. The survey is accessible (in Spanish and LSE) at https:/signamed.web.app/survey, accessed on 1 May 2025. Each time a respondent entered the survey, a random subset of videos was shown in each part, to prevent fatigue and encourage them to return and complete the survey at another time.

Figure 7 displays snapshots of the main survey questions, which are based on video viewing. Please note that the questions are presented in Spanish (text) and also available in Sign Language by clicking on the orange icon.

5. Results

5.1. Quality Metrics Results

All experimentation was carried out on a computer with a 12 Gb Nvidia 3080 GPU (Nvidia, Sunnyvale, California, USA) As explained in Section 4.1, LPIPS and FID metrics were calculated for all P1 anonymized videos with face swapping.

Studies such as [14,31,52] also employed FID and LPIPS metrics to evaluate anonymized videos. However, they did not provide direct comparisons with other works in the field, instead focusing on demonstrating improvements over their previous methods. Low LPIPS and FID values indicated that the generated images were visually similar to the real ones, suggesting that the generated videos maintained a high degree of visual realism and strong correlation with the original content.

We cropped the facial region in both the original and generated videos, rather than measuring the entire frame. This approach allowed for a more precise comparison, as the face is the only area modified during the face swapping process.

Figure 8 and Figure 9 show the distributions of FID and LPIPS metrics, respectively, grouped by signer. Although both metrics operate in different ranges of values, the distributions obtained showed a similar trend, at least for four out of five signers.

Figure 10 shows a sequence of frames from the original video and the synthesized video of signer p0036 with large FID and LPIPS values, while Figure 11 shows a sequence of signer p0006 with lower FID and LPIPS values.

As mentioned before, these two metrics inform us about perceptual similarities. Looking at the example in Figure 11, it seems clear that there are significant image similarities, so we can claim that the FaceSwapping process preserved the general structure and texture of the original images (low FID/LPIPS values). Can we also claim that the identity is obscured?

As we presented above, the five signers underwent face swapping with same-gender celebrities. Figure 12 shows the distribution of the IAEP metric for all the videos of each signer. These are exactly the same videos as used for the computation of LPIPS and FID, but using the strategy of frame selection justified in the IAEP definition.

The distributions per signer are notably inversely related to FID and LPIPS metrics, where high visual consistency (low FID/LPIPS values) corresponds to high IAEPD values, and vice versa. This seemingly contradictory behavior arises from the way these metrics operate. FID and LPIPS rely on generic visual features learned by deep learning models trained on broad datasets, while IAEP utilizes two specialized models: one trained for face verification to be invariant to changes unrelated to identity attributes, and another optimized for Action Unit (AU) intensity estimation to focus on facial muscle changes.

For instance, the video pair of signer p0036, previously shown in Figure 10, exhibited high FID/LPIPS values but low IAEPD values (see Figure 13). These low IAEPD values stemmed from the face verification model, which assigned scores close to the decision threshold of FaceNet-512 for inter-ID pairs. Although the videos appear visually distinct due to features like glasses, reflections, and gray/black beard (high FID/LPIPS values), these attributes are not identity-related, as FaceNet is trained to recognize identities invariant to such attribute differences. However, the expressivity transfer was not fully accurate, as indicated by the inter-AU values being, on average, larger than the intra-AU values.

Conversely, the video pair of signer p0006, shown in Figure 11, displayed low FID/LPIPS values but mid-range IAEPD values (see Figure 14). Although these two videos are visually similar, differences in eye color and the shapes of the eyebrows, nose, and mouth resulted in a relatively high inter-ID score of 0.71. This examples highlights the strengths of IAEP in measuring both anonymization and expressivity preservation.

5.2. Survey Results

The survey was conducted online using a platform that allowed the participants to watch videos and answer related questions. The survey is still accessible to anyone at the following address: https://signamed.web.app/survey, accessed on 1 May 2025. This platform is already well known within the deaf community, as it has been used for research on Sign Language Recognition technologies and has an associated Instagram channel. To recruit volunteers, a campaign was launched through Instagram, and the survey remained open for responses from 30 July to 13 September 2024. Since participants could leave the survey at any time, it was closed once 100 complete responses, including all three parts, had been collected. Demographic diversity could not be assessed, as anonymous access was granted, and no personal data were collected. Sign language proficiency was specified in the introductory LSE video. There might have been cultural and age-related bias in the participants, as the Instagram channel (with 888 followers) used to promote the campaign is mainly used to inform about the progress of the project.

Table 1. Survey Data Summary.

Metric	Value
Total number of responses	1403
Number of different respondents	76
Number of responses for part 1	580
Number of responses for part 2	523
Number of responses for part 3	300
Number of completed surveys	100
Average number of completed surveys per different user	1.32

shows the collected interactions during the campaign.

It is important to note that each time a user starts the survey, the videos they see are randomly selected from a pre-recorded set, so it is normal for users with more interest in the topic to respond to the survey more than once. It was also observed that the first and second parts, related to face swapping, were completed more often than the third part, which was related to avatars. In the following paragraphs, we provide an analysis of the responses to each part and a final discussion.

In the first part of the survey, participants were asked to evaluate the effectiveness of face-swapping technology in anonymizing a person who signs in a video. Question “p01a” assessed whether respondents recognized the signer based on their picture (a known person in the community), while question “p01b” involved the respondents watching a video where the signer’s face was swapped with that of a well-known celebrity. The video showed a blended appearance of the original signer and the celebrity, alongside photos of both individuals, and participants were asked if the identity anonymization was successful.

The analysis revealed that 58.1% of the respondents answered “TRUE” (i.e., believed the anonymization was successful) when exposed to the face-swapped video, indicating that the face swap was often effective. However, the results also showed a relationship between the answers to “p01a” and “p01b”: when respondents initially did not recognize the face of the original signer, their perception of successful anonymization was 56.5%. Conversely, if they recognized the signer, they were slightly more likely to perceive the anonymization as successful, with 59.8% responding “TRUE”. This result is summarized in Figure 15.

A statistical significance test was performed to evaluate the potential dependency between the respondents’ recognition of the original signer (“p01a”) and their assessment of successful anonymization (“p01b”). The chi-square test returned a p-value of 0.467, indicating no statistically significant relationship between the two variables at conventional significance levels. This result suggests that respondents’ familiarity with the signer did not significantly influence their perception of the success of the anonymization, implying that the effectiveness of the face-swapping process was independent of prior familiarity with the signer.

However, this finding should be interpreted with caution, as the survey only addressed familiarity with the signer and not the celebrity involved in the face swap. It is possible that familiarity with either of the subjects—signer, celebrity, or both—could have differently influenced responses regarding anonymization success. Further investigation is needed to account for these potential confounding factors.

Given the difference in the signer’s FID/LPIPS/IAEP measurements in Figure 8, Figure 9 and Figure 12, we wanted to verify whether this difference was also reflected in the subjective evaluation. Figure 16 shows a pair of stacked bars for each signer, indicating the number of responses with negative/positive anonymization (FALSE/TRUE), conditioned on whether the signer was known or unknown to the participants (red/blue).

Upon closer examination, we observed that participants more frequently perceived signers p0003 and p0006 as correctly anonymized compared to signers p0036, p0037, and p0041. Reviewing the corresponding videos, we found that the pictures used to match the first two signers with their celebrity counterparts depicted them with long hair, whereas in most of the videos, their hair was tied back. This perceptual discrepancy may have influenced the participants’ decisions, leading to a higher correct anonymization rate.

Analyzing the IAEP and FID metrics, these two signers exhibited the largest IAEPD and smallest FID values, while the other three showed the smallest IAEPD and largest FID. This observation might suggest an interesting correlation between IAEP/FID metrics and deaf individuals’ perception of anonymity; however, considering p0036, we remain cautious. The videos of signer p0036 were perceived as correctly anonymized a similar number of times as p0037 and p0041, with all three only slightly better than random choice. FID seemed to somewhat reflect the opinions of deaf individuals, while IAEPD identified more metric differences between signer p0036 and the rest (similarly to LPIPS). In any case, it appeared that additional factors influenced the deaf individuals’ decisions that were not captured by the three tested metrics.

In the second part of the study, participants were tasked with identifying the signed gesture by choosing from three options (question p02): the correct sign, a similar sign differing only in facial expression, or the“I don’t know” option. The analysis indicated that 399 participants (78.4%) selected the correct sign, 86 participants (16.9%) chose the similar sign with a different facial expression, and 18 participants (3.5%) opted for “I don’t know”. These findings suggest that the majority of participants accurately identified the correct sign, even with the option to express uncertainty, implying that the face-swapping process effectively preserved the original signer’s essential facial expressions.

In the third part, participants were asked whether the sentence signed by an avatar was comprehensible (question p03a). The results showed that 66.7% of their responses indicated incomprehension, while 33.3% selected the option labeled “YES” (indicating comprehension). In the next question of this part (p03b), the participants were shown the same sentence signed by a real person and were asked if what they understood from the avatar matched the sentence signed by the deaf contributor. Among the responses of positive comprehension, 80.2% also answered “YES”. This suggests that those who initially comprehended the avatar were also likely to perceive consistency between the avatar’s message and the deaf person’s message. However, was this comprehension ratio evenly distributed for all synthesized videos? Figure 17 displays the ratio of comprehension of the avatar message for each one of the 14 synthesized sentences. Most of the videos (11) ranged between 20 and 50% comprehension, two videos were lower than 10%, and only one had a comprehension of 76.2%. This variation suggests that the comprehensibility of the avatar’s signing was quite low in general and also influenced by factors specific to each video, such as the clarity of the original signing, the quality of the synthesizing process, and the complexity of the sentence being signed.

As a final question (p03c), we wanted to know the participants’ opinions about which components of Sign Language require more precision for the avatar’s message to be comprehensible. The two answer options for this question were ’Hands’ and ’Facial Expression’. Participants could select either or both options only if they did not understand the avatar message previously. Most of the responses (55%) indicated that both components (hands and facial expressions) needed more precision, followed by hands only (36.5%) and facial expressions only (8.5%).

These poor results in avatar comprehensibility may also explain the higher number of abandoned interactions during the third part of the survey, compared to the first and second parts (300 versus 523). Deaf individuals are generally frustrated by the low quality of avatars for continuous Sign Language, and, as these results reflect, participants struggled to understand the avatar in about two-thirds of the cases, largely due to issues with the fidelity of hand movements.

One of the main causes of the low comprehension rate for avatar-based videos (66.7%) lies in the technical limitations of the animation process. As explained in the methodology, we used MediaPipe to extract body, face, and hand keypoints, and then generated animations using Unreal Engine and its Metahuman system. However, the mapping between the extracted keypoints and the Metahuman skeleton is not perfect, which results in inaccurate motion transfer for critical components such as eyebrows, eyes, mouth, and head. While manual corrections are technically possible through Unreal’s blueprint system, they are extremely time-consuming, especially when working with continuous videos that involve a high level of facial expressiveness. Therefore, we only applied manual corrections in critical cases, such as evident self-intersections. This structural limitation of the current pipeline largely explains the observed lack of expressive fidelity and, consequently, the low comprehension rate reported by participants.

5.3. Discussion

The results in this section highlight several takeaways that are pivotal for advancing the field:

• Effectiveness of Face Swapping: Face swapping proved to be a viable anonymization technique, achieving decent rates of perceived anonymization among participants and large rates of measured face discriminability. However, certain discrepancies, such as variations in hairstyle, subtle visual inconsistencies, or even questionable understanding of the query about signer–celebrity anonymization, impacted participants’ perceptions, underscoring the need for a greater understanding of anonymity perception.
• Challenges with Avatar-Based Anonymization: The results demonstrate that avatar-based anonymization struggles with comprehensibility, particularly for complex sentences. The survey responses indicate that both hand movements and facial expressions need refinement to meet the high expectations of Sign Language users. The low comprehension rates observed highlight the need for further improvements in avatar synthesis technologies.
• The current work highlights the need for larger and more diverse datasets for sign language anonymization research. In future work, we aim to expand our dataset by defining clear collection targets—e.g., several hundred signers across different age groups, signing styles, and regions. This will require building long-term collaborations with deaf communities and institutions, possibly through partnerships with national federations or sign language interpretation networks. Such collaborations would not only increase the scale and demographic variety, but also promote inclusiveness and ethical engagement.
• Metrics for Evaluation: The introduction of the IAEP metric provided an innovative way to assess the balance between anonymization and expressivity preservation. While the metric showed promise, its performance in certain scenarios suggested that more robust or complementary subjective metrics may be needed to capture all relevant aspects of anonymization quality, particularly when dealing with varied signer swapped attributes.
• User Perception and Practical Usability: This small scale survey provided valuable insights into how deaf participants perceived the anonymization techniques. The alignment of their perceptions with FID and IAEP metrics for some signers indicates the potential of these tools to reflect user experience. However, variations in the comprehension and anonymization success across different signers suggest that additional contextual or subjective factors influence user perception.

6. Conclusions

This study explored two anonymization approaches for Sign Language videos: realistic avatar synthesis and face swapping, with a focus on balancing privacy concerns and the preservation of communicative content. The proposed IAEP metric offers a novel evaluation framework, addressing the limitations of existing visual similarity metrics like FID and LPIPS by incorporating both identity anonymization and expressivity preservation.

Our results show that face swapping achieved a reasonable level of anonymization while maintaining semantic clarity, as reflected by both the objective quality metrics and subjective evaluations. However, the avatar-based approach struggled with comprehension, particularly in conveying complex expressions critical for Sign Language communication. The survey results from deaf participants provided valuable insights into user perceptions, highlighting the strengths and limitations of each method.

Future work should prioritize refining metrics such as IAEP to enhance their robustness against subtle variations in expression and identity shifts, as well as comparing it with other related metrics in the literature. Additionally, improving avatar synthesis technologies to achieve higher fidelity in hand movements and facial expressions is critical to making this approach more comprehensible. Expanding datasets with diverse signer profiles and a greater number of source swapping faces is also essential for ensuring the scalability and generalizability of these anonymization techniques. By addressing these challenges, the field can move closer to enabling privacy-preserving Sign Language datasets that maintain their linguistic and communicative integrity.