Facial Attribute Prediction Using Knowledge Distillation ^†

Abstract

The rising need for intelligent applications on resource-limited devices presents a significant challenge. The deployment of big, high-accuracy deep learning models is impractical due to their processing costs. This thesis examines this issue in the realm of face attribute analysis. We present an approach for multi-attribute prediction (age, gender, ethnicity) using knowledge distillation to develop a model that is both highly accurate and computationally economical. The methodology comprises a two-phase technique utilizing the UTKFace dataset. A substantial, high-capacity “teacher” model, with a ResNet50 architecture, is trained to attain state-of-the-art precision in predicting age, gender, and ethnicity. Consequently, the extensive, intricate understanding from this educator is conveyed to a considerably smaller and more efficient “student” model utilizing a MobileNetV2 architecture. The transfer is accomplished by instructing the learner using both the accurate labels and the softened probability distributions produced by the teacher. The findings indicate that the distilled student model attains prediction accuracy nearly equivalent to that of its larger instructor model across all variables, while markedly decreasing model size and inference duration. The findings also confirm that knowledge distillation is a viable and efficient method for creating high-performance facial analysis systems appropriate for real-world implementation on mobile, edge, and other resource-constrained platforms.

1. Introduction

In the 21st century, digital photos have changed how we use technology and talk to each other. More and more, visual data is becoming an important and common source of information. Every day, billions of pictures are shared on social media. There are huge networks of surveillance cameras watching public places, and personal devices employ cameras built into them. The proliferation of digital information has enabled computers to process visual data in fundamentally new ways. This advancement has led to the development of complex algorithms capable of extracting meaningful information from images and videos on a scale previously considered unattainable. A notable application of this technology is the estimation of facial attributes such as age, gender, and ethnicity. Accurate and efficient prediction of these attributes has significant implications across domains, including business and science. For instance, in business, facial trait estimation plays a central role in next-generation personalized marketing and targeted advertising. Real-time systems that identify demographic groups can dynamically adjust content, product recommendations, and user experiences to enhance relevance and appeal. Such technologies also contribute to the design of adaptive and intuitive interfaces in human–computer interaction.

The rapid progress in this area is largely attributable to deep learning. The transition from traditional machine learning approaches, which relied on handcrafted features, to deep learning models capable of autonomously learning feature representations has been transformative. Convolutional neural networks (CNNs) have become the standard in computer vision applications. Architectures such as ResNet50, a 50-layer deep residual network, have demonstrated exceptional performance in learning complex hierarchical features from large-scale datasets such as ImageNet. Their deep structure enables the capture of both low-level edges and textures as well as high-level semantic concepts, making them highly effective for facial attribute estimation. When trained on diverse facial datasets such as UTKFace, these models can discern subtle visual cues that reveal age, gender, and ethnicity.

2. Related Works

Kalayeh [1] proposed a facial attribute prediction technique by using semantic segmentation as an auxiliary task. The method makes the attribute prediction model accurate and better at localizing relevant facial features, which it achieved by enabling the model to explicitly focus on the right parts of the face (wavy hair prediction), thereby leveraging the importance of spatial context for attribute prediction. Abdulnabi [2] proposed a multi-task CNN model for attribute prediction. The method enhances the precision of forecasting diverse facial characteristics by training a singular model to acquire multiple attributes concurrently, accomplished through the facilitation of shared feature learning across interconnected attribute tasks within the CNN architecture. Huang [3] proposed a deep imbalanced learning technique for face recognition and attribute prediction. The method addresses the challenges posed by imbalanced datasets, where specific classes or attributes have markedly fewer examples than others, by developing innovative learning strategies within deep neural networks to effectively handle these disparities and improve performance in both recognition and attribute tasks.

Huang [4] proposed an image captioning technique with end-to-end attribute detection and subsequent attribute prediction. The method generates descriptive and accurate image captions by integrating an initial phase of attribute detection directly into the captioning pipeline, which it identifies and then uses these detected attributes to better inform the generation of textual descriptions for images. Park [5] employed relational knowledge distillation (RKD) that uses the connections between data samples to make it easier for a teacher model to teach a student model. Traditional knowledge distillation methods focus on having the student copy what each teacher model does. RKD, on the other hand, encourages the student to learn the structural relationships in the data, like the distances and angles between feature representations, that the teacher model already knows. He [6] proposed deep residual learning for image recognition, introducing resnets (residual networks) that changed deep learning by making it possible to train much deeper neural networks.

The residual block is the main new idea. It uses shortcut connections to do identity mapping, which makes it easier for gradients to move through the network and helps with the vanishing/exploding gradient problem. This design fixes the problem of accuracy saturation and then degradation in very deep networks by making it easier for layers to learn residual functions instead of completely new mappings.

Cho and Hariharan [7] investigated the effectiveness of knowledge distillation. Their model learns and understands how and why knowledge distillation works. This is a method in which a smaller student model learns from a larger teacher model. The method improves the effectiveness of knowledge distillation by transferring not just individual feature representations but also the relationships between data samples, which it achieves by encouraging the student model to mimic the structural relationships that the teacher model learns among different instances in the dataset. Kim, Park, and Lee [8] used knowledge distillation for enhanced age and gender prediction accuracy. The method improves the age and gender prediction models by using knowledge distillation techniques Phuong [9] used knowledge distillation to examine the theoretical and empirical aspects of its functionality and rationale. They examined the effectiveness of a student model in learning from a teacher and the underlying factors and conditions that facilitate this process. Key factors affecting the effectiveness of knowledge distillation were identified by integrating theoretical insights with controlled experiments. These factors encompass the teacher’s proficiency, the student’s framework, and the attributes of the training data, thereby creating a more concrete foundation for the development of improved distillation strategies.

Zhang et al. [10] adopted age progression/regression by using a conditional adversarial autoencoder (CAAE) for realistically synthesizing facial images to show how a person might look at different ages. The CAAE framework uses an autoencoder to put images back and a conditional generative adversarial network to make sure the faces it makes look real and match the target age. By clearly separating age data from identity and employing a conditional adversarial learning methodology, the model performs age progression (making a face look older) and age regression (making a face look younger) while preserving the individual’s identity, providing a robust solution for age synthesis.

We used a teacher-student knowledge distillation framework to move deep, multi-task knowledge from a high-capacity ResNet50 model to a lightweight MobileNetV2 model for facial attribute prediction. The method improved prediction accuracy and efficiency by using a hybrid distillation loss that combined the Kullback–Leibler (KL) divergence for classification tasks (gender and race) with mean squared error for regression (age). This allows the small student model to perform almost as well as the teacher model while making the model smaller and cheaper.

3. Methodology

3.1. Data Processing and Model Training

3.1.1. Data Processing

The data were preprocessed before training and testing. PyTorch’s (Version 2.5.1) Dataset class was employed to index image files and define the procedure for loading individual data points. Image transformations were subsequently applied to prepare the data for the model and enhance it through augmentation techniques. The dataset was then partitioned using random_split into a training set (80%) and a testing set (20%). Each subset was wrapped in a DataLoader to enable efficient processing, with batches consisting of 32 samples. Images were converted into PyTorch tensors, and pixel values were rescaled to the floating-point range [0.0,1.0]. To align classification targets with this scale and stabilize multi-task training, age values extracted from file names were normalized to [0.0,1.0].

3.1.2. Training Model

We developed a multi-task knowledge distillation framework that transfers deep knowledge from a ResNet50 teacher to a lightweight MobileNetV2 student for accurate and efficient facial attribute prediction (age, gender, and race). We trained a MultiAttributeModel based on Res-Net50 to set a high-performance baseline. Then, we trained a MultiAttributeModel based on MobileNetV2 from the ground up using only the correct labels. Next, we trained a MultiAttributeModel based on MobileNetV2 using our own MultiTaskDistillationLoss.

The results show how useful our suggested method is. The standard ResNet50 and MobileNetV2 models have a fully connected layer at the end that classifies images into 1000 classes in ImageNet. We changed the last layer of these models to an nn.Identity() layer so that they work for our needs. This cuts off the network’s head, turning the whole pre-trained architecture into a strong, fixed-size feature extractor. The modified backbone produces a high-dimensional feature vector that gives a detailed semantic summary of the input facial image. The overall loss function for the distilled student was a weighted combination, controlled by an alpha hyperparameter (e.g., 0.7), balancing the contribution of the hard losses (from ground-truth labels) and the soft, distilled losses (from the teacher’s guidance). The learning rate is 1 × 10⁻⁴.

After training, the teacher model was tested on a set of data that it had never seen before to see how well it could generalize. We monitored the training process of the teacher model to ensure its stability. The training logs from the Jupyter notebook show that the model learned well on all three tasks.

3.2. Result and Analysis

We used a custom MultiTaskDistillationLoss to train the student. This loss is a mix of a hard loss and a soft loss. For age, we used mean squared error (MSE) to see how close the teachers guess the student’s age. For gender and ethnicity, we used KL divergence loss on probabilities that have been softened by temperature to copy how the teacher classifies things.

3.2.1. Dataset

We obtained a dataset from Kaggle (UTKFace). It is a large-scale, challenging “In-the-Wild” Dataset with over 23,000 images. We split this data into 80% for training and 20% for testing.

3.2.2. Results

We trained and tested all of the models on the UTKFace dataset. For the age regression task, the evaluation metrics are mean absolute error (MAE). For the gender and race classification tasks, the metrics are classification accuracy, precision, recall, F1-score, and confusion matrices (Table 1, Figure 1).

Table 1. Accuracy of the model.

Model	Architecture	MAE of Age (Years Old)	Gender Accuracy	Race Accuracy
Teacher	ResNet50	5.80	98.31%	98.47%
Non-Distilled Student	MobileNetV2	12.42	73.93%	62.47%
Distilled Student	MobileNetV2	8.17	95.40%	93.51%

Figure 1. Teacher model training metrics, per epoch iteration.

The training metrics of the teacher model across 10 epochs are presented in Figure 2, Figure 3 and Figure 4. The total multi-task loss decreased consistently, beginning at approximately 1.57 and reaching 0.24 by the final epoch. The MAE for the age prediction task (non-normalized) also declined steadily, indicating effective learning of the age regression task. Classification accuracy for both gender and race improved rapidly during training. By epoch 10, gender accuracy increased from 73% to 97.29%, while race accuracy rose from 65% to 96.50%. The plot_training_metrics function illustrates these learning curves, confirming the observed trends: the loss decreased smoothly, and the accuracy improved steadily. These results suggest that the model avoided overfitting and successfully learned transferable features applicable to broader contexts.

Figure 2. Comparison of student models age prediction accuracy (distilled vs. non-distilled).

Figure 3. Comparison of student models gender prediction accuracy (distilled vs. non-distilled).

Figure 4. Comparison of student models race prediction accuracy (distilled vs. non-distilled).

4. Conclusions

This thesis investigates the trade-off between prediction accuracy and inference speed in modern deep learning. It focuses on multi-attribute facial prediction, aiming to design a lightweight yet accurate model for age, gender, and race estimation. The proposed approach enables efficient deployment on resource-constrained devices such as mobile and edge systems.