Emotion Analysis for the Protection of Industrial Enterprises and Institutions via the IoT ^†

Abstract

Emotion analysis through facial recognition has significant potential for enhancing security within organizational and industrial contexts by enabling early risk detection, reducing human error, and supporting real-time decision-making. However, existing systems often demand high computational resources and advanced imaging equipment, limiting their adoption in small and medium-sized facilities. This study presents a cost-effective and practical emotion analysis application based on facial recognition, designed for seamless integration into diverse environments and IoT platforms. The system dynamically adapts its operation to the emotional state of the authorized user, mitigating potential errors when psychological resilience is insufficient. A Convolutional Neural Network (CNN) model was developed for emotion analysis using Conv2D, MaxPooling2D, and Dropout layers. Additionaly, several face recognition algorithms were tested to ensure reliable identification under different conditions. The proposed emotion analysis model achieved 98% overall accuracy and an 89% weighted F1 score. The system was trained and tested on a dataset with 58,179 training images and 28,706 test images, reaching the highest validation accuracy of 98.71% at the 25th epoch. By using a face recognition library, the system provided both high accuracy and fast processing. The developed face recognition-based folder security system offers a more user-friendly and innovative alternative to traditional password-based security methods.

1. Introduction

Deep learning, a recent advancement in artificial intelligence, has emerged due to its ability to minimize human errors. It demonstrates strong perception and interpretation capabilities across various domains by processing large datasets. The rapid growth of labeled data, particularly in image processing, has further increased the demand for deep learning algorithms [1]. Image processing techniques are widely applied in fields such as medicine, the military, industry, and geographic systems, and they play a crucial role in security systems. Applications like fingerprint, iris, and face recognition are made possible in the security domain through these techniques. To interpret images captured via scanners, cameras, or other imaging devices accurately, they must first undergo certain pre-processing steps. Numerous libraries and tools, available in various software development languages, facilitate this process [2]. Research findings highlight the potential of technology in facial recognition and the understanding of emotional intelligence, emphasizing the importance of advancements in this area. This study contributes to the scientific literature by enhancing the interpretation of facial expressions and the analysis of their emotional content. It introduces an original approach that expands the ability of technology to understand and interpret human emotions. Rapidly evolving digital technologies are revolutionizing human–computer interaction.

The automatic recognition of facial expressions and emotional responses holds significant potential in diverse fields, including psychology, security, education, and marketing. In this research, datasets were trained using advanced image processing and machine learning techniques. Human facial expressions were analyzed in detail, and emotional states were classified with high precision into categories such as angry, disgusted, fearful, happy, neutral, sad, and surprised. Automation of dangerous or repetitive tasks necessitates continuous supervision and management to ensure workplace safety and employee health. This study focuses on safeguarding the system and control software in the central computer control room of industrial automation systems, which is located in the management room and accessible only to authorized personnel [3]. Protecting critical system files is essential for maintaining security. In industrial automation systems, the central computer control room plays a vital role in ensuring the integrity of system and control software. Unauthorized personnel are prevented from accessing sensitive files, whether accidentally or intentionally. Improper interventions in control software can compromise performance and reliability. The implemented system prevents unauthorized employees from accessing critical system files. Additionally, as a second phase, emotion analysis is performed on images of authorized personnel. This step aims to reduce potential errors caused by lapses in attention if the person in charge lacks the psychological resilience necessary to manage the system effectively, depending on their emotional state [4]. The application of this study was designed to secure critical management units in industrial automation systems. However, the methods can be easily adapted to other domains.

A Convolutional Neural Network (CNN), capable of efficient analysis and classification, was developed using Conv2D layers, MaxPooling2D, Dropout layers, Flatten layers, fully connected layers, and Softmax activation functions combined with ReLU [5]. The folder security system allows users to hide folders or unlock a door that becomes accessible only to recognized individuals. High-accuracy models were developed using image-based datasets, and these were integrated into a user-specific folder security system [6]. This approach allowed precise classification of emotional states from human facial expressions while controlling file access through face recognition [7]. The labeled image processing data are combined with deep learning algorithms.

2. Materials and Methods

Face detection is the process of identifying the presence and location of human faces in an image or video frame. This is the first and most critical step in face recognition systems. Successful face detection directly affects the accuracy of subsequent feature extraction and recognition steps [8]. Cascade classifiers based on Haar features, such as the Viola–Jones algorithm [9], have been used as the standard in this field for many years. Deep learning-based methods, especially CNNs, provide high-performance face detection even in different poses, lighting, and complex background conditions. Once the face is detected, the distinctive features that make the individual unique need to be extracted. After the face is detected, the distinguishing features that make the individual unique must be extracted. These features can be geometric, such as distances and proportions between the eyes, nose, and mouth, or photometric, such as gray levels or color patterns of the facial surface. Principal Component Analysis (PCA), also known as Eigenfaces MA, USA, reduces face images from a high-dimensional space to a lower-dimensional feature space and obtains principal components that maximize the variance [9]. Local Binary Patterns (LBP) is a powerful verify used to analyze facial composition. It generates a binary code by comparing each pixel with its neighbors, and the histograms of these codes are used as feature vectors [10]. Deep learning-based features used for modern face recognition systems, especially CNNs, automatically learn hierarchical features from an image. These networks build increasingly complex and abstract representations starting from raw pixel data. Face recognition systems aim to detect a face in an image and identify the person using mathematical methods. The system first converts the input image into a pixel matrix. An image is essentially a two-dimensional matrix of numerical values, where each pixel represents brightness or color information. Small matrices called filters or kernels are applied over the image afterward. This process is called convolution and is represented by Equation 1, as follows:

$$I\ast K(i,j)=\sum m\sum nI(i-m,j-n)K(m,n)$$

(1)

During this operation, the filter slides across the image to extract important features such as edges, eyes, the nose, and the overall shape of the face. In this equation: (I) represents the input image, (K) represents the filter kernel, (i,j) represent the current position in the image, (m,n) represent the coordinates inside the kernel. After convolution, activation functions are used to make the extracted features more meaningful. The most commonly used activation function is ReLU, as given in Equation 2.

$$\mathrm{f}\left(\mathrm{x}\right)=\mathrm{m}\mathrm{a}\mathrm{x}(0,\mathrm{x})$$

(2)

$$\mathrm{f}\left(\mathrm{x}\right)=\left\{\begin{array}{c}0,\mathrm{x}<0\\ \mathrm{x},\mathrm{x}\ge 0\end{array}\right.$$

(3)

Here, (x) represents the input value given to the function. For example, it can be a value coming from a neuron in a neural network. (f(x)) represents the output value produced by the function. The ReLU function returns 0 when the input is negative and keeps the value unchanged when the input is positive, as seen in Equation 3. Next, pooling layers are applied to reduce the dimension of the data and remove unnecessary information, making the system more efficient and faster. In the next stage, the network converts the face into a numerical feature vector called a “face embedding.” Each face is represented as a vector of numbers, typically with 128 or 512 dimensions. To determine whether two faces belong to the same person, the distance between these vectors is calculated. One of the most common methods is Euclidean distance, shown in Equation 4.

$$\mathrm{d}=\sqrt{{\displaystyle {\sum }_{i=1}^n}(xi-{yi)}^2}$$

(4)

Here, (d) represents the distance between two faces, (xi) represents the feature values of the first face vector, (yi) represents the feature values of the second face vector, (i) is the index of each feature, and (n) is the total number of features. If the resulting distance is small, the system concludes that the faces belong to the same person; otherwise, they are considered different. Libraries such as OpenCV implement these mathematical operations to enable real-time face detection and recognition. Common methods, including Haar Cascade, Eigenfaces, LBPH, and CNN-based models, are widely used within OpenCV. The obtained image is analyzed with programming codes using methods defined in the Open CV library [11].

2.1. Emotion Analysis

Emotion analysis from facial expressions uses visual cues on the human face to analyze an individual’s emotional state (happiness, sadness, resentment, etc.) and identify it. It has potential applications in many areas such as enriching human–computer interaction, understanding psychological states, and improving marketing strategies. The six basic universal emotions identified by Paul Ekman—anger, disgust, fear, happiness, sadness, and surprise [10] are often taken as the basis for studies in this field. In this study, in addition to these six emotions, the “neutral” state was also included, and a seven-class analysis was conducted. The emotion analysis process generally includes the following steps: Face Detection: Detection of the face area in the image. Feature Extraction from Facial Expressions: Features such as the positions of prominent points on the face (e.g., eye corners, mouth edges, eyebrows), their geometric relationships, or textural patterns are extracted. Facial Movements Coding System (Facial Action Coding System—FACS) [10] is a comprehensive system used to analyze emotional expressions by identifying muscle movements. Classification: The extracted characteristics are assigned to one of the predefined emotion classes using machine learning or deep learning models. In this work, a model based on Convolutional Neural Networks (CNN) was developed for emotion classification. Machine learning is a subfield of artificial intelligence that enables computer systems to learn from data without being explicitly programmed and to perform specific tasks as a result of this learning [11]. An illustration of the relationship between AI, ML, DL, and CNN is given below in Figure 1.

Figure 1. An illustration of the relationship between Artifical Intelligence, Machine Learning, Deap Learning and CNN.

Machine learning is mainly divided into three categories according to the way it learns. Supervised Learning: The model is trained using labeled data (input-output pairs). The goal is to learn a mapping function from inputs to outputs. Classification (such as sentiment analysis) and regression problems fall into this category. The sentiment classification model in this work was trained with supervised learning. Unsupervised Learning: The model works with unlabeled data and tries to discover hidden structures or patterns in the data. Clustering and size reduction are common examples. Reinforcement Learning: An agent learns the best strategy by acting in an environment and receiving rewards or punishments according to the results of its actions. Machine learning has revolutionized many areas such as image recognition, natural language processing, medical diagnosis, financial forecasting, and more. In this thesis, machine learning principles and especially deep learning methods were used to solve emotion analysis and face recognition problems.

2.2. Deep Learning

Deep learning is a branch of machine learning that learns complex features from data in a hierarchical manner using multilayered artificial neural networks. It has achieved great success, especially in processing unstructured data such as images, sound, and text.

Convolutional Neural Networks (CNNs) are a deep learning architecture designed specifically for processing data with a grid-like topology (e.g., images). CNNs reduce the number of parameters while effectively learning spatial hierarchies in the data, thanks to shared weights and local connections. Various Python3 libraries were used in the realization of this thesis study; these are: Keras 3.15.1, a high-level API used to create and train deep learning models. It can work on lower-level libraries such as TensorFlow, Theano, or CNTK. In this thesis, it was used for the development and training of the CNN model. Scikit-learn 1.9.0 is a comprehensive machine learning library. It includes classification, regression, clustering algorithms, data preprocessing tools (e.g., data splitting, normalization), and model evaluation metrics. NumPy 2.0.0 is the core package for scientific computing in Python3. It allows efficient operations on multidimensional arrays and matrices. Pandas 3.0 provides powerful and flexible data structures (such as DataFrame) for data manipulation and analysis. It is widely used for data reading, writing, cleaning, and transformation operations. OpenCV 5.0.0 is an open-source computer vision and machine learning library. In this study, image data sets with different characteristics were used for two main tasks: emotion analysis and face recognition.

2.3. Data Set for Emotion Analysis

In order to train and test the emotion recognition model, the data sets named “dataset3modified” (training set) and “dataset-1-modified” (test set), which were shared on the Kaggle platform (https://www.kaggle.com/, accessed on 14 March 2026) and used specifically for this study, were taken as the basis. These data sets consist of facial expressions containing the following seven basic emotion categories: angry, disgust, fear, happy, neutral, sad, and surprised [12]. The class distribution of the data sets used is detailed in Table 1. A total of 58,179 images were used for the training set, and a total of 28,706 images were used for the test set. This training and test separation was used to measure the model’s generalization ability.

Table 1. Dataset used for sentiment analysis class distribution.

Emotion Class	Dataset-3-Modified	Dataset-1-Modified	Sum
Angry	8.164	3.997	12.161
Disgust	998	419	1.417
Fear	8.248	4.237	12.485
Happy	14.396	7.244	21.640
Neutral	10.104	4.783	14.887
Sad	9.742	4.837	14.579
Surprise	6.527	3.189	9.716
Sum	58.179	28.706	86.885

The effectiveness of the folder security system depends primarily on the accuracy of the face recognition component and the overall functionality of the system.

2.4. Emotion Analysis Through IoT

Security camera images can be sent to a central server via IoT for analysis. After the required identification and processing steps are completed, commands can be sent back through IoT to devices such as Raspberry Pi 4 Model B (Raspberry Pi Ltd., Cambridge, United Kingdom), ESP32-WROOM-32 (Espressif Systems, Shanghai, China), or Arduino Uno (Arduino S.r.l., Turin, Italy) to control servo or stepper motors [13]. Similarly, for facial recognition, IoT devices can transmit face images to a cloud server where face recognition and emotion analysis programs run. After the analysis results are obtained, servo motors can be activated again through IoT devices to open door locks. After the necessary identification and processing are completed, commands can be sent back through IoT to devices such as Raspberry Pi, ESP32, or Arduino to trigger servo or stepper motors. All methods described above may be adapted to an IoT system for facial emotion recognition [14]. In this way, a picture taken of a person is aligned, extracted, and identified for face recognition and transmitted through the IoT cloud to the server for emotion analysis, as seen in Figure 2.

Figure 2. Proposed cloud-based face recognition system.

Image acquisition: The process starts with capturing or uploading an image from a digital source, such as a camera, mobile device, or document scanner. The steps of the working system are shown in the UML diagram given in Figure 3 below. These steps are as follows. Preprocessing and enhancement: The raw image is processed to improve quality, adjust contrast, reduce noise, and resize or normalize input data for optimal recognition.

Figure 3. UML diagram of the proposed cloud-based recognition system (Details are indicated in yellow).

The AI model analyzes key visual elements, such as shapes, colors, textures, and edges, extracting meaningful features from the image. Model inference and classification: The extracted features are fed into a deep learning model trained to recognize specific objects, text, or faces. The model predicts categories, labels, or patterns in the image. Figure 4 below shows the real simulation of the proposed system. According to this, the image is taken with the PC camera, and face recognition is performed. Afterward, the picture is sent over IoT to the main server on the cloud for emotion analysis.

Figure 4. Real workplace of the proposed system.

Post-processing and insights generation: The system refines results by filtering irrelevant data, removing false positives, and structuring output insights for decision-making.

3. Results and Discussion

The performance of the face recognition and emotion analysis models developed within the scope of the study, and the application results of the innovative folder security system in which these models are integrated, are presented in detail, and the findings obtained are evaluated. The loss and accuracy metrics obtained during the training and verification process of the CNN model, which was developed to classify seven basic emotions (anger, disgust, fear, happiness, neutral, sadness, and surprise) from facial expressions, are presented in Figure 5 and Figure 6.

Figure 5. Emotion analysis model training and validation loss graphs.

Figure 6. Emotion analysis model training and validation accuracy graphs.

As illustrated in Figure 5, the training loss of the model consistently decreased as the number of training epochs increased, indicating that the model gradually learned the underlying patterns in the dataset more effectively over time. A similar trend can also be observed for the validation loss, which steadily declined throughout the training process. This decrease in both training and validation loss values suggests that the model improved its generalization capability without showing significant signs of overfitting during the 25 epochs. Furthermore, as presented in Figure 6, both the training accuracy and validation accuracy of the model increased progressively with each epoch. The steady improvement in accuracy metrics demonstrates that the model became more successful in correctly classifying the input data as training continued. According to the shared training results, by the end of the 25th epoch, the model achieved a validation accuracy of approximately 98.13% with a validation loss of 0.0287. These results indicate that the proposed model provided a strong and stable performance, achieving a high level of classification accuracy while maintaining a low validation loss value. This result shows that the model learned the patterns in the dataset with very high success.

The overall performance of the model on the 28,706-picture test set and the values of precision (precision), sensitivity (recall), and F1-score for each emotion class are shown in Table 2. The model achieved a 99% weighted average F1 score and 98% overall accuracy on the test set. Additionally, the confusion matrix showing which emotions the model confused with each other is given in Figure 6. When the confusion matrix and class-based metrics are examined, the model performs well across all emotion classes. The model performed very well within each class, with minimal confusion between classes. For example, 7202 out of 7244 samples were correctly classified for the “happy” class, while all 419 samples in the “disgust” class were correctly predicted, as seen in Table 2. The number of misclassifications in other classes was also quite low: 3967 out of 3997 samples for “angry” (angry), 4026 out of 4237 samples for “fear”, 4764 out of 4783 for neutral, 4799 out of 4837 for sad, and 3173 out of 3189 for surprise were correctly classified.

Table 2. Sentiment analysis model class-based performance metrics.

Emotion Class	Precision	Recall	F1-Score	Support
Angry	0.98	0.98	0.99	3.997
Disgust	0.98	1.00	0.98	419
Fear	0.98	0.99	0.99	4.237
Happy	1.00	0.99	1.00	7.244
Neutral	0.99	0.99	0.99	4.783
Sad	0.99	0.99	0.99	4.837
Surprise	0.99	1.00	0.98	3.189
Accuracy				0.98
Macro Avg.	0.98	0.99	0.99	28.706
Weight Avg.	0.99	0.99	0.99	28.706

This shows that the model can distinguish each emotional expression with high accuracy. The emotional analysis of the model confusion matrix is shown below in Table 3.

Table 3. Emotional analysis model confusion matrix.

True Labels	Angry	Disgust	Fear	Happy	Neutral	Sad	Surprise
angry	3967	5	9	1	4	8	3
disgust	1	419	0	0	0	0	1
fear	17	2	4026	11	22	25	34
happy	5	1	1	7202	3	1	29
neutral	4	0	3	4	4764	5	0
sad	12	0	14	2	9	4799	0
surprise	2	2	11	0	0	0	3173

4. Conclusions

The proposed emotion recognition framework demonstrated strong performance, attaining an overall accuracy of 98% and a weighted F1 score of 99% on the specialized datasets, namely “dataset3modified” comprising 58,179 training images and “dataset-1-modified” containing 28,706 test images. The highest verification accuracy, 98.71%, was observed at the 25th epoch. By utilizing the face_recognition library, the system achieved both high precision and efficient processing speeds. The implemented facial recognition-based folder security solution provides a practical and intuitive alternative to traditional password-based methods, offering enhanced usability and innovative security functionality. The face recognition module worked with high accuracy and speed by taking advantage of the power of the face_recognition library. The developed facial recognition-based folder security system successfully fulfilled the specified scenarios and offered a user-friendly and innovative security solution as an alternative to traditional password-based methods. Measures such as alarms, photo recording, and log-off, developed especially against unauthorized access attempts, increase the security level of the system. Currently, the system primarily focuses on a single authorized user.

Future work could extend the framework to monitor multiple users simultaneously through IoT, adapting security responses based on the emotional states of several individuals in shared environments. Combining facial recognition with additional physiological or behavioral data (e.g., voice, heart rate, body posture) could improve the robustness and reliability of emotion detection, especially in challenging lighting or occlusion conditions. To further reduce computational requirements, deploying the system on edge devices with optimized models could enable faster and offline processing, making it suitable for environments with limited network connectivity or computational resources.