Non-Invasive Multivariate Prediction of Human Thermal Comfort Based on Facial Temperatures and Thermal Adaptive Action Recognition

Abstract

Accurately assessing human thermal comfort plays a key role in improving indoor environmental quality and energy efficiency of buildings. Non-invasive thermal comfort recognition has shown great application potential compared with other methods. Based on thermal correlation analysis, human facial temperature recognition and body thermal adaptive action detection are both performed by one binocular infrared camera. The YOLOv5 algorithm is applied to extract facial temperatures of key regions, through which the random forest model is used for thermal comfort recognition. Meanwhile, the Mediapipe tool is used to detect probable thermal adaptive actions, based on which the corresponding thermal comfort level is also assessed. The two results are combined with PMV calculation for multivariate human thermal comfort prediction, and a weighted fusion strategy is designed. Seventeen subjects were invited to participate in experiments for data collection of facial temperatures and thermal adaptive actions in different thermal conditions. Prediction results show that, by using the experiment data, the overall accuracies of the proposed fusion strategy reach 82.86% (7-class thermal sensation voting, TSV) and 94.29% (3-class TSV), which are better than those of facial temperature-based thermal comfort prediction (7-class: 78.57%, 3-class: 90%) and PMV model (7-class: 20.71%, 3-class: 65%). If probable thermal adaptive actions are detected, the accuracy of the proposed fusion model is further improved to 86.8% (7-class) and 100% (3-class). Furthermore, by changing clothing thermal resistance and metabolic level of subjects in experiments, the influence on thermal comfort prediction is investigated. From the results, the proposed strategy still achieves better accuracy compared with other single methods, which shows good robustness and generalization performance in different applications.

1. Introduction

Over the past decade, population growth, increased indoor time, and the demand for enhanced indoor environmental quality have significantly impacted building energy consumption [1]. Research indicates that people now spend more than 80% of their time indoors, underscoring the critical role of indoor environments in personal and professional life [2,3,4,5,6]. In the United States, HVAC systems’ energy consumption has comprised 50% of total building energy usage [7], and the proportion in China is similar. However, personal dissatisfaction with the indoor thermal environment is still high [8,9]. For large-space buildings, the spatiotemporal uneven distribution of environmental parameters further exacerbates this thermal discomfort [10,11]. Analyzing the reasons, on the one hand, the thermal comfort of the human body is not only affected by external environmental factors, but also related to inner metabolic level and psychological factors. On the other hand, current thermal comfort evaluation methods primarily focus on the overall thermal comfort of the population, neglecting individual differences in thermal sensation [12]. Therefore, designing a fast and accurate identification method for thermal sensations of indoor residents is of great significance for improving the quality of building environment and energy efficiency.

1.1. Relevant Research

For human thermal comfort assessment, the predictive mean voting (PMV) method has been widely used and been recognized by ASHRAE [13] and ISO 7730 [14]. It divides human thermal sensation level into multiple classes by considering both environmental and physiological factors, including indoor air temperature, relative humidity, average radiation temperature, wind speed, clothing thermal resistance, and metabolic rate [15,16]. However, PMV shows difficulty at reflecting the difference in thermal comfort caused by dynamic environmental changes or population differences. With the rapid development of sensor and machine learning technologies, more and more attention has been paid to on-site data acquisition and data-driven thermal comfort modeling. Zhou et al. [17] used support vector machine (SVM [18,19]) and a public data set (RP-884 database) for thermal comfort modeling. Compared to the PMV model, the SVM’s sum of squares for residuals (SSE) was reduced by 96.4%. Similarly, Jiang and Yao [20] designed an SVM-based personal thermal sensation model and achieved better predictive performance than the PMV model. Luo et al. [21] compared the applications of logistic regression (LR), gradient boosting (GB), SVM, and random forest algorithms (RF [22,23]) in predicting thermal sensation voting (TSV). RF model based on a parallel ensemble learning strategy achieved the best performance. Similar to RF, the AdaBoost model using a serial ensemble strategy was also reported for personal thermal comfort predictions [24,25].

In data-driven modeling, the question of how to access reliable thermal sensation data plays the key role [26,27]. The collection of approaches can be roughly divided into three categories: traditional invasive measurements, semi-invasive, and non-invasive measurements [28,29]. For invasive methods, physical contact and subject cooperation are required during the measurement. The foreign body sensation during data collection can inevitably affect the accuracy of physiological markers and questionnaire surveys to some extent [30,31,32]. The semi-invasive methods mainly integrate sensors into wearable devices [33,34], which can collect various physiological data more conveniently, such as skin temperature [35], blood pressure, and heart rate [36]. This method reduces the physical contact between subjects and sensors. The foreign body sensation is relieved, but the subjective influence cannot be completely eliminated.

In recent years, non-invasive methods have shown great potential in personal thermal comfort assessment [37]. By using remote monitoring devices such as infrared camera, the measurement interference on human bodies is minimized [38]. At the same time, the data collection process can be quick and real-time, which makes HVAC control based on human thermal comfort assessment more achievable. At present, remote detection of human body temperature is the main non-invasive measuring approach used for thermal comfort prediction [39,40,41]. For example, Cheng et al. [42] adopted a contactless approach to measure human body temperatures by correlating skin temperature with skin color saturation using Eulerian video magnification (EVM). Jeoung et al. [43] recognized the ROI of the human face based on 68 facial feature points in thermal images, and predicted thermal comfort with an accuracy of 90.26% (3-class, TSV). Bai et al. [44] extracted facial features and applied high-resolution feature learning networks (HRNet) to obtain infrared skin temperatures and skeletal points. Li et al. [45] applied the YOLO algorithm (You Only Look Once [46]) to recognize infrared facial images at multiple angles/distances, and extracted two facial key-region temperatures (nose, cheek) for thermal comfort prediction with an accuracy of 85.68% (3-class). It is noted that inappropriate camera angles or facial covering would result in degraded accuracy or even prediction failure.

By using a contactless method for detecting the human body’s thermal adaptive actions, the corresponding thermal comfort level may also be evaluated. In this respect, Meier et al. [47] used Kinect devices to detect four human postures (self-embrace, button-up, pull collar, and wipe eyebrows). Yang et al. [48,49] identified 12 thermal comfort-related actions based on 369 questionnaires, and the OpenPose tool was used to detect these actions for predicting thermal comfort. It is noted that, although thermal adaptive actions are directly driven by thermal sensation, they do not occur continuously, which would lead to interruption of thermal comfort assessment. Based on the categories and detection objects of images,

Table 1. Related studies on human thermal comfort prediction by different contactless measurements.

Literatures	Detection Object/Image Type	Detection Tool	Thermal Comfort Prediction Model	TSV Class	Accuracy
Jeoung et al. (2024) [43]	facial temperature/infrared	YOLOv5	MLP, GBM, KNN, SVM, RF, DT	3-class *	90.26% (MLP)
Bai et al. (2024) [44]	facial temperature/infrared	Dlib/HRNet	BL, RF, GBM, GBDT, DCF	3-class	90.44% (BL)
Li et al. (2023) [45]	facial temperature/infrared	YOLOv5	CLPSO-SVM	7-class **	81.65%
				3-class	85.68%
Wu et al. (2022) [50]	facial temperature/infrared	not reported	SVM	2-class	79.9%
Cosma et al. (2019) [51]	facial temperature/infrared and RGB	Haar/OpenPose	SVM	7-class	76%
Li et al. (2018) [52]	facial temperature/infrared	Haar Cascade	RF	3-class	85%
Garg et al. (2023) [53]	body action/RGB	Mediapipe	CNN	not reported	97.09%
Makhijani et al. (2023) [54]	body action/RGB	Mediapipe	CatBoost	not reported	98.9%
Bucarell et al. (2023) [55]	body action/RGB	TensorFlow Object Detection API	ResNet101 + LSTM + MLP	3-class	86.7%
Duan et al. (2021) [56]	body action/RGB	OpenPose	ST-GCN	3-class	78%
Yang et al. (2019) [48]	body action/RGB	OpenPose	not reported	7-class	86.37%
Meier et al. (2017) [47]	body action/RGB	Kinect	not reported	not reported	not reported

* 3-class: warm discomfort (+1), comfort (0), and cool discomfort (−1); ** 7-class: hot (+3), warm (+2), slightly warm (+1), neutral (0), slightly cold (−1), cool (−2), and cold (−3).

briefly summarizes personal thermal comfort assessments by different contactless measurements. The used data-driven models with reported accuracies are also provided.

From the brief review of non-invasive studies, it is seen that human thermal comfort can be accurately estimated by contactless measurements that minimize subjective influence. It is also found that current studies mainly focus on single non-invasive measuring methods for thermal comfort assessment. When the detection conditions are not satisfied, it will easily lead to degraded accuracy or prediction failures [45]. Regarding this research gap, a combination of multiple non-invasive detection methods may compensate for their respective defects, making thermal comfort prediction more reliable and robust. The impact of parameter changes (such as clothing thermal resistance and physiological metabolism) on the performance of thermal comfort assessment needs to be well understood [57,58]. A more comprehensive case study simulating practical applications is required.

1.2. Main Contributions

To address the above-mentioned issues, this study analyzes features of facial temperature measurement and thermal adaptive action detection through field experiments. In order to make use of their respective advantages, a non-invasive multivariate prediction strategy for human thermal comfort is designed. The main contributions include the following:

(1) By using one binocular infrared camera, facial key-region temperatures and probable thermal adaptive actions are detected. The YOLOv5 algorithm is applied for detection of facial key regions, then the RF model is used for facial temperature-based thermal comfort prediction. Meanwhile, skeletal-point data of six thermal adaptive actions are detected by combining YOLOv5 and Mediapipe tool. The action recognition model is built by RF, and the corresponding TSV level is determined by survey questionnaire results.

(2) A comprehensive framework for multivariate thermal comfort prediction is proposed by combing results from facial temperature recognition, thermal adaptive action detection, and PMV model. The fusion strategy is based on a weighted average method. The higher the single model’s recognition accuracy, the greater the corresponding weight.

(3) Three field experiments are designed. Experiment I collects data for training the prediction model based on facial temperature. Experiment II and experiment III collect data including thermal adaptation behavior to test the performance of fusion strategies under different conditions (normal office condition, change clo, change met, and both change clo and met). The robustness and generalization performance of the proposed strategy are investigated by these experiments, and the results are discussed in detail.

1.3. Paper Organization

Section 2 describes the methods used of facial recognition and thermal adaptive action detection. The data collection experiments and the overall strategy are both presented in this section. Section 3 comprehensively analyzes the results of detection algorithms and fusion strategy. Section 4 presents the study’s findings and its limitations. Section 5 briefly offers the conclusions of this study.

2. Methods

2.1. Multi-Region Facial Recognition Method

(1)

Object detection algorithm

For the detection of human facial key regions, the object detection algorithm YOLOv5 is applied. Compared with other object detection algorithms, YOLO algorithm [59] is widely used because of its efficiency, reliability, and ease of use in recognizing objects within images. The network structure of YOLOv5 is mainly divided into three parts: backbone network, neck network, and head network [60]. The primary function of the backbone network is feature extraction, which extracts object information from images through convolutional neural networks and continuously shrinks the feature map for subsequent object detection. The main structures of backbone include three submodules, i.e., the conv module, C3 module, and SPPF module [61]. The function of the neck network is to combine shallow graphic features with deep semantic features to obtain more complete features, thereby enhancing the network’s robustness and object detection ability. The obtained features are then transmitted to the head network for prediction. The head network deduces and outputs the class and location information of recognized objects.

In practice, the trained YOLOv5 model identifies facial key regions and provides the position coordinates of detection boxes; using these as input, the average temperature of each region is extracted. The temperature extraction function refers to the Software Development Kit (SDK) provided by infrared thermal imager manufacturers (MAGNITY, China). The extracted average temperatures of facial regions are used for TSV prediction, which will be described in Section 2.3.

(2)

Dataset collection of thermal images

According to the sensitivity analysis results in our previous research [45], the nose and cheeks are selected from six facial regions of interest (ROI, nose, cheeks, forehead, mouth, and chin) as the key regions (see Figure 1a). Considering that the angles and distances between the camera and the object probably affectthe detection performance, four distances (1 m, 2 m, 3 m, and 4 m) and nine angles (0–180°) are set for image collection. All collected images are labeled in forms that the YOLOv5 network can use by the “LabelImg” tool. A total of 898 effective infrared images are collected, and through data enhancement (including adding Gaussian noise, changing the brightness of images, cropping, translation, and mirroring), the infrared images used are increased to 5394. Figure 1b shows one typical subject’s nine infrared images from nine different angles (two meters’ distance from the camera).

2.2. Thermal Adaptive Action Recognition Method

(1)

Action detection algorithm

For the thermal adaptive action detection, the Mediapipe tool box (developed by Google [62]) is applied in this study. It is based on a lightweight convolutional neural network architecture called BlazePose [63] for human posture estimation, and is characterized by ease of use, modular architecture, and good real-time performance. The topological structure of human posture consists of 33 key skeletal points [64]. A detector–tracker machine-learning pipeline [65] is used for two-stage human posture detection. Firstly, the pipeline uses the detector to locate ROI in the frame. Then, the tracker predicts the human posture’s landmarks (33 skeletal points) within the ROI. For this study, the detection of thermal adaptive actions has the following steps: (S1) Shoot video of thermal adaptive action and label it with corresponding action type. (S2) Input the video into Mediapipe, and detect/extract the position information of 33 skeletal points with a confidence level higher than 0.5. (S3) Use the skeletal points’ position data as inputs for data-driven model’s training and action prediction.

Considering practical application scenarios, a multi-person thermal adaptive action detection strategy is further designed by combining Mediapipe with the YOLO object detection algorithm. The main steps include the following: (S1) Identify human bodies in the multi-person image and obtain the position coordinate of each person’s recognition box by using the YOLO algorithm. (S2) Crop the multi-person image based on the recognition boxes, and transform the multi-person image into multiple single-person images. (S3) Detect the thermal adaptive action on each single-person image sequentially, and put back the detected images to achieve multi-person thermal adaptive action recognition. The schematic diagram is shown in Figure 2.

(2)

Dataset collection of skeletal points

Based on previous studies [48,49], six related thermal adaptive actions are selected for this study, which include wiping forehead, fanning oneself, shaking T-shirt, embracing both arms, blowing on hands, and rubbing hands. Considering that excessive angles and distances between the camera and human body probably affect the detection performance in practical applications, the thermal adaptive action video shooting is performed at three distances (1 m, 2 m, and 3 m) with three angles (45°, 90° and 135°), respectively. A total of 378 videos with a duration of approximately 6 s are recorded, with a total duration of 2268 s. Input the collected video into Mediapipe, and detect/extract the position information of skeletal points with a confidence level higher than 0.5. In this way, a dataset of thermal adaptive action skeletal points is constructed containing a total of 65,071 sets of skeletal points. Figure 3 records eighteen frames of six thermal adaptive actions performed by the typical subject (at three different angles from the camera). According to the visual changes between consecutive frames, the change of skeletal points is displayed by extracting one frame every ten. As an example, Figure 4 shows the changes in skeletal points while performing action of “fanning oneself” (three angles: 45°, 90°, and 135°).

(3)

Determination of TSV levels corresponding to thermal adaptive actions

In order to determine the correspondence between each thermal adaptive action and the thermal sensation level (7-class), this study uses the result of survey questionnaires from 100 college students (98 valid questionnaires received). The survey questionnaire is designed in the form of an online answer sheet, details of which are shown in the Appendix A (Figure A1). It is noted that thermal sensations represented by people’s thermal adaptive actions are subjective to some extent. The statistical results of the voted thermal sensation level represented by each action are shown in Figure 5. For this study, the thermal sensation level of each action is determined by the weighted sum of voting proportions on all levels. Based on the results of survey questionnaires, the TSV levels of all six thermal adaptive actions are set as follows: wiping forehead (2.66), fanning oneself (2.47), shaking T-shirt (2.5), embrace both arms (−2.16), rubbing hands (−2.38), and blowing on hands (−2.56).

2.3. Thermal Comfort Prediction Model

(1)

Random forest

Random forest [66] belongs to a bagging-type ensemble learning algorithm, and is suitable for multivariate classification problems such as thermal sensation assessment [21]. It combines multiple weak classifiers and can achieve high accuracy and generalization performance through voting or taking the mean. In this study, RF is used for thermal comfort prediction based on human facial temperatures (called Model_Fac_T). The proposed thermal adaptive action detection model (called Model_Therm_Act) is also based on RF algorithm. The main steps of RF are as follows:

(S1) Given a dataset T (containing facial skin temperatures or thermal adaptive actions), randomly generate N sample sets with dropout based on bootstrap method. By using the N sample sets, train the corresponding decision trees: Tree-1, Tree-2, …, Tree-N.

(S2) For each node on the decision tree, randomly select m features from the M features in the sample set ($m\ll M$), and calculate the optimal splitting method with these m features based on the Gini index.

(S3) During the decision tree’s forming process, each node must be split according to Step 2 until it can no longer be split. No pruning operation is used during the entire forming process.

(S4) The random forest model is built by establishing multiple decision trees according to the above steps. Count the prediction results of each decision tree and use a voting mechanism to select the predicted value with the most occurrences as the final prediction result.

The schematic diagram of the basic RF and its applications in this study are shown in Figure 6.

(2)

Fusion strategy

By using an infrared camera and a temperature/humidity sensor, and by utilizing the above detection and modeling methods, human thermal comfort prediction models based on facial temperature and thermal adaptive actions can be obtained, respectively. In addition, through environmental temperature and humidity measurements, the traditional PMV result can also be estimated. The thermal comfort prediction results of these three types of models are combined together with a weighted fusion method. A non-invasive multivariate human thermal comfort prediction framework is designed for this study. The basic fusion equations are as follows:

$$L_{\mathrm{total}}={\alpha }_1\ast L_{fac\_T}+{\alpha }_2\ast L_{therm\_act}+{\alpha }_3\ast L_{pmv},$$

(1)

$${\alpha }_1=\left\{\begin{array}{cc}{P}_{YOLO}\ast P_{RF}/2\hfill & ifactiondetected,\hfill \\ P_{YOLO}\ast P_{RF}\hfill & ifnoactiondetected,\hfill \end{array}\right.$$

(2)

$${\alpha }_2=\left\{\begin{array}{cc}{P}_{mediapipe}/2\hfill & ifactiondetected,\hfill \\ 0\hfill & ifnoactiondetected,\hfill \end{array}\right.$$

(3)

$${\alpha }_3=1-{\alpha }_1-{\alpha }_2,$$

(4)

where $L_{Fac\_T}$ is the thermal comfort level predicted by Model_Fac_T, $L_{therm\_act}$ is the product sum of the predicted action’s voted TSV values, $L_{pmv}$ is the PMV value calculated under the ASHRAE standard [13], and $L_{total}$ is the fusion result. The weighting values of ${\alpha }_1$ to ${\alpha }_3$ are related to the accuracy of non-invasive recognition. The higher the accuracy, the greater the corresponding weight. Specifically, ${\alpha }_1$ represents the weight of facial temperature-based TSV prediction, which is composed of the product of $P_{YOLO}$ and $P_{RF}$, where $P_{YOLO}$ is the mean average precision of facial key-region recognition and $P_{RF}$ is the TSV prediction accuracy of the RF model (Model_Fac_T). ${\alpha }_2$ represents the weight of thermal adaptive action prediction, and $P_{mediapipe}$ is the recognizing accuracy of thermal adaptive actions based on Mediapipe. $P_{YOLO}$, $P_{RF}$, and $P_{mediapipe}$ are static values obtained in offline evaluation to reflect the relative contribution of each model in the fusion strategy. ${\alpha }_3$ is the weight of PMV result. Such weight setting can normalize the sum of three weights to be 1. When no thermal adaptive action is detected, ${\alpha }_2$ is set to zero, and the fusion result mainly depends on the prediction of facial temperature and PMV. When the detections of facial temperature and thermal adaptive action both fail, ${\alpha }_1$ and ${\alpha }_2$ are zeros, which means the prediction model degenerates into the traditional PMV model.

2.4. Overall Framework

Based on the methods described above, the overall framework of this study is specified as follows. By using one binocular infrared camera, facial key-region temperatures are detected by YOLO algorithm. Based on collected field experimental data set (experiment I, Section 2.5 (3)), the corresponding thermal comfort predictive model (Model_Fac_T) is built using RF algorithm. Meanwhile, skeletal-point data of six typical thermal adaptive actions are detected and collected by YOLO and Mediapipe tool. The corresponding action recognition model (Model_Therm_Act) is also built by RF. It is noted that classifier RF is selected by performance comparison of five different data-driven models. Additionally, the PMV result is also calculated through environmental temperature/humidity measurements.

In the proposed multi-variable fusion strategy of thermal comfort prediction, the three types of models are combined with a weighted average method. The weighting values of ${\alpha }_1$–${\alpha }_3$ are related to the accuracies of non-invasive recognitions. The higher the accuracy, the greater the corresponding weight. The performance validation of the proposed strategy is carried out by using data sets collected by experiments II and III (Section 2.5 (3)). Figure 7 shows the overall framework of this study.

2.5. Data Collection Experiment

(1)

Experimental environment and equipments

Data collection experiments are conducted in a standard office environment (7.0 m in length and 3.6 m width) at Jiangsu University in Zhenjiang, China. The indoor environment is regulated by a suspended air conditioning system with the controlled temperature range of 16–30 °C. The indoor wind speed is less than 0.1 m/s. The environmental parameters include indoor temperature ($T_{air}$) and relative humidity ($RH$), collected through sensors of DHT11 (Sinochip Electronics, China) and PT100 (YIMENGWEI, China). These sensors are placed on a test bench at a height of 0.7 m from the ground. This is close to the ASHRAE standard (0.6 m) [67]. A binocular infrared thermal imager MAG-RT384 (MAGNITY, China) is placed in front of the subject (2 m, 90°). The equipment has the temperature measurement accuracy of 0.4 °C, visible light resolution of 1920 ∗ 1080, and infrared resolution of 384 ∗ 288. The experimental site and room layout are shown in Figure 8.

(2)

Subjects and TSV levels

This study recruits 17 subjects for data collection experiments, including 10 males and 7 females, aged between 23 and 28, with BMIs within the normal range. The essential physiological statistical data of all subjects are listed in

Table 2. Physiological information statistics of the subjects.

Gender	Age	Height (cm)	Weight (Kg)	BMI (Kg/m2)
Men	24.4 ±1.58	177 ± 6.27	69.8 ± 9.23	22.28 ± 2.60
Female	23.9 ± 1.35	166 ± 2.44	56.43 ± 3.51	20.37 ± 1.16

. All subjects undergo a physical and mental health (stress level) assessment before the experiment. They are also required to avoid activities such as staying up late, taking medication, drinking alcohol, or engaging in vigorous exercise before the experiment. All subjects are required to wear typical summer clothing in the experiment. The calculation of clothing thermal resistance is based on the Evaluation Standard for Indoor Thermal and Humid Environment of Civil Buildings (CHINA, GB/T 50785-2012) [68]. Under standard experimental conditions, participants wear a T-shirt (0.15 clo; clo is a commonly used unit to represent the clothing thermal resistance), trousers (0.15 clo), underwear (0.05 clo), short socks (0.02 clo), and sports shoes (0.02 clo), resulting in a total clothing thermal resistance of 0.39 clo. The clothing style is consistent with that shown in the experimental graph, with no significant differences between male and female participants. Under the altered thermal resistance condition, participants additionally wear a lightweight sweatshirt (0.2 clo), increasing the total thermal resistance to 0.59 clo. Due to the need for capturing infrared images of their faces, each subject’s consent is obtained prior to the experiment. In this study, two kinds of TSVs (7-class and 3-class) are both applied according to the ASHRAE standard.

(3)

Experiment process and experiment content

The data collection experiment was conducted from July to October 2023. The entire experiment procedure consists of three stages: the preparation stage, and the first/repetition stages. At the preparation stage, the experimental environment and equipment are prepared. The subject is asked to stabilize his (her) physical and mental states outside of the test room for 20 min. In the meantime, subjects are informed about the details of experimental procedure, including TSV voting levels, experimental procedures, data collection methods (non-invasive), confidentiality of personal information, and voluntary principles that comply with ethical requirements. To mitigate potential bias arising from psychological expectations, participants are not informed beforehand of the specific environmental temperature setpoints. They are only informed that the environmental conditions during the experiment can change. Then, the subject enters the room and the first stage begins. In the first 20 min of this stage ($B_1$), the subject re-stabilizes his physical and mental states to the indoor thermal conditions. During the following 5 min ($B_2$), the subject performs the thermal sensation voting. In the meantime, his infrared images are taken, and the environmental information is recorded. After that, the repetition stage begins. The ambient temperature is decreased by 2 °C a total of six times (from around 30 °C to around 18 °C). The experiment procedure is consistent with that in the first stage. It is noted that when a thermal adaptive action is detected, the corresponding thermal vote and infrared images are collected immediately to ensure the consistency of time tags. A detailed experimental procedure is shown in Figure 9. During the adaptation stage of the subject to the environment ($B_1$ and $C_2$), the subject is allowed to engage in some chatting, reading, and other relaxing activities. All permitted activities have low metabolic rate, aiming to maintain emotional stability and to prevent the effects of physiological discomfort caused by mood swings on voting. These activities show minimal differences in metabolic level and do not significantly affect skin temperature or subjective thermal perception.

According to the above steps, three experiment contents (I–III) are carried out separately. Experiment I is conducted to collect data for the training of Model_Fac_T. The collected data include environmental temperature/humidity, skin temperature of facial key regions ($T_{nose}$ and $T_{cheek}$), and TSV votes. Among them, the average temperatures of facial regions are manually extracted through ThermoScope (developed by MAGNITY, China). A total of 168 data sets are collected according to the above experimental procedure. It is noted that data collection for the training of Model_Therm_Act is performed in advance, which is already described in Section 2.2 (2).

Based on these two trained models, experiments II and III are conducted to collect data for performance testing of the proposed fusion strategy. In experiment II, the probable thermal adaptive actions are also recorded in addition to the collected data items of experiment I. A total of 140 sets of experimental data are collected. The data details can be found in

Table 3. Detailed experiment information.

Experiment	Quantity of Data	Collected Features	Subjects (Times/Subject)	Gender Ratio
I	168	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$	12 (2)	7:5
II	140	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$, Clo, Met, Thermal adaptive actions	10 (2)	3:2
III	70	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$, Clo($\Delta$0.2 clo), Met, Thermal adaptive actions	5 (2)	3:2
	70	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$, Clo, Met($\Delta$2 met), Thermal adaptive actions	5 (2)	3:2
	70	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$, Clo($\Delta$0.2 clo), Met($\Delta$2 met), Thermal adaptive actions	5 (2)	3:2

. It is noted that in experiments I and II, subjects are required to wear typical summer clothing (with a thermal resistance of about 0.39 clo) simulating normal office conditions (1 met; met is a unit of metabolic level).

To further test the prediction ability of the proposed model in different clothing and metabolic level, experiment III is designed to form a more widely distributed testing data set. In this experiment, subjects are required to put on an extra thin coat to change thermal resistance by 0.2 clo. Subjects are required to perform 10 push-ups immediately before stages $B_2$ or $C_3$ to change the metabolic level by 2 mets. The procedure of data collection is consistent with experiment II, with a total of 210 datasets collected. Note: A total of 17 subjects are recruited for these three experiments. Subjects randomly participate in different experiments, and each subject participates in the same experiment up to two times.

3. Results

3.1. Recognition Results of Facial Regions and Temperatures

According to Section 2.1’s description, the YOLOv5 algorithm is used for facial key-region recognition. A total of 5394 infrared images are obtained by data collection and image enhancement. They are randomly divided into training, validation, and testing sets by the ratios of 7:2:1. To evaluate the performance of YOLOv5 recognition, the metrics of precision (P), recall (R), and mean average precision calculated at an IoU (Intersection over Union) threshold of 0.5 ($mAP@0.5$) are provided, which are formulated as follows:

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(5)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(6)

$$AP={\int }_0^{1}P·RdR$$

(7)

$$mAP={\displaystyle \frac{1}{N}}\sum AP$$

(8)

where $TP$ (True Positions) is the number of correctly detecting “noses and cheeks”, $FP$ (False Positions) is the number of “non-nose and non-cheeks” falsely detected as “noses and cheeks”, $FN$ (False Negatives) is the number of missed “noses and cheeks”, and N is the number of images. Main parameter settings of YOLOv5 are listed in

Table 4. Parameter settings related to YOLOv5 and Mediapipe.

YOLOv5		Mediapipe
Size	640 ∗ 640	Static_image_mode	False
Batch size	16	Smooth_landmarks	True
Epoch	300	Min_tracking_confidence	0.5
Learning rate	0.01	Model_complexity	1
Weight decay	0.0005	Min_detection_confidence	0.5

.

The recognition results of facial regions by YOLOv5 are presented in Figure 10. It is seen that the nose region’s recognition accuracy is much higher than that of cheek region (0.992 > 0.936, $mAP@0.5$), which means in infrared photos taken from multiple angles, the feature of nose region is more prominent than cheek region. The average recognition accuracies of two regions achieve higher than 0.95, which shows high feasibility of the thermal comfort assessment based on facial-region temperature detection. A typical facial detection result at nine different angles is shown in Figure 11a. Further, multiple subjects’ facial regions and temperatures could also be recognized successfully within the available ranges (0–180°). Figure 11b shows a typical recognition result of three subjects with different facial angles. It is noted that the accuracy of temperature measurement by infrared camera is related to the hardware performance and measurement distance.

3.2. Recognition Results of Thermal Adaptive Actions

According to Section 2.2’s description, a total of 65,071 sets of valid skeletal-point data are collected. They are randomly divided into training and testing sets by the ratios of 8:2. The Mediapipe tool is used to detect/extract the position information of 33 skeletal points, based on which the RF classifier is applied for thermal adaptive action recognition.

As mentioned in Section 2.4, the RF model is selected by the comparison with four other classifiers in advance, including logistic regression (LR), ridge classifier (RC), gradient boosting classifier (GB), and support vector machine (SVM). For the comparison, all parameters are set to optimal values by trail. Both Model_Fac_T and Model_Therm_Act are built using the above five data-driven models, and the classification results are shown in Figure 12. It is seen that RF always achieves the best accuracies for the two prediction tasks. The combination of Mediapipe and RF has a great recognition accuracy of over 99% for all six thermal adaptive actions. Concretely, the recognition accuracy for “wiping forehead” is the lowest at 99.7%, while the recognition accuracies for “embrace both arms” and “rubbing hands” are the highest at 100%. Typical recognition results of six thermal adaptive actions (by one subject) from three angles are recorded in Figure 13a.

Further, as mentioned in Section 2.2, the Mediapipe tool could be combined with YOLO algorithm for multi-person’s thermal adaptive action detection simultaneously. Figure 13b shows a typical detection result when two subjects perform different thermal adaptive actions at the same time. Results show that the action recognition accuracy of multi-person is consistent with that of single person.

3.3. Results of Thermal Comfort Prediction

(1)

Results of TSV prediction based on facial region temperatures

Based on the results of Section 3.1, facial key-region temperatures are recognized by YOLOv5. The research shows that the ambient air temperature has the highest correlation with TSV, followed by the cheek and nose temperature, and finally the relative humidity [45]. On this basis, the thermal comfort prediction based on facial temperatures is carried out using the RF model (i.e., Model_Fac_T). The model has four inputs, including two facial temperatures (nose and cheek), ambient air temperature, and relative humidity. A total of 168 sets of labeled data collected from experiment I are used for model training and testing (ratios of 8:2). As described in Section 2.2, the RF model achieves the best accuracy compared with the other four data-driven models (see Figure 12 for details).

Further, from another field experiment (experiment II), 140 sets of data are purely used for testing the trained Model_Fac_T. With the 140 sets of testing data, the built RF model achieves accuracies of 78.57% (7-class) and 90% (3-class). With the estimated clothing thermal resistance (0.39 clo) and metabolic rate (1 met), the calculated PMV accuracies are only 20.71% (7-class) and 65% (3-class).

(2)

Results of TSV prediction based on fusion model

According to the fusion strategy described in Section 2.3 (2), three thermal comfort predictive models based on facial temperature, thermal adaptive action, and PMV are all combined in the form of a weighted sum. According to the weighting equations in Section 2.3 (2), the value of $P_{RF}$ is set to 0.794, $P_{YOLO}$ is 0.964, and $P_{mediapipe}$ is 0.99. In 140 sets of testing data from experiment II, 38 sets of data contain thermal adaptive actions, which help increase the overall fusion model’s accuracies to 82.86% (7-class) and 94.29% (3-class). If only taking those 38 sets of data into account, the fusion accuracy of thermal comfort prediction reaches 86.8% (7-class) and 100% (3-class). This means that the recognition of thermal adaptive actions can significantly improve the accuracy of thermal comfort prediction.

Table 5. Prediction accuracies of different models under normal office conditions (summer, 0.39 clo, 1 met).

Quantity of Data	Model	Parameter	Accuracy
			7-Class	3-Class
140 (experiment II)	PMV	$T_{air}$, $RH$, Clo, Met	20.71%	65%
	Model_Fac_T	$T_{air}$, $RH$, $T_{cheek}$, $T_{nose}$	78.57%	90%
	Fusion model (total data)	$L_{Fac\_T}$, $L_{therm\_act}$, $L_{pmv}$	82.86%	94.29%
	Fusion model (actions detected)	$L_{Fac\_T}$, $L_{therm\_act}$, $L_{pmv}$	86.8%	100%

lists the details of fusion results by different models using experiment II’s 140 data sets.

(3)

Prediction results of fusion model in changed condition (clo/met)

To investigate the effects of changed clothing thermal resistance and metabolic level on thermal comfort predictions, experiment III’s data sets are used for model testing. According to Table 3, 210 sets of data contain three situations: changing the clothing thermal resistance (by 0.2 clo), changing metabolism (by 2 met), and changing both at the same time. It should be noted that the trained predictive model is based on normal conditions (experiment I/II). The robustness and generalization ability of the model are tested by different data sets from experiment III. From the prediction results, it is found that although the changing of clothing and metabolism affects the accuracy of the predictive models, the proposed fusion strategy still achieves the best accuracies under all three changing situations (changing clothing: 75.71% (7-class), 92.86% (3-class); changing metabolism: 74.29% (7-class), 92.86% (3-class); changing both: 75.7% (7-class), 94.29% (3-class)). Compared with other models, the accuracy advantage of the fusion model is more significant. Especially when metabolism is changed, the accuracy of the fusion model is better than Model_Fac_T by nearly 20 percentage points (7-class). Details can be found in

Table 6. Prediction accuracies of different models under changed condition (changing clo and met).

Quantity of Data	Model	Accuracy
		Change Clo		Change Met		Change Clo and Met
		7-Class	3-Class	7-Class	3-Class	7-Class	3-Class
210 (experiment III)	PMV	31.42%	71.4%	22.85%	58.57%	18.57%	45.71%
	Model_Fac_T	74.28%	91.42%	54.29%	81.42%	57.14%	87.14%
	Fusion model (total data)	75.71%	92.86%	74.29%	92.86%	75.7%	94.29%
	Fusion model (actions detected)	80%	100%	86.38%	100 %	80.95 %	95.24%

.

4. Discussion

Based on the above experiments and prediction results, a brief discussion is conducted on this study:

(1) Compared to the traditional PMV model, the Model_Fac_T with the addition of skin temperatures of facial key regions improves the prediction accuracy from 20.71% to 78.57% (7-class). The facial skin temperatures can well reflect the thermal sensation of the human body, and can serve as the related parameters for thermal comfort assessment. This discovery is consistent with the existing literature [43,44,45,50,51]. Further, by combining with the recognition of thermal adaptive actions, the proposed fusion strategy achieves the best prediction accuracy compared with PMV and Model_Fac_T. The parallel ensemble strategy takes more features into account that can compensate for the defects of every single method. Compared with the prediction results of the relevant literature in Table 1, the proposed model also has significant accuracy advantages.

(2) To verify the robustness and generalization performance of the proposed strategy, the changing of clothing thermal resistance and metabolism are conducted in this study. The experimental results show that when parameters (clo or met) of testing set are changed, the accuracy of predictive models degrades to some extent. This is because increasing clothing thermal resistance and metabolism causes an increase of somatosensory temperature, which will lead to different thermal comfort votes. Taking the changing of both clothing and metabolic level as an example, the accuracy of Model_Fac_T decreases from 78.57% to 57.14% (decreased by 21.43%, 7-class), and that of the fusion model only decreases from 82.86% to 75.7% (decreased by 7.16%, 7-class). From the perspective of model robustness, when the testing set’s distribution changes, the fusion model has a better performance than Model_Fac_T (decreased accuracy: 7.16% vs. 21.43%, 7-class). This indicates that the fusion model has better generalization ability, and thus has greater potential for different applications. Given the availability of computing resources, multi-person thermal comfort identification is also technically feasible. How to integrate individual data from multiple residents to create a comprehensive thermal sensation voting value is thus the key issue needing to be addressed in future research.

There are still some limitations in this study. Firstly, this article does not consider certain facial occlusion factors (such as masks, glasses, and other factors). The facial region can be effectively detected within the range of 0° to 180°. When the effective detection range is exceeded, facial temperature data collection will fail. Similarly, the thermal adaptive action detection only collects data from 45° to 135°. A larger shooting angle will cause overlapping of skeletal points, resulting in poor or failed action recognition. Secondly, there are some prerequisites for achieving prediction accuracy, and the data collection experiment of this study is mainly conducted in the summer. The proposed fusion prediction strategy may not be suitable for winter. Furthermore, our datasets are collected from young participants with normal BMIs; the predictive performance may not be the same for other groups, such as the elderly. Thirdly, the error of the sensor itself can cause the actual value to be inconsistent with the measured value. Common assumptions are applied to parameters such as wind speed. However, these may differ from actual environmental conditions and potentially affect the accuracy of the analysis.

5. Conclusions

Accurately evaluating human thermal comfort is of great significance for improving indoor environment quality and building energy efficiency. Addressing the defects of single thermal comfort prediction methods, this study proposed a multivariate fusion predictive strategy with non-invasive detecting methods. The YOLOv5 algorithm combined with an infrared thermal imager was used to extract skin temperature in facial key regions. The Mediapipe tool and random forest algorithm were used to recognize human thermal adaptive actions. The human thermal comfort level was predicted by combining results of facial temperatures and thermal adaptive action-based predictions with PMV results. From the fusion results, it was seen that the proposed fusion prediction strategy achieved the best accuracy (82.86%, 7-class) compared with single non-invasive prediction (78.57%) and PMV model (20.71%) under normal thermal conditions. When the thermal adaptive action was detected, the proposed fusion model was able to reach higher performances of 86.8% (7-class) and 100% (3-class). The parallel ensemble strategy took more features into account that could compensate for the defects of other single methods. Its robustness and generalization performance were also validated by changing clothing thermal resistance and metabolism in the testing data set. The results also showed better performance compared with the other two methods, thus demonstrating greater potential for different applications in the future. The data collection and model validation of a wider population are the focus of the next research step.