Energy Efficiency Assessment for Buildings Based on the Generative Adversarial Network Structure

Abstract

Thermal images are highly dependent on outside environmental conditions. This paper proposes a method for improving the accuracy of the measured outside temperature on buildings with different surrounding parameters, such as air humidity, external temperature, and distance to the object. A model was proposed for improving thermal image quality based on KMeans and the modified generative adversarial network (GAN) structure. It uses a set of images collected for objects exposed to different outside conditions in terms of the required weather recommendations for the measurements. This method improves the diagnosis of thermal deficiencies in buildings. Its results point to the probability that areas of heat loss match multiple infrared measurements with inconsistent contrast for the same object. The model shows that comparable accuracy and higher matching were reached. This model enables effective and accurate infrared image analysis for buildings where repeated survey output shows large discrepancies in measured surface temperatures due to material properties.

1. Introduction

Global warming has made it essential to reduce harmful gas emissions and enable economic energy consumption. Methods to facilitate energy conservation are evolving fields of study worldwide due to dwindling energy resources and climate change caused by ${\mathrm{C}\mathrm{O}}_2$ emissions. Conserving energy has become a top priority in many countries in order to achieve prudent consumption, increase resource availability, and provide improved thermal comfort. In some countries, the aim is to transform electrical energy as the main resource to provide neutral generators for the environment, as discussed in [1,2]. Apart from the emerging development of renewable energy production and projects involving communication platforms between services to ensure sufficient energy availability when needed, the construction sector remains a critical area for energy conservation. Energy conservation in this sector involves several directions, including monitoring indoor environmental conditions, improving material properties, and detecting structural defects. Carbon nanomaterials and their energy-saving capabilities and applications are presented in [3]. The thermal energy storage techniques used for thermal energy conservation and consumption are shown in [4].

Remarkable advances in sensor technologies enable continuous indoor monitoring. The research in [5] proposes the thermal infrared fusion method for adapting thermo-regulation performance that uses smart sensors and temporal indoor environmental parameters. The authors aligned thermal and visual images to localize the face and recorded temperatures for the detected coordinates; they used a Gaussian mixture model to track changes and then combined the acquired data with historical measurements to adjust system operations. The authors in [6] studied desk illuminance sensors using a Power over Ethernet lighting model. The desk sensor was checked for blocking, and the result was that the illuminance did not reach the minimum expected value, using the SVM model as the classifier for blocked and unblocked sensors. The research presented in [7] describes a smart controller design with multiple feedback devices, such as sensors and wearable devices for temperature measurement. The rotation speed of the air conditioner’s motor was adjusted to achieve human comfort and energy conservation.

Nondestructive testing is widely used to localize areas of high energy loss. The infrared technique (IRT) enables the detection of various defects in buildings, such as thermal bridges, air flow, or moisture. However, the measurement results are dependent on the environmental conditions. Numerous external factors, such as air humidity, external and internal temperature, wind speed, light, etc., influence the results of infrared measurements. The proposed methodology can be used to automate diagnostics when using infrared cameras for monitoring and can be combined with emerging IoT technologies, as discussed in [8].

Modern computer vision methods are powerful tools for image and video processing, such as object detection, tracking, and image segmentation. The GAN deep learning-based model is commonly used for image translation, synthesis, and semantic segmentation. Several authors have used GAN to detect anomalies in thermal images. In [9], cracks were investigated through nondestructive testing using an adapted GAN to improve image segmentation. The authors used eddy current pulsed thermography (ECPT), which incorporates a heating device for induction warming and records the heat distribution measured with an infrared camera. In another study [10], the authors proposed a deblur SRRGAN for thermal image reconstruction and the light-weighted Mask R-CNN for object detection. The authors generated three-channel thermal images from the original one-channel thermal images. In [11], thermal images were transformed into visual images by developing cyclic attention-based GAN for thermal to visible domains. Thermal image synthesis based on generative neural networks is presented in [12] using a multispectral image-to-image transformation algorithm. In [13], the combination of visible and thermal modalities was explored, and the authors proposed a domain fitting that does not require RGB-to-image pairing. They achieved self-training by associating thermal domains of interest with schemes for learning general representations in interesting domains. These approaches primarily focus on thermal to visual image mapping and the detection of critical points. Numerous GAN variants have been used to explore electrical energy consumption. One of the more recent applications is TimeGAN, which generates new electrical loads from the temporal series at the input. In [14], the authors described TimeGAN implementation and the method for matching original and synthesized series. Recently developed frameworks for automated design solutions in construction offer significant advantages. SD GAN is the approach proposed in [15], which was used for creating models with spatial topology in accordance with energy efficiency requirements. The input parameters involve space, light, environmental considerations, and parameters for cold air flow avoidance.

Tremendous developments in image processing techniques in recent years have facilitated the ability to provide automatic defect detections. Emerging deep learning techniques enable automatic visual data processing. Under constantly changing environmental conditions in terms of temperature, humidity, sunlight, etc., it may be challenging to calibrate visual sensors that collect infrared measurements compared with established methodologies. In building energy assessments, it is challenging to make accurate and reliable decisions on thermal estimates for specific structural areas affected by high energy losses. The research conducted in [16] presents an improved methodology for energy loss detection based on image segmentation and deep convolutional neural networks, which combines the information from thermal and visual images. In [17], the flight path for an unmanned aerial vehicle (UAV) for thermographic assessment was calculated based on the 3D model to achieve reproducible data collection. For thermal loss detection in buildings, the authors in [18] present 3D reconstruction models and visual-to-thermal mapping using the structure from motion (SfM) techniques.

In this research, repeated measurements were carried out for different buildings to detect variations in measured object temperature under different external conditions. As such, this methodology aims to improve the energy assessment for buildings by providing a comprehensive automated analysis of infrared images to make decisions about their validity.

Typically, IR measurements can be repeated on the same structure to ensure proper analysis and accurate detection of thermal loss. When thermal measurements are repeated, they may represent different thermal states for similar external conditions. The aim of this method is to manipulate IR images to predict the energy loss in buildings by selecting reference infrared images that are considered to be accurate, possibly because they were obtained in the early morning hours or at lower external temperatures and wind speeds.

2. Materials and Methods

The infrared assessments were carried out according to the recommended procedures. The data collected included emissivity, object temperature, average reflected temperature, humidity, and outdoor temperature. They were analyzed in terms of emission, reflection, and atmospheric effects. The aim was to determine how these factors affected the validity of infrared images.

The reflected energy and the influence of the atmosphere were not dominant, and there were no noticeable differences in IR measurements. However, higher reflection led to variations in the signal for the block structure and alpine structures. This finding is interesting for the method presented, which is based on the GAN scheme. The KMeans algorithm was used to differentiate high temperatures in infrared images. Various samples of these images and the presented algorithm provided relative values for the results in the input images. Detected regions with high losses in samples of the collected images were compared, and if there was a match, these images were considered reliable.

Given the fact that materials with more complex thermal behavior were displayed in multiple infrared images with modified surface temperatures (which prevents consistent thermal analysis), the method presented using the modified GAN allowed the collected data to be analyzed and predicted to determine which images represent the correct behavior.

2.1. Infrared Measures

The infrared measurements were carried out in Zürich and an area around the Swiss mountains on buildings with brick, plastic, metal, and block facades and on older alpine structures. Pictures of the same object were taken in the period from 28 February until 3 March on different days and at different times for outdoor temperatures ranging from −1 °C to 11 °C. Before each measurement, the following procedure was followed: The emissivity was adjusted when the same object was photographed on different days, the reflection effect was measured, and as additional information, the humidity and the distance to the object at the time of the measurement were noted in order to estimate the energy of the atmosphere. The reflection was determined as described in [19], and the atmospheric transmission coefficient was determined according to the methodology presented in [20]. From the obtained data for each building and each measurement, the proportion of the emitted, radiated, atmospheric, and total radiated energy was calculated. The aim of the study was to understand how these three different parameters affect thermal images, by imaging the same objects on different days with different temperatures to show the behavior of the buildings.

2.2. Adapted GAN Mechanism

CNN is commonly applied for tasks such as object detection and image segmentation. With constantly changing environmental conditions in terms of temperature, humidity, sunlight, etc., it is challenging to calibrate visual sensors that collect infrared measurements compared with established methodologies. In addition to achieving efficiency in tasks such as anomaly detection, the aim is also to obtain reliable results. The introduction of GAN has enabled more complex image processing, such as image synthesis and image translation. For example, if the IRT samples collected for a building give contradicting information about surface temperatures, applying image segmentation to the input images can lead to inconsistent decisions about the thermal behavior of the object due to different results. Thus, GAN can be applied to solve this challenge, since it is used to synthetize images, on the one hand, and provide a discriminator that compares the images and makes decisions about their similarity, on the other hand. The samples with variations in measured temperatures were overlapped with one selected image with sufficient contrast that would be used as the input image in the network with the aim to improve the accuracy in conclusions about the regions with higher heat loss. The discriminator has the capability to compare the characteristics of the synthesized image and the subsequent thermal image from the collected set.

KMeans can be used to cluster images into segments. It is one of the computationally fastest algorithms where centroids are initially defined for each cluster, and each pixel is then associated with the nearest centroid [21]. The KMeans algorithm was used to distinguish regions with higher temperature loss. In this way, all pixels that belong to the clusters were colored with their representative centroid colors, allowing for rapid differentiation of characteristic regions. By locating these areas on both input images, their pixel coordinates were algorithmically saved in the array, and the discriminator processed those arrays providing the prediction, only if the input images matched. The array that corresponded to the subsequent image was evaluated as accurate in the discriminator network, which enabled the estimation of the synthesized image.

The parts affected by heat loss are visible in bright colors, and KMeans clustering enables information to be extracted about the surfaces with high temperatures, as shown in Figure 1. The proposed method uses a system based on the GAN principle, where a generator is adopted to combine two different images of the same object, and the discriminator predicts the validity of the new image by comparing the marked regions representing higher heat loss in the generated image with high-temperature regions in an image from the set. Some materials show large variations in the measured surface temperatures with small variations in weather conditions, and therefore it is challenging to distinguish regions with higher temperatures in an object. The GAN was modified with the aim of enhancing the contrast ratio in the images and improving the localization of high-temperature areas. The contrast in cluster data could be relatively low, making it difficult to detect heat loss. For this reason, the subsequent images were combined with a selected infrared image with optimal contrast and processed at the generator output using the chosen binarized values that identified regions with heat loss on the previous image. The discriminator was trained with descriptors related to the positional parameters of the heat loss region in the image from the sequence.

The regular generative adversarial network (GAN) contains two deep neural networks. One network is the generator, which produces images based on predictions, and the other network is the discriminator, which aims to distinguish between the original and the generated images. The discriminator uses a feature extractor to distinguish domain-invariant representations on the original and the generated image. In the common GAN, the generator creates the image based on random noise and learns during training to generate images that are similar to the original. This part was adapted by using the set of images acquired from the infrared measurements for the specific object. The actual images in this study were produced by synthesizing them with an image selected as a reference for the best contrast. It was assumed that the images were roughly aligned. Figure 2 displays the network structure that predicts the accuracy of heat loss detection.

In order to achieve greater accuracy in diagnosing insulation quality due to variations in the measurements obtained, the image with optimal contrast $I_{acc}$ was combined with the current image $I_{cur}$ from the sequence, and the resulting image was then compared with the subsequent image in the set for the specific object. This logic can be expressed using the following equation:

$$\underset{i∊[1,N]}{\max }{I}_{cur}·I_{acc}-I_{seq}=\Delta I_{max}$$

(1)

where N is the number of infrared images for the same building. Using this methodology, the correction was introduced into the infrared image with noise by combining it with the corresponding optimal image, and it was expected that there would be minimal deviations in the measured signals. To achieve better accuracy, the images should be aligned. In a previous study [22], a method was proposed for image registration in which parallel, rotational, and scaling transformations were used to minimize the difference between the shifted images. The optimization was performed using the gradient descent method with the mean square error.

The first stage in the proposed method is to create a new infrared image from the set by overlapping obtained infrared images with the reference infrared image. The new input image is a rough estimate of two different infrared images from the set. Overlapping is performed using the image matting given by [23]:

$$I=0.5\ast I^{\prime }+0.5\ast I^{″}$$

(2)

where α = 0.5 indicates the equal color contribution for both infrared images $I^{\prime }$ and $I^{″}$.

The second stage is KMeans clustering, where the cluster number can be defined.

After combining the current image with the selected one, unsupervised KMeans processing was used to enable comparison with the previous infrared image. The aim was to compare specific clusters between successive images representing warmer areas.

By determining the brightness level and selecting specific colors from the palette that correspond to high-temperature ranges, the Euclidean distances are algorithmically calculated, and the high-temperature cluster is differentiated. In this modified network structure, the discriminator predicts the probability that the generated image represents accurate temperature states by examining the positional overlapping for the differentiated high-temperature segments. The discriminator introduces the entropy cost in order to calculate label differences related to regions representing thermal loss. The collected images were used to train the descriptors in the discriminator to compare new images.

In the final stage, the discriminator predicts the localization arguments for thermal loss in the object and calculates whether the detected region matches the detected region with thermal loss in the previous image. The input in the discriminator includes a two-dimensional array corresponding to the pixel coordinates in the region affected by thermal loss and, as a second input value, the prediction of the infrared image accuracy. The input array with coordinates inside the region mask can be represented as X = ${\left\{\left(x_j, y_j\right)\right\}}_{j=1}^n$, where n is the number of pixels within the registered area with higher temperatures. The discriminator makes the prediction of whether the new image has the same state and outputs the calculated probability in the range (0, 1).

The parameters in the discriminator are as follows: The first layer has 32 inputs with the activation ‘relu’ and the kernel_initializer ‘he_uniform’, the second layer contains 16 input values with the activation ‘relu’ and the kernel_initializer ‘he_uniform’, and the third layer is a dense layer with one input value and the activation ‘sigmoid’. The model was compiled using the ‘binary cross-entropy’ and the ‘Adam’ optimizer.

3. Results

3.1. IRT Measurements

Detailed IRT measurements were carried out for buildings constructed with different materials, including the adjustment of the emissivity coefficient, radiation, reflected energy estimates, and atmospheric influence. These detailed measurements were carried out with the aim of identifying variations in the measured temperature under different weather conditions for buildings constructed with diverse materials. According to the formula given in [24], the radiated energy can be calculated by taking into account detailed measured parameters. Considering the detailed measurements, the percentage of radiated energy was determined, as well as the reflection and the atmospheric effect. The real-time IRT measurements show how thermal processes on buildings differ depending on environmental conditions. It was found that buildings constructed with metal, plastic, and brick generally have higher temperatures without visible fluctuations, and the contrast in the measured signal and varying external conditions do not affect the fluctuations in the measured signals.

The detailed parameter examinations in the IRT survey allow for the detection of energy fluctuations and variations in the measured signals. The atmospheric influence could be neglected as the transmission coefficient is close to 1 for small distances between the measuring device and the object, even at relatively high humidity.

Table 1. Measured values for the radiated, reflected, and atmospheric energy for different buildings.

31 March 8 °C Humidity 76% Wind 26 km/h	Metal	Marble	Plastic	Brick	Block
Emitted energy (W/${\mathrm{m}}^2$)	385.4	387.5	382.7	356.1	340.7
Reflected energy (W/${\mathrm{m}}^2$)	3.8	3.7	3.8	19	46.8
Atmospheric energy (W/${\mathrm{m}}^2$)	4.4	4.4	4.4	4.4	4.4
Total radiated energy (W/${\mathrm{m}}^2$)	388.95	390.9	386.3	375	387.25
Percentage emitted energy (%)	99	99	99	95	88

and

Table 2. Measured values for the radiated, reflected, and atmospheric energy for metal, plastic, and brick building envelopes.

1 April 8 °C Humidity 76% Wind 26 km/h	Metal	Plastic	Brick
Emitted energy (W/${\mathrm{m}}^2$)	371	363.9	336.1
Reflected energy (W/${\mathrm{m}}^2$)	3.7	3.7	24.7
Atmospheric energy (W/${\mathrm{m}}^2$)	4.3	4.3	4.3
Total Radiated energy (W/${\mathrm{m}}^2$)	375.2	367.5	360.8
Percentage emitted energy (%)	99	99	93

show the resulting radiated, reflected, and atmospheric energy. It is observed that the emitted energy is the major part of the total radiated energy.

3.2. Heat Loss Localization

The repeated measurements show that the different external conditions do not affect the fluctuations in the measured signal for certain materials, such as brick and plastic, as shown in Figure 3 and Figure 4. The thermal scale is also shown, and the temperature range is from −20 °C to 400 °C. The temperature values in the following figures represent the measured temperatures at points that the star signifies.

Figure 5 shows three measurement states for an older alpine building and the image obtained by overlapping the low-contrast image, which is assumed to be less accurate than the referenced image. By applying the KMeans algorithm, the cluster of regions with heat loss on the previous image was determined, and the corresponding array with the value of 1 was identified based on pixel coordinates belonging to this region. Running the new KMeans-processed image through the discriminator allowed us to determine the probability that the heat loss cluster matches the cluster on the reference image. At the output, the discriminator predicted that there was a 70% probability that the images would match, indicating that the regions of heat loss were correctly identified.

The IRT measurements, taken under slightly different environmental conditions and at different times of the day show that there can be discrepancies in the measured temperatures in buildings depending on the building material. The result for the block building is shown in Figure 6. These infrared images differ in contrast, and it was assumed that the lower temperatures would give more accurate results. Under different environmental conditions, the estimated emissivity coefficient varied from 0.78 to 0.88, which is a greater variation than for other buildings. It is noticeable that different factors affect the IR image.

Due to the lower emissivity, the measured temperatures fluctuate and affect the differences in radiated energy. At very low temperatures, the measured surface temperatures were obviously lower, thus resulting in a better contrast between the elements and making it easier to distinguish them according to the heat flow. In this case, the higher the temperature, the more radiated energy was detected. As the temperature increased, it became more difficult to differentiate the elements according to their temperature, as similar values were detected for the whole object, despite sufficient differences between the internal and external temperatures. For the image taken in the evening, higher temperatures were measured, and in order to determine the regions with heat loss for this object, the variances in the measured signals were studied with the aim of enabling algorithmic diagnosis in infrared image analysis in cases when there were large discrepancies in the measurements.

Figure 6a was chosen as the reference image, and it was combined with other images and run through the discriminator with subsequent thermal imaging to make predictions about localized heat loss. It was ascertained that, for the first image examined, the heat loss regions were approximately 95% accurate. For the second image in the sequence, it was observed that the predicted accuracy was 69%, while for the third image, the minimum accuracy value was 69%. Figure 7 shows the observed results for this image sequence per iteration.

Figure 8 represents the measurement results for another object. The calculated scores for this building were constant at 0.72.

4. Discussion and Future Work Scope

The continuous monitoring and measurements for the different buildings were based on recommended procedures for infrared measurements, and according to the results, similar temperatures were measured for materials such as metal, plastic, and brick, despite outside temperature variations of up to 9 °C. Quantitative analysis included relevant environmental parameters, including distance from the measuring device to the object and humidity for atmospheric effects, as well as emissivity calibration and reflectance measurements, to provide information on how the temperature measurements obtained represent the energy distribution from the object.

However, the analyzed objects had different behaviors in terms of heat transfer, and with regard to the recommended weather conditions, different temperature states were obtained, making the energy analysis more complex. Due to the lack of brightness relationships in multiple images, it was difficult to predict the exact heat contrast. The modified GAN structure improved image contrast and reduced signal noise. The image representation is generalized from the variations in the input data. With this method, it is possible to reduce the uncertainty in false detections by combining an image selected as the most accurate in terms of contrast with the images from the image set. In this way, the contrast was improved where regions were difficult to distinguish, and congruence with the next real image was predicted. The result was a good match for the detected high-temperature regions.

Several recent studies exist in the literature that measure quantitative data and apply neural networks for thermal leak detection, and this study focused on the reliability of the data collected. In the future, larger quantities of data should be collected and analyzed. The limitations encountered in terms of material properties could be overcome with a more comprehensive study in order to distinguish specific thermal behaviors in more detail. The measured object temperatures were roughly estimated, which affects the accuracy of the results.

Newer IRT instruments allow for rapid data collection in the environment. Methodologies for reliable analysis of the collected data require processing algorithms that can predict and detect anomalies. Future frameworks should address the uncertainty in signal measurements caused by changes in position, daylight intensity, material properties, etc. This could lead to significant improvements in energy-efficient construction and environmental comfort. These goals can be achieved by applying AI mechanisms with smart visual devices designed for thermal scanning in residential areas.

5. Conclusions

In this research, the thermal behavior of buildings constructed with different materials was quantitatively measured and analyzed using thermograms. Materials such as brick and metal were found to have consistently high surface temperatures. Materials with more complex behavior such as higher energy reflection exhibited modified surface temperatures that affect consistent thermal analysis.

The proposed method involves thermal image segmentation using KMeans and a neural network that predicts the match between surfaces in the generated and original samples. A model was built to process the infrared images of the same building that were taken at different times and different external temperatures. As can be seen, the noise in the infrared images was reduced, the identification of regions with thermal loss was successful, and the contrast relationship was improved. The proposed adversarial constellation exploits the domain fitting in the input thermal images. Regions with heat loss in combined images matched with the regions undergoing heat loss in subsequent samples at a high percentage.

When there are larger discrepancies in the measured surface temperatures, the proposed method provides estimated values for sample classification. The adapted GAN with KMeans provides more accurate texture segmentation by integrating the results of multiple evaluations. This method allows for effective comparison in cases where multiple infrared measurements show discontinuities and provides the resulting probability that the detected regions of energy loss are consistent. Future experiments should optimize smart communications between IRT instruments and data processing platforms and terminals.