Accurate Automatic Object Identification Under Complex Lighting Conditions via AI Vision on Enhanced Infrared Polarization Images

Abstract

Object identification (OI) is widely used in fields like autonomous driving, security, robotics, environmental monitoring, and medical diagnostics. OI using infrared (IR) images provides high visibility in low light for all-day operation compared to visible light. However, the low contrast often causes OI failure in complex scenes with similar target and background temperatures. Therefore, there is a stringent requirement to enhance IR image contrast for accurate OI, and it is ideal to develop a fully automatic process for identifying objects in IR images under any lighting condition, especially in photon-deficient conditions. Here, we demonstrate for the first time a highly accurate automatic IR OI process based on the combination of polarization IR imaging and artificial intelligence (AI) vision (Yolov7), which can quickly identify objects with a high discrimination confidence level (DCL, up to 0.96). In addition, we demonstrate that it is possible to achieve accurate IR OI in complex environments, such as photon-deficient, foggy conditions, and opaque-covered objects with a high DCL. Finally, through training the model, we can identify any object. In this paper, we use a UAV as an example to conduct experiments, further expanding the capabilities of this method. Therefore, our method enables broad OI applications with high all-day performance.

1. Introduction

Object identification (OI) [1] is one of the promising areas where computer vision technologies can be applied, e.g., OI in static photographs, videos, or real-time scenarios, such as self-driving vehicles [2], rescue [3], biometric [4] and facial recognition [5], safety control [6], crowd counting [7] and traffic monitoring [8], inventory and warehousing [9], quality control [10], medical image analysis [11], and crop monitoring [12] and pest detection [13]. In particular, OI is a key technology for realizing the concept of autonomous cars and assisting vehicles in navigating our visual world. The technology of OI in a visible light environment has been well developed. However, several key issues need to be addressed to further improve its capabilities. For example, visible light OI cannot work properly under photon-deficient conditions, such as during nighttime. In addition, the visible light OI cannot work in foggy, rainy, or smoky environments, or an object may not be properly identified when it is hiding in a complex environment. Therefore, it is desired to develop OI technology beyond the visible wavelength range to operate under such conditions.

In comparison, infrared (IR) imaging [14,15] is a technique for measuring the temperature distribution of a scene by detecting thermal radiation differences between the object and the background. Any object with a temperature higher than absolute zero (−273.15 °C) emits thermal radiation according to Planck’s law of black body radiation [16], which is detectable by thermal cameras. As a result, IR imaging can work under any lighting conditions, particularly in low-light conditions. More importantly, weather conditions, e.g., rainy, foggy, or smoky, do not significantly affect IR imaging performance. This technology can work day and night without being significantly affected by environmental conditions. However, when the target and background temperatures are identical, distinguishing the target from the background cannot be achieved with infrared imaging alone. Moreover, with the development of optoelectronic countermeasures [17], stealth materials and stealth techniques will affect infrared detection [18].

In addition to thermal radiation, the surface of an object has an important photophysical property polarization. The polarization properties of man-made objects and natural backgrounds differ significantly; thus, polarization imaging offers unique advantages for object identification, allowing broad applications. For example, it has been demonstrated that IR polarization images enable the detection of an aircraft obscured by smoke [19] and enhance image quality, including resolution, details, and contrast [20]. Although infrared polarization imaging is relatively mature for object identification, recognition results are often evaluated by the human eye. There have been no studies addressing the role of polarization in improving OI accuracy, and no quantitative characterization of the recognition results of IR polarization imaging. To conduct a quantitative analysis of OI performance, it is necessary to incorporate machine vision [21,22,23] for a more accurate assessment. Machine vision has been widely applied in remote sensing [24], agriculture [25], transportation [26], and other fields.

Here, we demonstrate for the first time a highly accurate automatic IR object identification process based on the combination of polarization IR imaging and artificial intelligence (AI) vision (Yolov7), which can quickly identify objects (cars and unmanned aerial vehicles (UAVs)) with a high discrimination confidence level (DCL, up to 0.96). The DCL represents the degree to which a model is certain about its individual prediction results, typically expressed as a probability value. In simple terms, DCL is a mechanism that encourages the model to “tell the truth”, enabling it to provide a more honest and accurate assessment of the quality of its detected bounding boxes. This ultimately enhances the overall reliability of detection. By processing the polarization properties, including the degree of linear polarization (DOLP) and the angle of polarization (AOP), we find that the contrast can be significantly enhanced and that the background can be largely removed, thereby improving the DCL. Moreover, we demonstrate that it is possible to achieve accurate IR OI in complex environments, such as photon-deficient, foggy conditions and opaque-covered objects, thereby highlighting the unique advantage of IR OI over visible light OI. In addition, we further demonstrate the OI of UAVs by training Yolov7 on images of UAVs that are not in the database. Our work further expands the capability of IR OI, thus finding broad applications in OI with high all-day performance.

2. Design of the Automatic Infrared Object Identification System

The conceptual demonstration of our designed automatic IR OI system is shown in Figure 1, which can quickly identify objects in the scene (Figure 1a) and can handle multiple identifiable objects, using the system shown in Figure 1b. The OI system includes a high-resolution IR camera (FLIR T560 with 640 × 480 pixels) (FLIR, Wilsonville, OR, USA) with a linear polarizer (WP25M-IRC) (Thorlabs, Newton, NJ, USA) in front, which is mounted on a rotational mount to take polarization images at different angles (0°, 45°, 90°, and 135° in this study) sequentially. There are four common types of polarization imaging systems: (1) division of time polarization imaging systems (DoTP) [27,28], (2) division of amplitude polarization imaging systems (DoAmP) [29], (3) division of aperture polarization imaging systems (DoAP) [30], and (4) division of focal-plane array polarization imaging systems (DoFP) [31] (detailed information and characteristics of all the polarization imaging systems is provided in Supplementary Section S1). Here, DOTP by rotating the linear polarizer is used because of the simple setup, which can be generally applied to any commercial IR camera without requiring the modification of the structure of the IR camera. In addition, the DoTP does not require a specially designed algorithm to process the data.

The Stokes vector [32] is defined as (S₀, S₁, S₂, S₃)^T, where S₀ = I = E_0x² + E_0y² is the light intensity, and E_0x and E_0y are the electric field along x and y directions, respectively. Meanwhile, S₁ = E_0x² − E_0y² is the difference between horizontally and vertically polarized light intensities. S₂ = 2E_0xE_0ycosΦ is the difference between linear 45° and 135° light intensities, and S₃ = 2E_0xE_0ysinΦ is the difference between left-handed circularly polarized (LCP) and right-handed circularly polarized (RCP) light intensities. Φ is the phase difference between E_0x and E_0y. Since this paper focuses on linearly polarized light, S₃ is regarded as 0. The value usually normalizes these three quantities S₀, so it is between 0 and 1. In the experiment, polarization images are taken at four different angles, 0°, 45°, 90°, and 135°; then, the Stokes calculation is performed. I₁, I₂, I₃, and I₄ denote the light intensity of 0°, 90°, 45°, and 135° polarized images, respectively. So, here S₀ = I = I₁ + I₂, S₁ = Q = I₁ − I₂, and S₂ = U = I₃ − I₄.

The Stokes components calculated above can be used to solve for the angle of polarization (AOP) and the degree of polarization (DOLP). The angle of polarization (AOP) is the angle with respect to the x-axis, expressed in terms of the Stokes vector as

$$\theta ={\displaystyle \frac{1}{2}}arc\mathit{tan}{\displaystyle \frac{U}{Q}}$$

(1)

The degree of polarization (DOLP) is the fraction of the intensity attributable to the state of linearly polarized light and is expressed in terms of the Stokes vector as

$$P={\displaystyle \frac{\sqrt{\left(Q^2+U^2\right)}}{I}}$$

(2)

3. Results

3.1. Infrared Polarization Object Identification of a Single Target

We first study the IR OI using polarization images with a single tarobtain, which is a vehicle (TRAMONT 380tsi) in this case, as shown in the photo in Figure 2c. The camera emissivity is set to ε = 0.8, and the temperature detection range is set to −20 °C to 250 °C. The polarization IR images, along the angles of 0°, 45°, 90°, and 135°, are shown in Figure 2a. The windshield presents shadows of different contrasts at four different polarization angles. In the scene, the shadow is weaker in the 0° image and 90° image, and stronger in the 45° image and 135° image. This is due to the fact that the windshield is close to the Brewster angle and has a different transmittance for particular polarized states, where the reflected IR light from the windshield is stronger at 0° and 90° and weaker at 45° and 135°. In order to get accurate Stokes images, we perform scale-invariant feature transform (SIFT) [33] to align the four images. SIFT is an algorithm used for image alignment and stitching by detecting key points in an image and computing their local feature descriptions, which is schematically shown in Figure 2b. It is found that the SIFT process plays an important role in DoTP systems, since slight vibration during the imaging process causes misalignment of the images, even during a high-speed imaging process, compromising the quality of the Stokes images. The current research mainly focuses on stationary objects to demonstrate the advantages of polarization IR imaging in enhancing the OI results. We used the division of time polarization imaging systems (DoTP), and the SIFT method was used to accurately align target and background pixels simultaneously. In future work on capturing fast-moving objects, tracking techniques can be used to maintain the object at a consistent position within the frame while minimizing capture time. Furthermore, the SIFT method can be used for alignment to ensure that the fast-moving object can be clearly identified in the IR polarization images, which will be the next stage of the work.

This intensity (I) image is shown in Figure 2d, which fuses the total IR radiation intensity information of the four polarization angles, including the target and the background. In comparison, the Q and U images shown in Figure 2e,f can effectively suppress background information, such as trees and shadows on the ground. Compared to other types of polarization images, these two images highlight the profile and the details of the object (the car in the theme), due to the linear polarization properties of the man-made objects. The Q image is sensitive to the polarization properties in the horizontal and vertical directions. Meanwhile, the U image is decided by the polarization properties along the oblique direction. Therefore, the vertical surface on the door shows high brightness in the U image, and the windshield shows high brightness in the Q image. As window bumpers, etc., are made of different materials with varying smoothness, resulting in different angles of polarization, the AOP image shown in Figure 2g can be used to clearly distinguish between different materials in the theme. The metallic edges of the windows and the front grille appear brighter than other parts. However, it shows a messier background, interrupting the overall profile of the car, which may affect the OI process. The DOLP image in Figure 2h exhibits a relatively low gray value, as the degree of polarization is influenced by the surface roughness and material of the object. One can see that the smooth surface of the windshield has a relatively high degree of polarization as the light is reflected by the surface very close to the Brewster angle with minimum scattering. In comparison, the car door and other parts exhibit a relatively low degree of polarization due to surface roughness and the reflection angle.

Then, AI vision is used to identify the object in the polarization IR images. Here, a software called “You Only Look Once (Yolo)” is used, which is an OI algorithm proposed by Joseph Redmon et al. in 2015 [34]. Yolov7 [35] is the seventh version of the Yolo series. The Yolov7 OI algorithm can currently recognize more than 80 kinds of objects in daily life, such as people, bicycles, cars, motorcycles, etc. The speed and accuracy of Yolov7 have already surpassed most of the known object identifiers, and it requires low computational power and cost, which can meet the requirements of fast OI in IR images. Yolov7 comprises five parts (shown in Figure 3a): input, backbone, neck, head, and output. First, the image is fed into the backbone network after a series of pre-processing operations, such as data enhancement in the input section, and the backbone network performs feature extraction on the processed image. Subsequently, the extracted features are processed by neck’s feature fusion to obtain features of three sizes: large, medium, and small. Eventually, the fused features are fed into the head, and the results are output after OI. The output includes the identification box and the discrimination confidence level (DCL). The identification box will frame the object in the image. DCL from 0 to 1 indicates how confident the model is that the object exists in the prediction frame. A DCL close to 1 means that the model is very confident that the prediction box contains the object. If the DCL is close to 0, it means that the model thinks that the prediction box may not contain the object.

The OI results from Yolov7 are shown in Figure 3. One example is the optical photo, which is used to benchmark the DCL. The DCL of the optical photo (the last panel in the first row in Figure 3a) is 0.94, also shown in Supplement Figures S1 and S2. In comparison, the DCL of the I image is 0.95 (the last panel in the second row of Figure 3a), demonstrating the advantages of OI in IR images. The DCL of the Q and U images is up to 0.96, as seen in Figure 3b,e, due to the removal of background, confirming the capability of achieving a higher DCL by using polarization imaging. This is an improvement over the DCL of 0.95 for the I image. This means that these two polarization images are useful for object identification because they highlight the object profile. Also, compared with the highest DCL of 0.93 for the high-resolution photo of the same model vehicle on Volkswagen’s official website (Supplementary Figure S1), the DCL of 0.96 is a very high level of identification.

In comparison, although the AOP image in Figure 3d shows a lot of information, such as the roughness and the difference in materials compared to the intensity image, its DCL is not the highest, only 0.62, which could be attributed to the fact that the background information interrupts the profile of the car. The DCL of the AOP image can be significantly improved to 0.92 through image fusion and edge extraction (the detailed process is shown in Supplementary Sections S4 and S5). In comparison, the DCL of the DOLP image from Figure 3e is surprisingly high (0.93) despite the overall dark image. The reason is that the polarization of man-made and natural objects is significantly different, so the natural background is suppressed in the polarization image, leaving the remaining information primarily composed of man-made objects. In addition, we found that DOLP images achieve remarkable results in OI. So far, we have achieved accurate and automatic OI in IR polarization images via AI vision, confirming the feasibility and accuracy of OI in polarization images. The method also applies to OI in IR polarization images with a small number of pixels, to improve the DCL (Supplementary Section S6).

3.2. Infrared Polarization Object Identification of Multiple Targets

In practice, a scene typically contains multiple complex objects, with the possibility of overlap and occlusion. The method presented in this paper remains applicable to object recognition in complex scenes. The scenario selected for this work contains nine vehicles and one person, with objects overlapping. (See object labeling in multi-object images in Supplementary Section S7.) Figure 4 shows the results of OI with Yolov7 on IR polarization images, including the number and DCL of objects (including vehicles and a person). Statistically, each identification time is within 0.6 s, realizing fast and accurate OI in IR polarization images. Five vehicles (the DCL of the frontmost vehicle is 0.91) and one person (DCL is 0.89) are identified in the I image, which is shown in Figure 4b. As shown in Figure 4c,d, both the Q and U images significantly suppress the background, and both images highlight the profile of the car and the person, improving the number of identified cars and the DCL for both the vehicle and the person. As shown in Figure 4e, the AOP image contains significant polarization information, resulting in a more complex background and fewer identified cars, leading to a lower DCL than in the Q and U images. The result is consistent with the OI outcome of the single object shown in Figure 3. However, the DCL is still higher than the I image shown in Figure 4b, confirming the advantage of using polarization images in OI applications. In the DOLP image (Figure 4f), the windshields of the cars in the scene appear the whitest, indicating a high degree of polarization, which is similar to the results shown in Figure 3. The OI results of the DOLP image are similar to the Q and U images, confirming the advantages.

Additionally, we further process the polarization images to examine the effect on enhancing OI performance. We first sharpen the AOP image through a clarification process [36], enabling us to increase the number of identified vehicles to three. However, the DCLs of both vehicles and the person remain low, confirming that achieving good OI results in AOP images alone is challenging. Meanwhile, we fused the Q and U images with the optimal outcome to explore the possibility of further improving the results. One can see from Figure 4h that the fusion improves the DCL of the vehicles and the person. Finally, we applied the edge extraction technique on the I image (Figure 4b) and fused it with the sharpened AOP image (Figure 4g) to further increase the number of identified vehicles to seven. The principles of edge extraction and image fusion are described in Supplementary Notes S4 and S5. Therefore, the results demonstrate the potential to further improve OI performance in real-world scenes (containing multiple objects) by applying advanced image processing techniques to polarization images. Due to the overlapping vehicles at the edge, it is not possible to identify all nine vehicles in the scheme. The results demonstrate that the IR OI system can operate effectively in complex scenes, including those with multiple objects and overlapping features. DCL and accuracy can also be further improved by processing such as image fusion and edge extraction.

The OI results of all images are summarized in the line graph in Figure 5a. Most of the vehicle identification rates of the above methods exceed 0.5, and the DCLs are generally above 0.9, indicating the high feasibility and accuracy of OI in polarized images.

In addition, in order to further explore the relationship between the OI results and images, we also calculate and analyze the evaluation indices [37] of Figure 4b–i (See Supplementary Section S8 for original data), which is shown in Figure 5b, including gray scale mean, gray scale variance, gray scale entropy, gray scale energy, and mean gradient. The normalized values of the five evaluation indices are shown in the bar chart in Figure 5 (See Supplementary Section S9 for original data). Figure 4c,d,h has high DCL and identification rates, indicating low gray scale variance, suggesting that uniform gray scale distributions are preferred in the OI process. In comparison, when the gray scale variance is high, as in the AOP images, the identification rate is low. Therefore, it is found that DCL and accuracy are mainly affected by the gray scale variance.

4. Discussion

4.1. Performance Validation of the System in Photon-Deficient Environments

The experiments mentioned above were conducted under normal light intensity conditions to analyze different polarized images. To demonstrate the unique advantage of IR imaging under low-light (photon-deficient) conditions, Figure 6 further investigates the IR OI system’s performance in complex environments. The scene depicted in Figure 6a shows a hot cup placed in an extremely low-light environment. In a visible image, the cup cannot be seen by the eye or detected by AI vision. In comparison, the cup’s shape and its position within the scene can be extracted from infrared polarization imaging. Both the Q and DOLP images exhibit very high DCLs. As shown in Figure 6b, if a hot cup is placed in a cluttered scene, IR polarization imaging can clearly highlight it.

Furthermore, when interference factors in the scene increase, such as when objects are obscured by other objects in dark environments, IR polarization imaging can still distinguish target objects to a certain extent by leveraging its unique polarization characteristics. The scene depicted in Figure 7a shows three people standing in darkness, their heads covered with black plastic bags. At this point, in the visible light image, neither the human eye nor AI vision can effectively identify these three people. However, IR polarization imaging technology not only clearly captures the general outlines of the three people but also distinguishes the positions of their obscured heads. In addition, the profiles of the plastic bags and their heads can be clearly seen. This fully demonstrates that IR polarization imaging retains outstanding OI capabilities even in complex environments with numerous interfering factors.

Under foggy conditions, conventional visible imaging technology will be disrupted, leading to blurred or even unrecognizable objects. Fog exhibits weak scattering of IR light, and different objects exhibit varying polarization characteristics for IR radiation. Therefore, IR polarization imaging can effectively penetrate fog to capture clear outlines and the position of objects. It is necessary to create a fully controlled indoor environment to quantitatively test the effects of foggy conditions on our OI results without being affected by complex outdoor situations. More importantly, since the real-world environment is uncontrollable and unstable, it cannot be representative and may affect the accuracy of our OI results. In this work, we generate the water mist using dry ice, which shows very similar physical properties of natural fog as a scattering medium [38] (due to the Tyndall effect [39]); therefore, it is suitable for the standard testing of OI performance in IR polarization images under scattering conditions. As shown in Figure 8, dry ice mist was used to simulate dense fog. The hot cup and the person were almost completely covered by the mist. Those objects were nearly invisible in the visible image, and the AI struggled to identify them in the foggy environment, though they are still clearly visible in the IR polarization images. More importantly, the IR polarization image can show a clear profile of the objects, allowing for the distinction from the background or obstacles for robust OI applications. In this way, we demonstrate the distinct advantages of IR polarization imaging over visible imaging in OI, which is less affected by environmental lighting and other interfering factors.

In Figure 3 of the manuscript, we compared the OI results for a single vehicle target across visible images, infrared images (I images), and various polarization images. We found that the DCL in IR-only images is higher than in visible images, confirming the robustness of the performance advantages of using IR polarization images. In addition, we performed OI on visible-IR fusion images as shown in Figure 9, Figure 10 and Figure 11. It can be seen in Figure 9 that the visible-IR fusion can be useful when the object is clear in the visible images, which, however, will lower the DCL of the OI results in Figure 10 and Figure 11, when the object cannot be seen in visible images, for example, in dark or scattering (rainy/foggy) environments.

However, in the foggy environment, the CNR and SNR are lower in the IR polarization images compared to optical images, in which AI still obtains better OI results in IR polarization images. Therefore, there is no direct relation between the image metrics (CNR, SNR, and PSNR) and OI results, which may be attributed to the different key features to identify objects used by AI, for example, the shape. This part of the information has been added to the Supplementary Section S11 in the revised Supplementary Materials.

In addition, we have presented an analysis of gradient-based metrics on different polarization images and the calculation of image evaluation indexes, including gray scale mean, gray scale variance, gray scale entropy, gray scale energy, and mean gradient, in Figure 5 (See Supplementary Section S9 for original data) of the manuscript and Section S8 of the Supplementary Materials. Figure 4c,d,h has high DCL and identification rates, indicating low gray scale variance, suggesting that uniform gray scale distributions are preferred in the OI process. In comparison, when the gray scale variance is high, as in the AOP images, the identification rate is low.

4.2. Multiple Arbitrary Infrared Object Identification Enabled by Model Training

The OI can now identify 80 types of objects. (See Supplementary Section S12 for the specific categories for Yolov7.) To make the method proposed in this work more universal and facilitate the OI of arbitrary objects in IR polarization images, we used a UAV as an example and trained a Yolov7 model for identifying the UAV. We collect the 1000 UAV dataset, and the number of training iterations is set to 100 due to limited computational power (see Supplementary Section S13 for the training steps, hyperparameters, validation split, loss curves, and assessments for overfitting for Yolov7). We find that the framework model trained on visible light datasets can be applied to OI in IR images. The OI results of the I image (Figure 12b) do not show the identified UAV. In comparison, Figure 12c,d shows the Q and U images in which the entire UAV can be clearly identified, corresponding to DCLs of 0.43 and 0.57, respectively, confirming the applicability of our training method. More importantly, Figure 11e shows a partially identified UAV with a substantially high DCL of 0.7. In this case, the details in the AOP image help to improve the DCL of the QI results. However, there is no identification result for the DOLP image due to the DOLP of the UAV surface, as shown by the dark color in the image. In addition to UAVs, we can train Yolov7 with any object species for OI applications using polarization IR images.

The relatively low DCL is due to our limited computational resources, resulting in insufficient training time and iterations for the model. Additionally, our dataset of 1000 images may not be big enough to achieve a high DCL result. Moreover, the form factor and the characteristics of the drones in our own dataset differ from those in the built-in dataset in YOLOv7. However, this section verifies that it is possible to perform OI in IR polarization images of new objects through training. Although the DCL is not very high, the DCL for infrared polarization images is higher than that for infrared imaging alone, sufficiently demonstrating that IR polarization images enhance the OI performance.

5. Conclusions

In conclusion, we have experimentally demonstrated a highly accurate automatic IR object identification process based on the combination of polarization IR imaging and Yolov7, which can quickly identify objects (cars and unmanned aerial vehicles (UAVs)) with a high discrimination confidence level (DCL, up to 0.96) within 0.57 s, which is even higher than the DCL of an optical photo from the manufacturer’s website. Additionally, we demonstrate that this method can also be applied to enhance the DCL of OI in low-resolution IR images. More importantly, it is possible to achieve accurate IR OI in complex environments, such as photon-deficient (completely dark), foggy conditions and opaque-covered objects, highlighting the unique advantage of IR OI over visible light OI. More importantly, by training Yolov7 on images of a UAV not in the database, we can use this method to identify the UAV, thereby further expanding its capabilities. The training method can be applied to any object to achieve OI in IR polarization images. Combined with the unique advantages of its whole-day working capability, OI in polarization images shows great potential for real-world applications, especially under low-light conditions, such as at night and in foggy, rainy, or smoky environments. Therefore, this method can be applied across multiple fields, such as military reconnaissance, security surveillance, and autonomous driving.