Four-band Thermal Mosaicking: A New Method to Process Infrared Thermal Imagery of Urban Landscapes from UAV Flights
Round 1
Reviewer 1 Report
Review of the paper entitled “Four-band Thermal Mosaicking: A New Methdos for Process Infrared ...” by Yang, Y. And Lee, X.
The work presents an algorithm for integrating thermal images onto RGB ones taken at the same time from an UAV. RGB and thermal cameras have different spatial and radiometric resolution. The integration of the thermal image leads to the mosaicking of the whole image.
Now, I include some comments
1.- Abstract and Introduction are well done and the reader will know the objective of the work.
2.- Material and Methods
All is allright but I have some questions about the four band images
The RGB and thermal cameras have different FOVs (thermal is 25º less) in the horizontal. IN addition, the radiometric resolution is also different (16bits for thermal and 8 bits for RGB). Managing different radiometric and spatial resolutions is not easy but the best way is working in the continuum HSV or IHS domain instead of in the RGB.
The magnification without interpolation of the thermal image (Figure 3) it could be worst than magnificate and interpolate. So the option of interpolate must be taken into account. However, the best way, and I encourage you to use it, is the HSV or IHS. On the other hand, multiplying by a factor of ten the DN of the RGB bands to get a quite artificial 16 bits image seems to be the only way out.
3.- Results
The problem of misaligment is quite well treated but it would be probably easier t o manage with a slightly better pre-treatment of the thermal image (see comments above). In the same way, the linear relationship presented in Figure 10b is not so good as expected because the points out of the straight line probably corresponds to a side effect of the magnification. In addition, the results of the regression must include the uncertainty of the coefficients.
4.- Discussion
At the end of the section, authors state that the positional error is higher in actual distance (39cm) but within the thermal pixel (57cm). This could be probably reduced considering an alternative pre-treatment of the thermal image.
Good luck!
Author Response
Response to Reviewer 1 Comments Point 1: The RGB and thermal cameras have different FOVs (thermal is 25º less) in the horizontal. IN addition, the radiometric resolution is also different (16bits for thermal and 8 bits for RGB). Managing different radiometric and spatial resolutions is not easy but the best way is working in the continuum HSV or IHS domain instead of in the RGB. The magnification without interpolation of the thermal image (Figure 3) it could be worst than magnificate and interpolate. So the option of interpolate must be taken into account. However, the best way, and I encourage you to use it, is the HSV or IHS. On the other hand, multiplying by a factor of ten the DN of the RGB bands to get a quite artificial 16 bits image seems to be the only way out. Response 1: The reviewer recommends two changes: interpolating the thermal images, and processing the RGB photos in the continuous HSV domain. (1) We did not try interpolation for up-sampling the original thermal pixels to the RGB pixels because each of the original thermal pixel corresponds exactly to a 9 by 9 matrix of RGB pixels. Therefore, we only copied the DN values from the raw thermal pixels to the 81 up-sampled pixels. (2) In theory, we agree that we should process images in a continuous domain like HSV since the HSV domain separates the information of color (Hue) and intensity (Value) from the RGB composite. It is also possible that the contrast between adjacent objects becomes more significant after the conversion from RGB to HSV. However, one limitation is that the SfM workflow we are using for mosaicking can only process JPG or TIFF images with integer DN values. We tested this idea by converting the RGB photos and HSV images and then scaling the HSV values to a 16-bit format. We then processed the images three times using Pix4D with different band allocations: (1) main band-weight on Hue, (2) main band-weight on Saturation, and (3) main band-weight on Value. We found that the orthomosaic is more successful with the main band-weight allocated to Value. This HSV-based orthomosaic (new Supplementary Figure S3) is overall consistent to the original RGB-based orthomosaic. Supplementary Figure S3. Comparison between the HSV-based orthomosaic and the original RGB-based orthomosiac: (a) The HSV-based orthomosaic in HSV color. (b) The HSV-based orthomosaic converted to its corresponding RGB colormap. (c) The original RGB-based orthomosaic (Figure 4a). However, the two orthomosaics differ in detail. In many places, the HSV-based orthomosaic shows serious error in mapping the true colors (see examples in new Supplementary Figure 4). The ‘chaotic’ color distribution would lead to failed registration of the thermal band at the same places. For this reason, we have decided to use the original mosaicking method. Nevertheless, to acknowledge the reviewer’s point, we have added these results to the online supplement (Supplementary Figures S3 and S4) the Discussion Section: “In theory, the UAV images can also be processed in a continuous domain like HSV (hue-saturation-value) since the HSV domain separates the information of color (Hue) and intensity (Value) from the RGB composite. It is also possible that the contrast between adjacent objects becomes more significant after the conversion from RGB to HSV. We tested this idea by converting the RGB photos and HSV images and then scaling the HSV values to a 16-bit format. We them processed the images three times using Pix4D with different band allocations: (1) main band-weight on Hue, (2) main band-weight on Saturation, and (3) main band-weight on Value. We found that the orthomosaic is more successful with the main band-weight allocated to Value. This HSV-based orthomosaic is overall consistent to the original RGB-based orthomosaic (Supplementary Figure S3). However, the two orthomosaics differ in detail. In many places, the HSV-based orthomosaic shows serious error in mapping the true colors (Supplementray Figure S4). The ‘chaotic’ color distribution would lead to failed registration of the thermal band at the same places. For this reason, we have decided to use the original RGB-mosaicking method.” (Line 413-425) Supplementary Figure S4. Detailed comparison between the HSV-based and the original RGB-based orthomosiac. (a) and (c) are two enlarged views from the RGB orthomosaic, and (b) and (d) are their corresponding views from the HSV orthomosaic. The locations of these two views are marked in (e). Point 2: The problem of misaligment is quite well treated but it would be probably easier to manage with a slightly better pre-treatment of the thermal image (see comments above). In the same way, the linear relationship presented in Figure 10b is not so good as expected because the points out of the straight line probably corresponds to a side effect of the magnification. In addition, the results of the regression must include the uncertainty of the coefficients. Response 2: As for the pre-treatment, we have shown in the Point 1 that the conversion from RGB to HSV colormap might be not suitable for our FTM method. Figure 10b (now the 7b) is the relationship between the temperature values collected by the thermal camera and the IR thermometer. For the purpose of calibration, the 40 thermal images here were collected separately and no magnification was done to them. Instead, the central matrix with the size of 20 by 20 pixels in each thermal image is extracted and averaged to get the representative DN value of the object being measured by the IR thermometer at the same time (Line 343-352). The required uncertainty of coefficients in the regression is added to the figure, along with other statistics. Point 3: At the end of the section, authors state that the positional error is higher in actual distance (39cm) but within the thermal pixel (57cm). This could be probably reduced considering an alternative pre-treatment of the thermal image. Response 3: Thank you for your comments. We hope that the reviewer agrees that having a positional accuracy better than one pixel is an encouraging result. The original coarse thermal pixel size (57 cm) is a fundamental physical limitation and can only be overcome with improved thermal cameras. Please also refer to our response to Point 1.Author Response File: 
Author Response.docx
Reviewer 2 Report
Authors investigate the Four-band thermal mosaicking: A new method to process infrared thermal imagery of urban landscapes from UAV flights. The results are quite novel since they show a significant contribution. However, the manuscript needs a major revision. The details comments as below mention:
1. The introduction part needs to revise. The authors need to better justify the methods described. The problem statement should be the focus as per the title of MS
Literature review part is needed to be elaborated more;
Need to be elaborating the Atmospheric corrections (MODTRAN) connect to photogrammetric methods to solve the thermal path length.
The circumstances of the field survey need to be specified in detail.
The methodology section should be completed with some relevant information and a more complex discussion would be needed.
A more complex statistical processing would be necessary.
The discussion needs to be rewritten, authors should take into consideration to evaluate their results in terms of the topic-related studies
Discuss the performance and quality metrics used for evaluating mosaicked images from UAVs
Conclusion part needs to be revised, recommendations for future studies are not significant as per the findng of this study, so rewrite the future works
10. Rechcek the cited references and also references list is not formatted according to journal criteria.
Author Response
Response to Reviewer 2 Comments Point 1: The introduction part needs to revise. The authors need to better justify the methods described. The problem statement should be the focus as per the title of MS. Response 1: We have modified the text here to better map the introduction to the title: “So far, we lack efficient procedure for processing thermal images. Often thermal images are used as individual frames and without clear characterization of positional accuracy. Mosaicking UAV thermal infrared (TIR) imagery has usually been not as successful as RGB and multispectral imagery for the following reasons…” (Line 73-76) Point 2: Literature review part is needed to be elaborated more. Response 2: The literature review has been enhanced. (Line 36-46; 69-72) Point 3: Need to be elaborating the Atmospheric corrections (MODTRAN) connect to photogrammetric methods to solve the thermal path length. Response 3: The option of using MODTRAN for calibration is now discussed “Although not helpful to positional accuracy, correction for atmospheric interference may bring improvement on surface temperature estimation. MODTRAN (radiative transfer modeling for atmospheric correction) is regarded as a reliable model to correct the impact of atmospheric absorption of radiation [43]. Berni et al. [44] applied MODTRAN to their UAV photogrammetry and found that the error due to atmospheric absorption is dependent on surface temperature and is lower for cooler objects. At the maximum temperature value of 15 °C in our landscape, we infer from their data that the error in our study should be no more than 1 °C. The actual error should be lower than this because our UAV flight was carried out in late fall at a lower atmospheric moisture and at a lower flight height than theirs.” (Line: 437-445) Point 4: The circumstances of the field survey need to be specified in detail. Response 4: More information about the weather conditions and the atmospheric correction method was added to the description of the UAV flight and the ground measurement. “The flight route over Beaver Pond Park was set up using DJI GS PRO, the DJI App for mission planning. The flight mission was carried out at about 10:00 AM on October 31, 2018, a clear day. Air temperature and relative humidity at the time of the flight was about 8 °C and 60%, respectively. These atmospheric parameters and the sky condition werecan be set to the camera prior to the flight in order to correct the effect of atmospheric absorption. The flying height was 85 m and the whole flight took about 10 minutes. Four swaths of consecutive photos were taken in the NW-SE direction in parallel to the long dimension of the rectangular target area shown in Figure 1, with the front and side overlaps of 85% and 75% for RGB photos and of 85% and 50% for thermal photos, respectively. A total of 440 pairs of RGB and thermal photos were selected after manually filtering out some pairs with blurry RGB photos.” (Line 148-157) “The measurements were carried out at 14:00 PM on January 9, 2019, on 40 ground objects. The air temperature and the sky condition were close to the circumstance of the UAV flight. However, the relative humidity was much lower (~40%), requiring a proper adjustment of atmospheric parameters of the camera to solve the atmospheric absorption.” (Line 346-349) Point 5: The methodology section should be completed with some relevant information and a more complex discussion would be needed. Response 5: We have added the information on weather condition and atmospheric correction to the Method section (Point 4). The discussion section has also been enhanced (please refer to response to Review 1, point 1) Point 6: A more complex statistical processing would be necessary. Response 6: Thank you for your suggestion. We do feel that cluster analysis, histograms and simple statistical measures such as Root Mean Squared Error (Table 4) are adequate for our study objective. Nevertheless, we have re-processed the data on radiometric calibration and added more statistics for the linear regression (Figure 7b). Point 7: The discussion needs to be rewritten, authors should take into consideration to evaluate their results in terms of the topic-related studies. Response 7: Our results are now discussed more now by considering some topic-related studies. Special attention is paid to the quality evaluation of the temperature map (Line: 429-431). Point 8: Discuss the performance and quality metrics used for evaluating mosaicked images from UAVs. Response 8: The metric used for evaluating performance is RMS shown in Table 4. “The error analysis is based on the ground features found in the final orthomosaics. The overall performance is essentially limited by the resolution of the thermal lens: features smaller than one original thermal pixel are not identifiable. The primary goal of the object-based calibration in this work is just to reduce the positional error to less than the original thermal pixel size (57 cm), which we have achieved. Further improvement to the performance requires use of higher-resolution thermal cameras with a smaller raw pixel size.” (Line 407 - 412) Point 9: Conclusion part needs to be revised, recommendations for future studies are not significant as per the finding of this study, so rewrite the future works. Response 9: In response, we have added a short paragraph here: “The FTM workflow opens opportunities for low-cost but high-resolution UAV observation on the thermal environment. It has scientific merits in many fields of application, such as wildlife surveys [45,46], wildfire detections [1,2], and urban climate studies [47], all of which require detailed thermal data. Future work is needed to test the method with different camera configurations. With the help of higher-resolution thermal cameras, further reduction of inter-band positional error is possible. Additional improvements may also result from more rigorous correction for atmospheric interference and use of ground targets of known temperatures to generate the actual temperature map.” (Line 459 - 466) Point 10: Rechcek the cited references and also references list is not formatted according to journal criteria. Response 10: Re-checked.Author Response File: Author Response.docx
Reviewer 3 Report
The work presents a novel method, called Four-band Thermal Mosaicking (FTM), to process thermal images acquired by drone platforms by stacking the thermal band onto the RGB bands acquired on the same flight, and mosaicking the four bands simultaneously.
The paper is well written and presented, although some figures need improvements (e.g. Fig. 3, 4 and 5), with special regards to the workflow (Fig. 3): a lot of tools and software were used in the post-processing, such as Matlab, EXIFTOOL, Pix4D, ESRI ArcMap, therefore my suggestion is to expand figure 3 workflow by including the post processing phase and showing each step and the correspondent tool/software to obtain the final result.
The introduction section needs improvements, in particular regarding lines 32-38. The literature background should be expanded in the following fields:
1) UAV environmental applications and image acquisitions (e.g. geomorphological analysis such as landslide mapping and slope 3D modelling);
2) pros and cons of UAV (e.g. weather conditions, sensor weight and resolution vs flight time/distance, see Colomina, I., & Molina, P. (2014). Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS Journal of photogrammetry and remote sensing, 92, 79-97);
3) on 3D SfM (e.g. Niethammer, U., James, M. R., Rothmund, S., Travelletti, J., & Joswig, M. (2012). UAV-based remote sensing of the Super-Sauze landslide: Evaluation and results. Engineering Geology, 128, 2-11; Eisenbeiß, H. (2009). UAV photogrammetry (Doctoral dissertation, ETH Zurich).
The thermal section in particular needs improvements:
1) Lines 59-61, with regards to literature review on mapping for scientific applications (e.g. geology and geomorphology);
2) Lines 192-198: a brief introduction on the theoretical background of thermal data is needed, since thermal principles are abruptly introduced. A brief section (e.g. Thermal data theoretical background), could be added in chapter 2, following the overview on FTM.
The first chain of the pre- and post- processing is sound and well performed, but the thermal calibration section looks to me the weakest part of the paper: thermal cameras have complex radiometric calibration procedures to convert each pixel DN to a surface temperature value. You are doing this with measurements made with a radiometer that works in a different IR wavelenght with respect to the FLIR DUO sensor (is that correct?), and acquired at a very short distance, avoiding even the slight atmospheric disturbance that a UAV has by acquiring at a 80 m distance; Furthermore, you state that the measurements were made simultaneously, but the UAV flight was carried out on October 31th 2018, while the radiometer measurements on January 9th 2019: wouldn’t all these points affect a proper thermal calibration? Please explain
This way the obtained temperature map in the supplementary material (why not incorporating it in the manuscript, since it’s the final product of all the work?) is not displaying the expected data referring to the UAV flight, is that correct? Why not use the FLIR DUO camera calibration algorithms? Please explain.
In this perspective the whole cluster analysis, although interesting, looks to me a bit speculative, and on top of this, the whole work, if not providing consistent thermal mapping for environmental purposes seems an elegant novel excercise of a UAV image processing chain, more useful from a technical point of view with respect to a scientific one. Please clarify your thermal calibration procedure.
Specific comments are given in the attached file
Comments for author File: 
Comments.pdf
Author Response
Response to Reviewer 3 Comments Point 1: The paper is well written and presented, although some figures need improvements (e.g. Fig. 3, 4 and 5), with special regards to the workflow (Fig. 3): a lot of tools and software were used in the post-processing, such as Matlab, EXIFTOOL, Pix4D, ESRI ArcMap, therefore my suggestion is to expand figure 3 workflow by including the post processing phase and showing each step and the correspondent tool/software to obtain the final result. Response 1: The figure 3 workflow is expanded now to include the mosaicking and the post-processing procedures. Point 2: The introduction section needs improvements, in particular regarding lines 32-38. The literature background should be expanded in the following fields: 1) UAV environmental applications and image acquisitions (e.g. geomorphological analysis such as landslide mapping and slope 3D modelling); 2) pros and cons of UAV (e.g. weather conditions, sensor weight and resolution vs flight time/distance, see Colomina, I., & Molina, P. (2014). Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS Journal of photogrammetry and remote sensing, 92, 79-97); 3) on 3D SfM (e.g. Niethammer, U., James, M. R., Rothmund, S., Travelletti, J., & Joswig, M. (2012). UAV-based remote sensing of the Super-Sauze landslide: Evaluation and results. Engineering Geology, 128, 2-11; Eisenbeiß, H. (2009). UAV photogrammetry (Doctoral dissertation, ETH Zurich). Response 2: The introduction has been revised “In spite of some limitations related to weather tolerance, maximum payload, civil aviation regulation, and traveling distance [13], UAVs can collect high temporal resolution data at ultra-high spatial resolutions [14], and are versatile due to its ability to change speed, direction and elevation as required [15]. Benefited from the advantages of the UAV platform, UAV photogrammetry and remote sensing have opened new opportunities for landscape visualization and analysis. Lucieer et al. mapped the landslide displacement using high-resolution and multi-temporal UAV photos [16]. Techniques of 3-D modelling, like Structure-from-Motion (SfM) [17], was combined to process the acquired UAV photos and to construct the Digital Elevation Model (DEM). The landscape of urban area can be investigated in detail with the acquisition of multi-band UAV imagery [18], which can yield more valuable information than satellite imagery. (Line 36-46) “For example, thermal maps can be combined with geomorphologic data to monitor potential volcanic unrest in a 3-D view [30], or used to reconstruct the thermal property of plant canopies [31]. Other applications include assisting model calculation of surface energy balance [32] and surveying land use and local climate [33-35]” (Line 69-72) Point 3: The thermal section in particular needs improvements: 1) Lines 59-61, with regards to literature review on mapping for scientific applications (e.g. geology and geomorphology); 2) Lines 192-198: a brief introduction on the theoretical background of thermal data is needed, since thermal principles are abruptly introduced. A brief section (e.g. Thermal data theoretical background), could be added in chapter 2, following the overview on FTM. Response 3: We have added the following text in Section 2: “The pixel DN value of the thermal band is proportional to the flux of infrared radiation received by the thermal lens. The DN value can be converted to temperature on the principle of the Stefan-Boltzmann Law [42].” (Line 202-204) Please also refer to Point 2 above. Point 4: The first chain of the pre- and post- processing is sound and well performed, but the thermal calibration section looks to me the weakest part of the paper: thermal cameras have complex radiometric calibration procedures to convert each pixel DN to a surface temperature value. You are doing this with measurements made with a radiometer that works in a different IR wavelenght with respect to the FLIR DUO sensor (is that correct?), and acquired at a very short distance, avoiding even the slight atmospheric disturbance that a UAV has by acquiring at a 80 m distance; Furthermore, you state that the measurements were made simultaneously, but the UAV flight was carried out on October 31th 2018, while the radiometer measurements on January 9th 2019: wouldn’t all these points affect a proper thermal calibration? Please explain Response 4: These are all excellent points. We agree that the temperature map may not be highly accurate, but it still revealed the thermal contrasts between different objects within the scene, helping to validate our FTM method. The Flir camera has some capacity to correct atmospheric interference. Prior to acquiring the images, we set the distance of measurement and the atmospheric parameters, including air temperature, air humidity, and weather condition via the Flir APP for the UAV flight. This information is now added to the paper: “Air temperature and relative humidity at the time of the flight were about 8 °C and 60%, respectively. These atmospheric parameters and the sky condition were set to the camera prior to the flight in order to correct the effect of atmospheric absorption.” (Line 150 - 152) “…, the relative humidity was much lower (~40%), requiring a proper adjustment of atmospheric parameters of the camera to solve the atmospheric absorption.” (Line 346 - 349) Additional correction, such as that based on MODTRAN, is unlikely to bring further improvement to the temperature estimation, as explained in the revision as follows: “Berni et al. [44] applied MODTRAN to their UAV photogrammetry and found that the error due to atmospheric absorption is dependent on surface temperature and is lower for cooler objects. Using the maximum temperature value of 15 °C in our landscape, we infer from their data that the error in our study should be no more than 1 °C. The actual error should be lower than this because our UAV flight was carried out in late fall at a lower atmospheric moisture and at a lower flight height than theirs.” (Line 440 - 445) The IR thermometer and the thermal camera are working in similar IR wavelengths: 8 – 14 μm for the IR thermometer and 7.5 – 13.5 for the thermal camera. This information is now added to the paper. (Line 344 - 345) Point 5: This way the obtained temperature map in the supplementary material (why not incorporating it in the manuscript, since it’s the final product of all the work?) is not displaying the expected data referring to the UAV flight, is that correct? Why not use the FLIR DUO camera calibration algorithms? Please explain. Response 5: The temperature map was not essential for our primary goal, which is to develop a thermal mosaicking method. For this reason, we decided to include it in the supplementary document. We did not use the camera calibration algorithm because the resulting temperature is not reliable. An example is given below. Here the temperature from the Flir’s algorithm was biased low by as much as 10 oC. (a) (b) Figure 1. (a) The laboratory scene for which a thermal image was taken. (b) The relationship between the temperature values given by Flir’s algorithm and the infrared thermometer. Point 6: In this perspective the whole cluster analysis, although interesting, looks to me a bit speculative, and on top of this, the whole work, if not providing consistent thermal mapping for environmental purposes seems an elegant novel excercise of a UAV image processing chain, more useful from a technical point of view with respect to a scientific one. Please clarify your thermal calibration procedure. Response 6: More details have been added to the thermal calibration procedure “The temperature conversion formula was obtained with measurements made simultaneously with an infrared thermometer (Figure 7a; working wavelengths 8 – 14 μm) and the thermal camera (Figure 2b; working wavelengths 7.5 – 13.5 μm). The emissivity of the infrared therometer was set to 0.95. The measurements were carried out at 14:00 PM on January 9, 2019, on 40 ground objects. The air temperature and the sky condition were close to the circumstance of the UAV flight. However, the relative humidity was much lower (~40%), requiring a proper adjustment of atmospheric parameters of the camera to solve the atmospheric absorption. For each thermal image, a matrix of 20 by 20 pixels centered on the object was used to obtain the mean DN value of the object. Linear regression was done between the sampled temperature values and the corresponding DN values (Figure 7b). The regression equation was used to convert the thermal orthomosaic to the temperature map (Supplemetary Figure S1).” (Line 343 - 352)Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
Dear authors,
You have made a good work. Congratulations.
Reviewer 2 Report
Comments and suggestions are incorporated. Ready to publish
Reviewer 3 Report
Dear Authors,
thank you for addressing all of my comments. I hope they have helped to improve the quality of the work. I think your work can give an interesting contribution in the field of UAV thermal remote sensing.
Alle the best
