Remote Sensing Mapping Infrared Data on Terrestrial Laser Scanning 3d Models of Buildings

A new 3D acquisition and processing procedure to map RGB, thermal IR and near infrared images (NIR) on a detailed 3D model of a building is presented. The combination and fusion of different data sources allows the generation of 3D thermal data useful for different purposes such as localization, visualization, and analysis of anomalies in contemporary architecture. The classic approach, which is currently used to map IR images on 3D models, is based on the direct registration of each single image by using space resection or homography. This approach is largely time consuming and in many cases suffers from poor object texture. To overcome these drawbacks, a " bi-camera " system coupling a thermal IR camera to a RGB camera has been setup. The second sensor is used to orient the " bi-camera " through a photogrammetric network also including free-handled camera stations to strengthen the block geometry. In many cases the bundle adjustment can be executed through a procedure for automatic extraction of tie points. Terrestrial laser scanning is adopted to retrieve the 3D model building. The integration of a low-cost NIR camera accumulates further radiometric information on the final 3D model. The use of such a sensor has not been exploited until now to assess the conservation state of buildings. Here some interesting findings from this kind of analysis are reported. The paper shows the methodology and its experimental application to a couple of buildings in the main Campus of Politecnico di Milano University, where IR thermography has previously been carried out for conservation and maintenance purposes.


Infrared Imaging Techniques for the Analysis of Buildings
The use of infrared (IR) sensors is today a fundamental tool in many close-range and terrestrial applications.Thermal infrared cameras operate in the bandwidth 3.5 µm < λ < 14 µm (Long Wave IR) and allows one to visualize thermal differences on the surface of an object.In [1] and [2] a review of these sensors is reported, along with several applications including diagnostics of building heat insulations, power line and pipeline monitoring, livestock monitoring, animal studies, sport sciences, detection of archeological remains, security and medicine.Near infrared cameras (NIR) are sensitive to the wavelengths in the range 0.75 µm < λ < 1.4 µm and are usually used for the analysis of vegetation [3].Terrestrial applications might also exploit findings and achievements of IR sensors that have been used for many years in satellite [4] and airborne applications [5,6].
In this paper the focus is concentrated on the use of IR sensors for applications concerning the analysis of performances and state of health of buildings.Thermal scanning of a structure allows one to collect information regarding technological elements, shape, physical characteristics of materials, and state of decay.Different kinds of defects affecting building structures can be detected by the analysis of surface temperature, submitted at particular boundary conditions.
As infrared thermography (IRT) is mostly used as a preliminary investigation tool, a direct survey of the shape, materials and their damages is necessary to know the real state of the surface to analyze.Moreover, planning the acquisition of IR images suffers from the approximation depending on the accuracy of the preliminary reconnaissance.For instance, the heating time may vary in different portions of the building, depending on changes in material or exposition of the structure.The integration of 3D metric models and IRT seems a great improvement of this investigation technique, because it can overcome the lack of reliable surveys and assessments, joining both steps in one and reducing time and inspection costs.In the case of ancient buildings, the walls to investigate are usually irregular.Indeed, thickness, structure and number of layers may be different even in the same part of the construction.Therefore, the metric location of thermal anomalies that should be further investigated with destructive tests is crucial for reducing as much as possible the size and number of samples needed.
The inspection of building envelopes by IRT is based on the effects of heat flow across the structure.The surface temperature is a function of heat flow crossing the wall and local boundary conditions.This parameter may give information regarding the interior layers of the structure.The heat is transferred more quickly throughout the most cohesive materials and/or materials with greater thermal effusivity.Differences in surface temperature due to different thermal properties of elements such as timber, bricks, stone, and mortar can be visualized at proper time as a "footprint" of their shapes projected on the overlapping plaster.Any thin delamination of the coating and detachment of the finishing layer strongly reduces the heat transfer and adds its signal to that given by the structure.For example, in Section 6 the authors will present two case studies of inspections on two different buildings, both belonging to the Italian preservation register record.One is the historical headquarter of Politecnico di Milano, the Rector's office, dating back to 1927.The second is a famous building in the university campus, the "Trifoglio" building, which was designed by Gio Ponti in 1961.

Integration of IR Images and 3D Models of Buildings
Most IR sensors adopted in close-range applications are capable of capturing small rectangular images.If the user is interested in investigating pathologies localized in tiny areas, the analysis of each single image can provide sufficient information (see e.g., [7]).The only required processing step is a preliminary image equalization in order to concentrate attention on the range of temperatures that have been effectively revealed on the surface.Due to the simplicity of this task and the reduced cost of several up-to-date thermal cameras with low geometric resolutions, IRT is becoming largely popular among professionals working in building maintenance.
On the other hand, in the case of large constructions or when the temperature information has to be related to the three-dimensional structure of a building, the independent analysis of single images is not enough.Images have to be mapped and mosaicked on the surfaces to be analyzed.This operation is usually called photo-texturing (or simply texturing) and requires a 3D model of the object [8].Such a model can be derived by existing drawings or can be obtained from photogrammetric or terrestrial laser scanning (TLS) surveys, as illustrated in Section 3. In the case of small and flat surfaces, mapping an image is reduced to the computation of a homography.This requires the identification of at least four corresponding points on both IR image and surface, operation that could be difficult in the case of homogeneous walls.To overcome this problem, an image in the visible wavelength (RGB) can be mapped first on the model.Mapping an RGB image is a simple task when operating with either photogrammetry or TLS (see Sections 3.1 and 3.2).A further remarkable task to improve the final quality of the IR images used for texturing is the geometric calibration of the adopted IR thermocamera.
A more comprehensive approach to map images on 3D models is based on the use of 3D analytical relationships which approximate the process of image formation.Collinearity equations [9] are used in photogrammetry to describe the connection of a 3D point (X,Y,Z) in the object space to its image point (x,y) in a 2D photograph.These equations incorporate the observed coordinates of each point in the image and three groups of parameters: inner orientation (IO) parameters (principal distance c and principal point coordinates x p and y p ), exterior orientation parameters (EO), including rotation angles and coordinates of perspective center, and object coordinates of the observed point.Furthermore, a set of additional parameters (APs), which describes lens distortion according to a mathematical model [10], can be included into the collinearity equations.Usually IO and APs are computed in a preliminary stage called camera calibration.Their values will remain constant for all the images captured by the same camera, at least for the period of the measurement campaign and in absence of mechanical or thermal shocks.Alternatively, the perspective model of collinearity equations can be replaced by the projective model provided by Direct Linear Transformation (DLT, [11]).Here the relationship between 2D and 3D coordinates is described by means of 11 algebraic coefficients without an immediate physical meaning.In both cases, model parameters are evaluated by a procedure referred to as image orientation.Usually, the most popular procedure to accomplish this task in the case of thermal imagery relies on the orientation of each single image independently from the others.This procedure, generally defined space resection, is based on the measurement of a few control points as in the case of homographic transformations.The number of points to measure depends on the adopted model (6 for DLT, 3 for collinearity), on the method adopted for the linearization of equations, and on the desired redundancy of observations.In practice, this number usually ranges from 5 to 10 points per image.Also in this case, a preliminary texturing with RGB images could be useful to simplify and speed up the IR image orientation.Once an image has been oriented in the same reference system of the surface model, each 3D point of a point cloud or each portion of a polygonal model can be textured (see Section 4).This method is today viable in many photogrammetric and point cloud processing software packages.Unfortunately, it suffers from three main drawbacks: (i) it is time-consuming because each image has to be processed individually; (ii) some IR images might have poor texture and control points cannot be found; (iii) mosaicing of IR images on the 3D model is prone to show discontinuities in areas where two overlapping images are textured.
In the presented paper, an alternative procedure to carry out this task is proposed.A geometric 3D model of the structure under investigation is obtained by using TLS (see Section 3.2), if this is not already available.Photogrammetry can also be used in some cases as described in Section 3.1.Both techniques allow one to obtain a vector representation of the model (polygonal model), whose facets can be depicted by the image content.Another simple representation is directly given by texturing the point cloud obtained from laser scanning or from image matching if photogrammetry has been used.In the case of TLS, such photo-texturing can be obtained in a straight-forward way only if the instrument can integrate an RGB camera.Alternatively, an NIR camera can replace the RGB one to obtain NIR texture of the object.The use of this information for building investigations is not yet popular.However, as demonstrated by the example in Subsection 6.2, the interpretation of NIR images could provide further information, different from that obtained from RGB or IR textures.In the proposed data acquisition system, IR images are captured by a thermal camera which is mounted on one side of a short steel rod.A high resolution RGB camera is positioned on the other side.After the calibration and relative orientation of both sensors, the resulting "bi-camera" system can provide two outcomes: (1) the IR coverage of the building; (2) the EO of all images captured by the RGB camera, which is obtained through a bundle adjustment.Here some additional images can be integrated to strengthen the photogrammetric block geometry.The knowledge of relative orientation between IR and RGB cameras allows one to compute the EO of both sensors.Finally, the EO of IR-RGB "bi-camera" system is computed with respect to the reference system of the object.This solution allows one to overcome, at the same time, almost all drawbacks listed for traditional single-image registration methods, as described in Section 5 and witnessed by the examples reported in Section 6.The level of automation of this procedure depends on the texture of the target object.Indeed, the only manual task is due to the orientation of RGB images, which can only be carried out by an automatic procedure when they consist of a rich texture without repetitive patterns [12].However, no measurements between the RGB and IR images are required, except during the calibration of the "bi-camera" system.

Thermal Imagery
Infrared thermography (IRT) is a non-destructive and non-contact technique based on the measurement of the heat energy and its conversion into an electrical signal which is turned into a thermal digital image by a microprocessor.Luhmann et al. [13] gives a synthetic but comprehensive review of IR sensor technology.As is well known from Wien's displacement law [14], the maximum emitted electro-magnetic wavelength (λ max ) of an object is inversely proportional to its absolute temperature (T): (1) This means that the higher the temperature, the shorter the maximum wavelength emitted.In the field of building analysis, the detection of temperatures in the range between −20 °C and 100 °C is required, corresponding to emitted maximum wavelengths ranging from 7.7 µm to 11.4 µm.As a result, the sensors to be adopted must be able to work in the Long Wave IR spectrum.A second problem concerning the sensor technology is related to the minimum size of the sensor unit, depending upon the diameter (d) of the diffraction disk: where λ is the wavelength of the recorded signal and k = f/D (called "F" or "stop" number) is the ratio between focal length (f) and lens aperture (diameter of the pupil) (D).According to the shorter value of λ implied in the IR spectrum with respect to the visible one, resulting diffraction diameters will be much larger.As a consequence, while SLR (Single-Lens Reflex) cameras can be equipped with CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensors having a pixel size of a few micrometers, thermal cameras range between 30 and 50 µm.
Two different kinds of sensors are used in IR thermocameras.The most largely adopted technology is based on thermal detectors, which feature a sensitivity in the order of ±0.1 K. Currently, cameras with sensor size inferior to 320 × 240 pixels con be purchased for a few thousand euros and are used for analyses in building maintenance practice.Larger sensors are also available (up to a size of 1,280 × 960 pixels), although at an absolutely different cost.If a higher thermal sensitivity or a faster acquisition speed is desired, cameras equipped by quantum detectors can be adopted.In this case, an internal cooling system is needed, which makes the device cumbersome and difficult to operate in some environments.
Thermocameras are based on a solid state sensor technology and can be handled as standard RGB cameras in photogrammetric applications.The pinhole camera model can be used, with lens distortion calibrated by standard photogrammetric procedures [2,9].In the experiments reported in this paper (Section 6), a standard photogrammetric procedure for camera calibration has been used.This is based on a calibration framework made up of 38 control points (CPs) fixed on a wooden structure (see Figure 1(a)).A set of iron nails have been chosen as CPs, whose 3D coordinates have been accurately measured by using a theodolite from multiple stations.The precision of CPs is then much higher than the geometric resolution of the IR images.Indeed, the simple sun-lighting is sufficient to heat each CP, which assumes a temperature higher than background and becomes distinguishable in IR data.All the The efficiency of IRT as a non-destructive technique is well documented in many fields of engineering to support restoration or conservation projects and treatments.In civil engineering and architecture, IRT can be successfully used as an alternative to conventional inspection technologies, especially for the detection of subsurface defects and hidden structures in wide areas (see the examples in Section 6).Moreover, IRT is often complemented by other non-destructive techniques such as GPR or sonic measurements.
Nowadays the guidelines ( [1]; Chapter 18) of testing procedures recommend the repetition of thermographic recapture at different conditions of thermal exchange.In addition, comparisons of temperatures of selected areas in the same framing at the same boundary conditions are suggested.This method allows the operator to reduce systematic errors due to ambient irradiation variations and fluctuations.This detection is successful at the transient state during the early heating phase (by solar or artificial source of irradiation) of the surface, when the heat transfers to the inner layers of the materials, according to the thermal properties of the object.Areas of detachment appear warmer as the heat flux enters the surface material.The heat remains in the most exterior layer, which is insulated by the air underneath the surface, instead of flowing into the substrate.Therefore there is an interruption of the linear diffusion of heating.
The law of energy preservation indicates that the variation of energy in time is equal to the exchanged energy between surface and environment.This exchange is due to irradiation on and from the surface, convection, conduction to the interior layers, state changes of chemical species on the surface (evaporation/condensation).All these terms can be in the masonry, although in the presented case study condensation and evaporation did not occur because of the weather conditions.

Near Infrared Imagery
A wide range of sensors that allow the acquisition of near infrared (NIR) images are available on the market.On the other hand, here a solution based on the use of a low-cost sensor derived from a modified consumer SLR camera is proposed.This approach presents a few important key-points: (i) a large sensor format, comparable to that of RGB cameras; this property results in the reduction of the number of camera poses needed for texturing a 3D model of a large object; (ii) the possibility of changing the lens in order to operate with variable camera-object distances; (iii) the opportunity to integrate the NIR sensor into a TLS instrument for the direct texturing of point clouds.
Most SLR digital cameras available on the market have CCD (or CMOS) sensors able to register the electromagnetic radiation in both visible and NIR spectral bandwidths.As a SLR digital camera is sold for normal photographic purposes, the visual quality of the final image is enhanced with a filter applied in front of the CCD sensor.The main goal of this filter (usually called Bayer filter) is the decomposition of the electromagnetic information that each pixel acquires.Light is therefore decomposed into three components, because each pixel can measure only a single channel (red, green, blue).Most lenses cannot correctly focus each wavelength on the sensor plane.This problem generates a loss of definition.To overcome this drawback, a filter that removes the NIR component is applied, but this filter can be manually removed from the camera, while the Bayer filter is often fixed.The NIR filter was then replaced by an optical window transparent to visible and NIR wavelengths and preserving the same optical path for the light rays.After this operation the cameras are sensitive above 0.75 µm and an additional "black" filter is added in front of the lens to obtain the desired range for the NIR image acquisition.
In the experiments carried out during this research a digital SLR camera Nikon D100 (3,008 × 2,000 pixels) was modified in order to acquire also images in the NIR region of the light spectrum.In the Nikon D100 adopted, the NIR filter can be manually removed and substituted, as described in [3] and [16].
The use of NIR technology from both satellite and airborne platforms has been widely exploited for many purposes.Most applications concerned studies about vegetation, due to the emission in the NIR spectrum related to chlorophyll.On the other hand, terrestrial applications have been less relevant up until today.As demonstrated in the example reported in Section 6.2, the use of NIR information in the analysis of building surfaces allows one to highlight the presence of processes and pathologies that are not revealed in the other ranges of the light spectrum.
The geometric calibration of the low-cost NIR camera can be carried out by using standard photogrammetric procedures utilized for RGB cameras [10].

RGB Imagery
The technological development of SLR cameras has provided a reliable solution for the acquisition of RGB images for close range photogrammetric applications.Thanks to a standard calibration procedure [9,10], this kind of camera can be transformed into metric cameras.Commercial and low-cost software packages can be used to this aim.A calibrated SLR camera can be used in applications with a required a precision of 1:100,000 [17,18].
The main advantages of SLR cameras are (i) the possibility of changing lenses in order to cope with several camera-object distances; (ii) the availability of large formats for CCD (or CMOS) sensors that allow one to capture large views of the object and to improve the image block geometry.

3D Modeling Techniques: Photogrammetry and Terrestrial Laser Scanning
The construction of a 3D model is fundamental if the building has a complex shape.The model must be specifically acquired with a survey campaign, except when a model is already available from drawings or previous geometric surveys.Two main approaches can be followed to obtain 3D models and are based on close range photogrammetry and TLS.Advantages and drawbacks of both methods are briefly presented in the next subsections.

Close Range Photogrammetry
Photogrammetry adopts images to derive 3D models of objects [9].After the computation of camera calibration and orientation, images can be used to draw the features describing the geometry of the target object.In the standard practice this task is mostly afforded by interactive 3D reconstruction, where the user identifies corresponding features in at least two oriented images.Such approach, although largely time-consuming, yields to the production of high quality and complete 3D Both point cloud and polygonal models can be colored or textured by using images with a variable radiometric content (e.g., RGB, IR, NIR).A fast solution exploits the possibility of integrating the camera in the TLS instrument (see 3rd figure in Section 5).In this case, the camera is oriented in the intrinsic reference system of each scan in a preliminary stage, so that the image can be directly mapped onto the point clouds (or 3D polygonal models).Usually, the camera has a much smaller field-of-view (FoV) than a TLS.Consequently, the camera can be rotated in different prefixed positions to cover the whole point cloud.A second approach is based on the use of a free-handled camera which is then registered to the point cloud by computing a space resection per each pose.In this case, a set of a few CPs has to be manually measured by the user on either the point cloud or the image to register them.This approach is widely used for mapping IR images.Indeed, due to the small sensor size of such cameras, the integration to the TLS does not provide a geometric resolution comparable to that of laser scans.
The main drawbacks of TLS are the high cost of instruments and data processing software packages, and the big size of sensors and required equipment (batteries or electric generators, tripods, notebook for data acquisition).

A Procedure for Texturing 3D Models with Multispectral Images
In this section, a procedure able to texture 3D models acquired by using TLS with IR imagery is proposed.The method can be run also with other sources for 3D models such as photogrammetric surveys or existing CAD or procedural models [23].The advantages of the methods with respect to the current state-of-the-art are twofold.First, it allows one to overcome the main drawbacks encountered in many applications where the registration of single IR images is achieved through space resection, as mentioned in Section 2. Second, mapping the 3D model of the building with NIR images taken by a low-cost camera becomes quite simple.The proposed solution is quite flexible and incorporates different options to cope with specific properties of each case study.An example showing the operational principle of the proposed methodology is reported in Figure 3.
In the last decades the concept adopted in photogrammetry to reduce the number of control points for image orientation is based on triangulation procedures.The same solution is not directly applicable to IR images, due to a small format with low geometric resolutions and large radiometric changes between images mainly due to the variability of environmental conditions during data acquisition.Indeed, as can be seen in the example in Figure 4, the surface temperature of a building depends on solar radiation.Consequently a slight change of this parameter results in modifying the surface temperature.On the other hand, RGB images can be easily oriented through bundle adjustment and several solutions have been developed in photogrammetry to perform this task [12].This consideration has led to the design of a "bi-camera" system made up of a SLR digital camera and an IR thermocamera (Figure 5).Here the aim is not to exploit stereo-vision (the baseline is limited to 45 cm only), but the purpose of the system is to use the SLR camera for computing the orientation of the other sensor as well.A set of "bi-camera" stations are planned to cover the entire region of interest.Some further RGB images can be taken from other positions in order to strengthen the geometry of the photogrammetric block including new RGB images (Figure 3).In fact, in many cases the geometry required for the IR mapping is not stable for orientation purpose, as shown in the example reported in Section 6.2.The orientation process of RGB images is the subject of Section 5.1.Once this task has been worked out, the knowledge of the relative orientation of the "bi-camera" system allows one to compute the EO of the thermal IR thermocamera as well (Section 5.2).Finally th can be mapp procedure to photogramm ndependent products tha Different options depend upon the modality adopted for tie point measurements and the following bundle adjustment process.Although in the case of buildings, the manual measurement is still the more reliable approach, a partial automation is possible.The use of coded targets is an opportunity given in many photogrammetric packages for close-range applications [24].It allows one to use automated procedures with objects without proper textures, but it also needs a direct access to the façades where targets have to be positioned.In this procedure the automatic marker-less orientation technique ATiPE [20] has been included.Its application is optional, depending upon the image content and the block geometry, as shown in Section 6.
The datum definition of the photogrammetric block is carried out in a standard way, by using ground control points or inner constraints.

Relative Orientation of the "Bi-Camera" System
One of the most remarkable advantages of the "bi-camera" system is the opportunity to employ the RGB data in the processing pipeline.Orientation parameters will be transferred to the thermal camera according to a mathematical relationship that considers the positions of both cameras.The EO parameters of both cameras are expressed through their rotation matrices (R RGB , R IR ) and vectors of perspective centers (X 0RGB , X 0IR ).The parameters corresponding to the RGB camera are computed in the object reference system within a bundle adjustment.The ones of the IR thermocamera are derived by exploiting the relative orientation between both sensors as follows.
In the case of a stereo system where cameras are mounted on a bar, the relative rotation matrix (R*) between both cameras can be expressed using the following condition: (3) The matrix R* does not change if the stereo system is translated or rotated and can be determined with a calibration procedure where both thermal and visible images are oriented within a bundle adjustment using a photogrammetric block made up of stereo pairs.The knowledge of the rotation matrix R RGB is adopted to derive R IR by inverting Equation (3) as R IR = R RGB R*.
The second constraint due to relative orientation concerns the perspective centers of cameras.Although the length of the baseline ||X 0RGB − X 0IR || is a fixed value, the difference between the perspective center components ∆X = X 0IR − X 0RGB cannot be a constant if the stereo system is shifted and rotated.On the other hand, if this difference is written by considering the reciprocal position of the cameras (e.g., using the intrinsic reference system of the right camera), it assumes a constant value t: As a calibration project provides the value of the vector t, during the survey of a building façade the perspective center of the thermocamera can be estimated as: (5) An important operational problem related to the acquisition of IR images is the small format of current sensors.This limitation requires short camera-object ranges if a high resolution of thermal maps is needed.As a consequence, in many cases a fixed focusing distance cannot be applied for the acquisition of each image.The computation of a multiple sets of APs according to diverse focusing distances is not a practical solution.On the other hand, in such cases the method based on the orientation of each single IR image by means of space resection is surely better.Indeed, the main difference between sets of calibration parameters is due to the principal distance c.Consequently, the use of space resection allows one to adjust this parameter per each camera pose.
The relative orientation of the system is estimated within a block of images where both datasets (RGB and IR) are oriented.After the selection of an object with a good texture, series of stereo pairs are acquired.Both projects are oriented separately and then they are registered in the same reference system, where a metric distance is used to remove the scale ambiguity and to estimate R* and t.
The use of constraints like those presented is not new in photogrammetric applications with multi-sensor systems.In [25], a system with three cameras, two visible and one thermal infrared, was calibrated with a bundle adjustment incorporating distance constraints.

Laser Scanning Modeling and NIR/RGB Texturing
Laser scanning is a general solution for 3D modeling.Here the instrument adopted to accomplish the procedure (see the workflow in Figure 3) is a Riegl LMS-Z420i (technical data can be found at www.riegl.com).Although this laser scanner yields lower precisions in surface measurement than phase-shift sensors specifically designed for architectural surveying (see [26] for a review of up-to-date sensors), the integration of a high resolution camera allows us to generate in a direct way a photo-textured 3D model.It is believed to be quite important to select this kind of TLS for multi-source data texturing of building 3D models.Calibration and orientation of both camera devices in the TLS intrinsic reference system (IRS) are obtained in a preliminary stage by adopting standard procedures implemented in the Riegl company data acquisition and processing software Riscan Pro.
The horizontal FoV of Riegl LMS-Z420i is a panoramic kind, but the vertical FoV is limited to a band of 80°.On the other hand, the availability of a calibrated tilt-mount support can be exploited to acquire scans at different vertical inclinations from the same stand-point.A set of rigid-body transformation parameters allow one to register these scans together.In a similar way, the digital camera mounted on the scanner head can be rotated around the instrumental main axis and blocked to fixed steps.Therefore, several small FoV images can be used for texturing a full panoramic laser scan.
In the case the TLS survey has to be carried out from more than one stand-point, scan registration can be accomplished by using retro-reflective targets or surface matching algorithms like ICP [22].If the survey needs to be geo-referenced into an external reference system, some points can be measured with a theodolite from the stations of a geodetic network.

Data Fusion and Photo-Texturing
At the current stage of the procedure, two different datasets are available, each of them defined in a different reference system.The first includes RGB and IR images gathered by using the "bi-camera" system.The orientation of RGB images can be computed by using the options described at Section 5.1, and then the orientation of IR images is derived by exploiting the relative orientation of the "bi-camera" system.The second group collects laser scan point clouds and NIR images captured by  can be used for data delivering and interpretation.Different layers can be overlapped and visualized: the TIN/DEM surface model; the RGB, NIR and IR mosaic or ortho-rectified images; a vector layer if available.As proposed by [13], pan-sharpening of IR images on the basis of RGB images can be applied to improve their raw resolution.However, the authors do not believe that such images could provide additional information to the rectified IR images.This data structure, although only 2D with additional information on the elevations, can be used to perform visual inspections of single layers, comparisons between different layers, but also analytical correlations between raster layers.Figure 7.A portion of the building in the case study reported in Section 6.2 that has been textured by different kinds of multispectral images.From left to right: thermal IR, RGB, NIR, laser return intensity.

A Complex Facade: Rectorate of Politecnico di Milano University
The first case study is the main building at Campus Leonardo (Politecnico di Milano University), in Milan.It hosts the Rectorate and it was built in 1927 in a classical revival style.The building has two storeys, with a banister at the eaves level.Moldings and half relief decorations frame the openings at the ground and second floor.The finish is stucco and decorative concrete.The assessment of the façade allowed one to discover stains, black crusts, misused patches and integration, cracks, lack of mortar joints, and several small delaminated parts of the decorative moldings.These problems are due to pollution, ageing and past maintenance interventions with non-compatible finishing.In 2010 the building management office of the campus committed an assessment to map the detachments of the stucco and decorations, with the aim of planning the next preservation and maintenance activities.
Thermal scanning was accomplished after heating by solar irradiation, in the cooling phase.The targets of the investigation were thermal anomalies due to stucco delamination, thermal loss of the structure, and damages of the finishing.The IRT shooting had the aim of testing the procedure of image recapturing, calibrating, superimposing to the visual 3D model.Thermal analysis was simply based on the comparison of the temperature distribution on a sound zone's surface and a close one under investigation, at the same environmental conditions: the procedure reduces the effects of systematic mistakes and variations of diffused irradiation, as shown in the scientific literature [29].
The survey was based on three TLS stations aimed at covering the whole façade with laser scanning and RGB data; NIR images were not gathered, due to the unavailability of the low-cost camera.A set of seven GCPs (retro-reflective targets positioned on the facade) were adopted to establish a local datum.The IRT acquisition was carried out by adopting the "bi-camera" system in Figure 5(a) incorporating an AVIO thermocamera, featuring a 320 × 240 CCD sensor and equipped with a 74 mm lens.Such FoV results in covering an area of 4.5 × 3.4 m 2 on a planar surface at 10 m from the sensor (average GSD = 14 mm).A total number of 35 IR images were taken to cover the whole façade.These correspond to 35 RGB taken from the "bi-camera" system in different stand-points.These were integrated by further 21 RGB images in free positions to strengthen the geometry of the photogrammetric network adopted for computing the orientation of "bi-camera" stations.Images featured a quite regular geometry and a good texture, so that they could be automatically oriented with the ATiPE procedure.This allowed the automatic extraction of feature-based tie points to compute a bundle adjustment with inner constraints.The photogrammetric project was then aligned to the triangulated 3D model coming from laser scanning by using a few corresponding points and some retro-reflective targets.The thermal images were then mapped obtaining the results shown in Figure 8.As can be seen there are some occlusions generated by the presence of vegetation.The parallax between the laser sensor and the camera gave rise to registration misalignments in correspondence of these objects.This demonstrated that particular attention must be paid during image acquisition.On the other hand, the goal here was to investigate the feasibility of the procedure, obtaining satisfactory results. .In particular, in frames labeled as "1", an anomalous distribution of the surface temperature is shown, which is partially due to prominent and curved elements, and partially to damages.Windows labeled as "2" frame the joint between two decorative elements, where the thermal gradient is due to the damage of the surface caused by the lack of sealing and rainwater infiltrations.Windows labeled as "3" frame small thermal anomalies located on the surface of the volutes (decoration reliefs), that indicate possible detachments of the moldings.The 3D imagery improves the quality of the assessment, because of the precise location and size of the defects and improves the reading of the output.

Application to Clinker Assessment: The "Trifoglio" Building by Gio Ponti (Politecnico di Milano)
The present study case is the three-storey building called "Trifoglio" at Campus Leonardo (Politecnico di Milano).It is a concrete structure that hosts three large class-rooms and other smaller rooms.Its outer walls are protected by a clinker finishing at the first/second floors, and by opaque metallic finishing and concrete on the ground floor.It was built in 1961 (designer Gio Ponti) and recently (2007) went under inspection and restoration for Cultural Heritage.
The use of clinker tiles for finishing the façades of contemporary architecture is a common practice since the 1950-60s.The durability of the ceramic materials, their low cost, stainless and apparently low sensitivity to the effects of pollution, low requirement for maintenance are some of the reasons of their diffused application in the middle-southern region of Europe, as an effective alternative to brick faced masonry, timber cladding and stucco.After more than 50 years since its early use, most of the ceramic finishing shows damage due to weathering/pollution in the mortar joints and the mortar underneath the tiles [30][31][32] that cause the tiles' detachment.
Few non-destructive tests are available to assess the façades: most of these tests require the contact with the surface under investigation such as knocking by a hammer or hands, or the vibration test [33][34][35].IRT is the only technique that ensures a non-contact investigation of the surface, and it has no competition in terms of costs (neither scaffolding nor forklift basket is required) and speed of inspection.Moreover the expected results are a map that localizes and gives an approximated evaluation of the delaminated finishing size.This qualitative approach is the common use of IRT up until now.Nevertheless the resulting evaluation of defects and their size has an approximation that could range between 20% and 40%.Commission errors are often the worst risk for the assessment.They are due to the optical properties (color, reflectivity, damage of the glazing surface), the geometric shape, their application with a non-homogeneous layer of mortar, and damage occurring over time [36].To increase the reliability of the test in [37] a new approach is proposed.Nevertheless, in the present paper, the thermal analysis relies on the traditional recapture of thermal gradients by means of a single scanning, and further developments in research will take into account the integration between the results of the dynamic analysis of the increase of temperatures with the 3D model.
The authors applied the proposed procedure on the north/eastern and eastern sides of "Trifoglio".The experiment was based on the use of a laser scanning Riegl LMS-Z420i equipped with a digital camera Nikon D100.Two scans were acquired and registered by using a few retro-reflective targets.Thermal scanning was accomplished by using a thermocamera NEC H2640 featuring a 640 × 480 CCD sensor and equipped with a 50 mm lens.The resulting FoV covers an area of 3.7 × 2.8 m 2 (camera-object distance is ca. 10 m) with an average GSD = 6 mm.This sensor was setup on the "bi-camera" system in Figure 5(b).Data acquisition was carried out during heating (60 min) by solar irradiation and by convection due to the higher temperature of the air.The targets of the investigation were thermal anomalies due to tile detachments, thermal loss of the structure, and damage to the concrete finishing.The IRT shooting had the aim of testing the procedure of image recapturing, calibrating, and superimposing IR results to the visual 3D model.Therefore, in this phase of the research, also this thermal analysis was based on the comparison of the temperature distribution on a sound zone's surface and a close one under investigation, at the same environmental conditions, as already seen in the previous example.The heating time was evaluated by considering the power/inclination of the sun irradiation hitting the surface, thickness and thermal properties of finishing and masonry, localization of the defects inside the investigated materials, air temperature and relative humidity, wind speed.
Images were oriented with interactive measurements because of the repetitive texture of the object that prevented the successful use of feature-based matching strategies.In this case, more images than those acquired with the bi-camera are necessary to run the automatic orientation procedure.Indeed, images featuring short baselines reduce the perspective differences and simplify the feature-base extraction phase.As the goal in this experiment was to check out the mapping procedure, the authors decided to orient the RGB images manually.The project comprised 77 images taken with the RGB camera and 33 thermal images.Also in this case, this disparity is due to the use of some additional convergent RGB images in order to strengthen the block.Indeed, several IR images were acquired by rotating the camera around its perspective centre without a baseline, leading to an unstable network geometry.The GSD of a RGB shot (estimated in the middle of the façade) was about 5 mm.The average theoretical accuracy of object coordinates resulted σ fac = ±5 mm in direction parallel to the plane of the facade and σ orth = ±13 mm in the orthogonal direction.
The environmental condition has been favorable to the thermal analysis during the recapture of the images: air temperature 17 °C, RH 65%, wind speed 1-5 m/s.The orientation of the inspected façades is north, north-east.Shown in Figure 9 is the final laser model with thermal textures.It is quite simple to verify the correctness of the result by using a visual inspection of elements such as doors and windows.From this model, some considerations can be derived.For instance, windows 1 and 2 frame the thermal loss due to a concrete pillar.Framed zones 3, 4, 7, and 8 show the distribution of small delaminated areas.In squares 5 and 6 the thermal loss due to the floor beams is evident.The referenced location of the anomalies and their metric dimensions enhance the accuracy of the evaluation.
In this case study also NIR images were utilized to texture the triangulated 3D model of the "Trifoglio" building.As can be seen in Figures 7 and 10, this wavelength allowed one to highlight some groups of tiles that had not been enhanced in RGB, nor shown in thermal IR images.These tiles have been positioned at the same time as the ones around them and did not correspond to areas that had undergone previous restoration work.The reason for this behavior is not completely clear and will require further investigation.Probably NIR images have detected a different chemical or geometric structure of their surface, e.g., concerning a diverse degree of opacity, roughness, or a slightly different laying plane.However, it seemed that NIR images are able to reveal different properties of the surface of an object.  .Some details of NIR images captured on the "Trifoglio" building.As can be seen in all sub-images, some groups of tiles appear in a lighter color intensity than background.This radiometric response in the NIR wavelength does not correspond to correlated areas in other kinds of images, but reveals that some groups of tiles might have different chemical or geometrical properties on their surface.

Conclusions and Future Work
In this paper, a procedure for texturing thermal IR images on a 3D building model was introduced.Terrestrial laser scanning is used as principal technique for the acquisition of a 3D model, but photogrammetry or existing CAD models can be applied as alternative solutions.In addition, NIR images captured by a low cost-camera integrated to the laser scanner can be added to the textured model.The proposed methodology tries to simplify the texturing process of IR images, which are usually acquired by low-resolution, small format digital cameras with respect to standard RGB images.The common approach used today is based on mapping each single image through space resection (or by homography if the model is flat).Here a bundle adjustment solution is proposed, but it requires the setup of a "bi-camera" system coupling a high resolution RGB camera to a thermocamera.In future, the orientation of the IR thermocamera could be carried out by exploiting the integrated coaxial RGB sensor that is supposed to impressively improve its performance in the next few years.A direct orientation solution using a low-cost IMU (Inertial Measurement Unit) device is also expected to help image orientation.
On the other hand, thermal sensors will soon enlarge their format, although a geometric resolution comparable to those of RGB digital cameras cannot be obtained due to diffraction limits.Consequently, efforts to improve the registration of IR images and with respect to other data sources deserve further studies.Both tasks are involved in either the calibration of the IR sensor and the relative orientation of the "bi-camera" system.The registration of the stereo system to the laser scanning system is still a manual task, if targets are not used.The recognition of common features between RGB images and laser intensity data could be a way to obtain this result, as suggested by [38].The problem of the variable focusing distance is still a drawback for accurate applications.Further studies will be required both for vendors (e.g., by producing lenses with calibrated focusing steps) and for end-users.In the latter case, calibration models that can be adaptively changed according to the range could be developed for IR thermocameras (if the effective focal length can be digitally recorded by the sensor) as proposed in [39] and [40] for RGB cameras.
The integration between photogrammetry, terrestrial laser scanning, and IR thermography allows one to optimize mapping of thermal anomalies, to ascertain their location, and to improve the geometric resolution of the final textured 3D model.Therefore the better evaluation of the defect/anomaly extension allows one to prioritize the conservation plan and the modality of the maintenance activity.Furthermore, the released graphic documentation consists of orthoimages that can support the preliminary projects for the conservation plan.This is complemented by some digital outputs like photo-textured 3D models and the implementation of data corresponding to different façades in a 2.5D GIS environment.
The optimization of the thermal recaptures in the preliminary phase of the project allows one to improve the decision making process for the economic evaluation of the necessary resources for conservation.On the other hand, the large scale application of the presented procedure could require a higher investment (funds and time) in the preliminary phase instead of at the end of the decision making process.The main costs depend on the articulation/complexity of the buildings (prominent and non-planar parts, projecting decorations) and the extension of the thermal anomalies under investigation.It will be reduced in the further step of the application thanks to the optimization of procedures.
Finally, the application of NIR images in building analysis is somewhat innovative.Such kinds of images can be easily used to texture 3D models if a low-cost camera integrated to a terrestrial laser scanner is employed.On the other hand, further studies are needed to understand the response of construction materials in the NIR spectrum.

Figure 3 .
Figure 3. Example of data acquisition process for mapping IR/NIR/RGB images on a 3D model of a building.

Figure 4 .
Figure 4. Couple of thermal IR images of overlapping portions of the same facade taken at different epochs (1 h).The use of a color palette shows the strong differences due to temperature changes.

Figure 8 .
Figure 8. Thermal IR images mapped onto the laser model of "Rector's office".(a) Global view of the full textured 3D model; (b-d) Details on different parts of the façade, with some windows where different kinds of damages have been detected.

Figure 9 .
Figure 9.The "Trifoglio" building surveyed with the proposed technique.The squares illustrate the areas where some anomalies were found.The discontinuities in the overlapping zones are due to the variation of the temperature during the image acquisition phase.

Figure 10
Figure10.Some details of NIR images captured on the "Trifoglio" building.As can be seen in all sub-images, some groups of tiles appear in a lighter color intensity than background.This radiometric response in the NIR wavelength does not correspond to correlated areas in other kinds of images, but reveals that some groups of tiles might have different chemical or geometrical properties on their surface.
On the other hand, its low resolution could provide only results with a limited precision with current technological development.