Large Scale Automatic Analysis and Classification of Roof Surfaces for the Installation of Solar Panels Using a Multi-Sensor Aerial Platform

Abstract

A low-cost multi-sensor aerial platform, aerial trike, equipped with visible and thermographic sensors is used for the acquisition of all the data needed for the automatic analysis and classification of roof surfaces regarding their suitability to harbor solar panels. The geometry of a georeferenced 3D point cloud generated from visible images using photogrammetric and computer vision algorithms, and the temperatures measured on thermographic images are decisive to evaluate the areas, tilts, orientations and the existence of obstacles to locate the optimal zones inside each roof surface for the installation of solar panels. This information is complemented with the estimation of the solar irradiation received by each surface. This way, large areas may be efficiently analyzed obtaining as final result the optimal locations for the placement of solar panels as well as the information necessary (location, orientation, tilt, area and solar irradiation) to estimate the productivity of a solar panel from its technical characteristics.

1. Introduction

Several techniques have been applied so far for the calculation of the solar incidence on roofs trying to find the most suitable roof surfaces for the installation of solar panels with optimal performance without assessing the possible obstacles that can prevent their installation. In most cases the analysis is done from Geographical Information Systems (GIS) fed with low-resolution raster or vector cadastral data sources [1,2,3,4], GML (Geography Markup Language) systems with simplified LOD (Level Of Detail) [5] or topologically consistent 3D city models obtained through the extrusion of building footprints to a stimated average building height [6]. These techniques, that generally use simplified models and approximations of the positions and orientations of the roofs, could imply errors in the calculation of the solar incidence on the surfaces of the roofs and thus cause large variations in their productivity [7]. In order to avoid these limitations, we developed a methodology able to identify the optimal location for the installation of solar panels [8] and the estimation of their solar irradiation from a precise dataset obtained with a low cost aerial platform equipped with RGB and thermographic cameras. The data processing methodology is scalable to any other aerial data sources like LiDAR (Light Detection and Ranging), acquired from piloted aerial vehicles or UAV’s (Unmanned Aerial Vehicles). The use of geo-referenced 3D dense point clouds of roofs will allow the performance of accurate analysis of areas, orientations and tilts of the roofs. Furthermore, if these data are complemented with qualitative thermal information, which provides direct knowledge about the relative temperature difference between roofs and consequently about the influence of the solar radiation on them, the removal of surfaces that could present ideal geometric conditions to install solar panels will be possible. The thermal values recorded by the thermographic camera cannot be considered as the rigorous temperature values of the object due to the difficulty to perform an accurate emissivity correction on the measurement. However, these values provide useful information for the detection of differences in solar incidence over the object surface. For instance, those surfaces containing elements that could prevent the installation of solar panels such as skylights or chimneys will be automatically detected.

This work proposes and tests a methodology useful for the performance of the automatic classification of roofs regarding their suitability to harbor solar panels, the detection of the optimal location inside these surfaces and the estimation of the solar irradiation received. The methodology consists on the processing of visible and thermographic images acquired from a low-cost aerial trike equipped with a multi-sensor platform (MUSAS, which stands for MUltiSpectral Airborne Sensors) towards the generation of 3D point clouds of roofs.

The paper has been structured as follows: after this introduction, Section 2 includes a detailed explanation of the materials and methods used for data acquisition and processing towards the automatic segmentation and classification of the roof surfaces for the installation of solar panels. Section 3 is devoted to be an explanation of the methodology presented through its application to an urban area selected as case study; finally, Section 4 establishes the most relevant conclusions of the proposed approach. The procedure is presented through its application to a real case study, based on the classification of the roofs in a neighborhood in Ávila, located near the center of Spain (coordinates 40°38'27.6"N, 4°41'27.8"W). The location of the case study determines the restrictions applied for the installation of solar panels following the Spanish Regulation for Construction [9].

2. Materials and Methods

2.1. Equipment

2.1.1. Aerial Trike

The aerial trike is a driven low-cost aerial system equipped with a MUSAS platform able to accommodate different types of remote sensors, from imaging sensors (RGB and thermographic cameras) to navigation systems (GNSS/IMU). The use of these kind of aerial vehicles shall be regulated by the local legislation on air navigation as long as it does not conflict with the general air navigation law of the DGAC (General Guidance of Civil Aviation). The aerial trike is an experimental “Tandem Trike AIRGES” (Table 1), with a weight capacity up to 220 kg and flying altitudes up to 300 m, which is the limit established by the current Spanish legislation for this type of aerial vehicles [10]. The main advantages of this platform over the usual driven aerial vehicles are its low cost, ease of use and the possibility of flying below 300 m, consequently obtaining better GSD (Ground Sample Distance) with the imaging sensors. This increase in the spatial resolution of the RGB images regarding usual aerial photogrammetric flights allows the generation of dense point clouds, as well as improves the use of thermographic images, given that the thermographic sensor presents very limited resolution implying a high GSD that, for altitudes over 300 m could prevent the differentiation of characteristic elements of roofs. Regarding UAV platforms, the aerial trike is able to transport greater weights, which implies the possibility of using more and higher quality sensors with longer autonomy than copter-type platforms. Thus, this aerial platform is used to obtain good quality georeferenced images from a zenithal point of view, allowing for a better spatial perception given the absence of obstacles between the camera and the object.

Table 1. Technical specifications of the manned aerial platform, aerial trike.

Parameter	Value
Empty weight	110 kg
Maximum load	220 kg
Autonomy	3.5 h
Maximum speed	60 km/h
Motor	Rotax 503
Tandem paraglide	MACPARA Pasha 4
Emergency system	Ballistic parachutes GRS 350
Gimbal	Stabilized with 2 degrees of freedom (MUSAS)
Minimum sink rate	1.10
Maximum glide rate	8.60
Plant surface	42.23 m2
Projected area	37.80 m2
Wingspan	15.03 m
Plant elongation rate	5.35
Central string	3.51 m
Boxes	54 boxes
Zoom factor	100%

Table 1. Technical specifications of the manned aerial platform, aerial trike.

Parameter	Value
Empty weight	110 kg
Maximum load	220 kg
Autonomy	3.5 h
Maximum speed	60 km/h
Motor	Rotax 503
Tandem paraglide	MACPARA Pasha 4
Emergency system	Ballistic parachutes GRS 350
Gimbal	Stabilized with 2 degrees of freedom (MUSAS)
Minimum sink rate	1.10
Maximum glide rate	8.60
Plant surface	42.23 m2
Projected area	37.80 m2
Wingspan	15.03 m
Plant elongation rate	5.35
Central string	3.51 m
Boxes	54 boxes
Zoom factor	100%

Sensors are supported by a specific gyro-stabilized platform (MUSAS) (Figure 1) allowing for a full coverage of the study area with an appropriate GSD. This device includes two servomotors arranged on the x and y axes to maintain the vertical position of the camera along the flight path with precision. The servomotors are controlled by an Arduino board, which incorporates an IMU with 6 degrees of freedom: 3 accelerometers with a range of ± 3.6 G m/s², a double-shaft gyroscope (for pitch and roll control) and an additional gyroscope for yaw control (both gyroscopes have a measurement range of ± 300°/s). The navigation devices allow the geolocation of each datum for the successive generation of geo-referenced point clouds with real and thermographic texture.

Last, the software developed for the control of the device was based on Quad1_mini V 20 software, with DCM (Direction Cosine Matrix) as the management algorithm for the IMU [11].

Figure 1. Aerial trike and MUSAS platform used in this study; bottom-left corner: detail of the gyro-stabilized platform for the installation of sensors.

Figure 1. Aerial trike and MUSAS platform used in this study; bottom-left corner: detail of the gyro-stabilized platform for the installation of sensors.

2.1.2. RGB Camera

RGB cameras are used for the acquisition of images towards the reconstruction of 3D point clouds as well as to provide visual information of the state of the roofs. The visible camera selected for this work is a full frame reflex camera Canon 5D mkII. This camera has a CMOS sensor which size is 36 × 24 mm with a resolution of 21.1 megapixel and equipped with a 50 mm focal length lens. The size of the image captured with this sensor is 21 MP with a pixel size of 6.4 µm.

The camera is calibrated prior acquisition in order to allow the correction of the distortion and perspective effects from the data collected and the 3D reconstruction in the photogrammetric process. The calibration of cameras working in the visible band of the electromagnetic spectrum (Table 2) is performed through the acquisition of multiple convergent images of a geometric pattern (known as calibration grid) with different orientations of the camera. The adjustment of the rays ruling the position of the camera and the image in each acquisition allows the determination of the inner orientation parameters of the camera (focal length, format size, principal point, radial lens distortion and decentering lens distortion). The camera calibration was processed in the commercial software ImageMaster which performs the automatic detection of the targets in each image and computes and adjusts the orientation of each image, resulting in the computation of the internal calibration parameters of the camera. Since the flight was performed at medium speed and low altitude, there should not be any change in the “camera-lens” system caused by sudden movements of the platform or major changes in the atmosphere. For this reason, the camera calibrations performed in the laboratory have been considered as fixed for both cameras.

Table 2. Interior orientation parameters of visible camera Canon 5D mkII, result of its geometric calibration.

Parameter	--	Value
Focal length (mm)	Value	50.1
Format size (mm)	Value	34.819 × 23.213
Principal point displacement (mm)	X value	−0.21
	Y value	−0.11
Radial lens distortion	K1 value (mm−2)	6.035546 × 10−5
	K2 value (mm−4)	−1.266639 × 10−8
Decentering lens distortion	P1 value (mm−1)	1.585075 × 10−5
	P2 value (mm−1)	6.541129 × 10−5
Point marking residuals	Overall RMSE (pixels)	0.244

Table 2. Interior orientation parameters of visible camera Canon 5D mkII, result of its geometric calibration.

Parameter	--	Value
Focal length (mm)	Value	50.1
Format size (mm)	Value	34.819 × 23.213
Principal point displacement (mm)	X value	−0.21
	Y value	−0.11
Radial lens distortion	K1 value (mm−2)	6.035546 × 10−5
	K2 value (mm−4)	−1.266639 × 10−8
Decentering lens distortion	P1 value (mm−1)	1.585075 × 10−5
	P2 value (mm−1)	6.541129 × 10−5
Point marking residuals	Overall RMSE (pixels)	0.244

2.1.3. Thermographic Camera

The thermographic camera selected for this study is the FLIR SC655. This device has been specially developed for scientific applications. It allows the capture and recording of thermal variations (Figure 2) in real time, enabling the measurement of heat dissipation or leakage. Its sensor is an Uncooled Focal Plane Array (UFPA) 0.3 MP, capturing radiation with 7.5 to 13.0 µm wavelengths and measuring temperatures in a range from −20 °C to 60 °C. The Instant Field of View (IFOV) of the camera is 0.69 mrad, and its Field of View (FOV) is 25° (Horizontal) and 18.8° (Vertical) with the current lens of 25 mm focal length.

Figure 2. Comparison between the products generated by the different used imaging sensors. (Left) RGB image. (Right) Thermographic image with the thermal values represented using a color map.

Figure 2. Comparison between the products generated by the different used imaging sensors. (Left) RGB image. (Right) Thermographic image with the thermal values represented using a color map.

Thermographic cameras capture radiation in the infrared range of the spectrum, in contrast to photographic cameras that work in the visible range. For this reason, the geometric calibration (Table 3) of the camera is performed using a specially designed calibration field, presented in [12], which is based on the capability of the thermographic cameras for the detection of objects being at different temperatures even if they are at the same temperature but with different emissivity values. This calibration field consists on a wooden plate with black background (high emissivity) on which foil targets are placed (low emissivity). In this case, the calibration was processed in the commercial photogrammetric station Photomodeler Pro5©, which not only computes the calibration parameters of the cameras in a procedure analogous to the one previously explained for ImageMaster, but it also provides the standard deviation of the calibration parameters as a value to test their quality.

Table 3. Interior orientation parameters of thermographic camera FLIR SC655, result of its geometric calibration.

Parameter	--	Value	Std. Deviation
Focal length (mm)	Value	25.063	0.022
Format size (mm)	Value	10.874 × 8.160	0.002
Principal point displacement (mm)	X value	−0.174	0.022
	Y value	0.024	0.026
Radial lens distortion	K1 value (mm−2)	5.281 × 10−5	2.4 × 10−5
	K2 value (mm−4)	7.798 × 10−7	6.1 × 10−7
Decentering lens distortion	P1 value (mm−1)	1.023 × 10−4	1.2 × 10−5
	P2 value (mm−1)	−3.401 × 10−5	1.3 × 10−5
Point marking residuals (pixels)	Overall RMSE	0.173	--

Table 3. Interior orientation parameters of thermographic camera FLIR SC655, result of its geometric calibration.

Parameter	--	Value	Std. Deviation
Focal length (mm)	Value	25.063	0.022
Format size (mm)	Value	10.874 × 8.160	0.002
Principal point displacement (mm)	X value	−0.174	0.022
	Y value	0.024	0.026
Radial lens distortion	K1 value (mm−2)	5.281 × 10−5	2.4 × 10−5
	K2 value (mm−4)	7.798 × 10−7	6.1 × 10−7
Decentering lens distortion	P1 value (mm−1)	1.023 × 10−4	1.2 × 10−5
	P2 value (mm−1)	−3.401 × 10−5	1.3 × 10−5
Point marking residuals (pixels)	Overall RMSE	0.173	--

2.2. Methodology

2.2.1. Flight Planning and Data Acquisition

Proper flight planning is important to optimize available resources, ensuring a high quality of the images and minimizing capture time. The spatial information required for the flight planning can be obtained free of charge from the National Center of Geographic information in Spain (CNIG), from its National Aerial Orthoimage Plan (PNOA, 2009) with a GSD of 0.25 m and a Digital Terrain Model (DTM) with a 5 m grid resolution. The flight planning was carried out based on the classical aerial photogrammetric principles [13] but adapted to the new algorithms and structure from motion (SfM) strategies [14], ensuring image acquisition with forward and side overlaps of 70% and 30%, respectively. Given the format difference between the thermographic and RGB sensors, time between shots will be considered independently in order to ensure these overlaps.

The gyro-stabilized platform ensures the theoretical geometric setup of a photogrammetric aerial image capture in each shot, which stablishes that the optical axis of the camera should be zenithal. The theoretical definition of scale in digital aerial photogrammetry is related to the geometric resolution of the pixel size projected over the ground (GSD). This parameter can be calculated by considering the relationship between flight altitude over the ground, the GSD, the focal length of the sensor and the pixel size (Equation (1)). Considering that the objective of this study is not to identify small entities, and in order to allow the procedure to be scalable to sparse 3D point clouds from other measurement systems, we fixed the target GSD for the lower resolution sensor between 10 and 15 cm.

$$\frac{f}{H}=\frac{pixel size}{GSD}$$

(1)

where $f$ is the focal length of the sensor; $H$ is the flight altitude over the ground and $GSD$ is the Ground Sample Distance.

Due to the lower resolution of the thermographic images, the flight planning is performed according to the characteristics of this sensor (Figure 3), considering the planning completely valid also for the RGB sensor.

In order to locate the solar incidence deficiencies over the objects under study in the higher production time zone of the solar panels, the survey should be performed when the Sun is located at the highest point of the solar path.

2.2.2. 3D Point Cloud Reconstruction

The image-based modelling technique based on the combination of photogrammetry and computer vision algorithms allows the reconstruction of dense 3D point clouds. The absolute orientation (position and attitude) of each image is known because the position of the imaging sensors is registered with respect to the GNSS/IMU navigation devices of the aerial trike, and data acquisition is synchronized with the navigation. For this reason, using these absolute orientation as initial approximations, only an orientation refinement was required for the precise geolocation of the images.

The orientation refinement process starts with the automatic extraction and matching of image features through a SIFT [15] (Scale-Invariant Feature transform) algorithm which provides effectiveness against other feature detection algorithms. In addition, these features present optimal results for these type of aerial surveys where scale variations are minimal and perspective effects are almost nonexistent thanks to the gyro-stabilized platform. Next, taking as initial approximation the external orientation parameters provided by the GNSS system and as fixed the laboratory internal calibration parameters, an orientation refinement was performed through a global bundle adjustment between all images by means of the collinearity equations [16,17]. As a result, the spatial and angular geo-positioning of the RGB sensor was computed enabling the dense point cloud generation. Next, a dense matching method through the MicMac implementation [18] based on the semi global matching technique (SGM) [19,20] allows the generation of a dense and scaled 3D point cloud resulting from the determination of the 3D coordinates of each pixel. This process was performed using the RGB images because their higher resolution provides a 3D point cloud with higher point density than the point cloud generated from thermographic images. Regarding computation effort, the last step is the most expensive and time-consuming. All these photogrammetric tasks were performed using the Photogrammetry Workbench (PW) [21].

Last but not least, a dense 3D point cloud with thermographic information was obtained for the case study. This thermographic mapping was obtained thanks to the known baseline between both sensors which was previously calibrated in laboratory. In particular, this calibration consisted in solving the relative orientation of the thermographic camera regarding the RGB camera through the simultaneous visualization of a common pattern (the same used for the thermographic camera calibration). After the identification of homologous entities between both sensors, the calculation of the baseline (bx, by, bz) and the boresight (Rx, Ry, Rz) values was performed.

2.2.3. Automatic Planes Segmentation

Once the 3D point cloud is generated from the RGB images and the thermographic texture is applied, the following procedure is the segmentation of the roofs. This is performed in different steps using the Point Cloud Library (PCL) [22], open source and licensed under BSD (Berkeley Software Distribution) terms, which includes a collection of state-of-the-art algorithms and tools useful for 3D processing, computer vision and robotic perception. First, ground and vegetation are removed using a pass through filter with a Z coordinate restriction. A pass through filter performs a simple filtering along a specified dimension removing the elements that are either inside or outside a given range. In this case, the filter gets the minimum Z value of the point cloud and removes all points with a Z value close to the minimum Z. The distance threshold is established as a parameter set for the user, being 5 m a recommendable value, established experimentally by the authors after the segmentation of several point clouds. In addition, although the presence of points belonging to the facades is minimal due to the vertical configuration of the capture, a conditional filter based on the angle between the normal vector of each point and the vertical axis is applied to remove these points. Then, the point cloud including the roofs is segmented using a Euclidean cluster segmentation, providing better and faster results for the subsequent extraction of the different planes of each roof [23,24] by dividing the point cloud in individual roofs. This way, RANSAC (Random Sample Consensus) algorithm [25] is applied to each roof individually for the extraction of the composing planes.

Once each roof is clustered in different planes, and the coefficients that describe each surface in a Cartesian coordinate system by the general equation of the plane are known (Equation (2)), the geometric evaluation is performed, resulting in the orientation and tilt values for each surface from which we can calculate the percentage of solar energy productivity regarding the maximum productivity of the installed solar panels.

$$Ax+By+Cz=D$$

(2)

where A, B and C are the components of the vector normal to the plane, whereas D is the independent term.

From the components A, B and C of the general equation of the plane that match with the values of the unit vector normal to the plane, denoted as $\overrightarrow{v}$ (Equation (3)) we can proceed to compute the orientation and tilt values for each roof. The orientation of the surface requires the computation of the angle between the projection of vector $\overrightarrow{v}$ on the horizontal plane and the cartesian Y-axis (Equation (4)). The quadrant to which the angle belongs, identified through the evaluation of the sign of A and B components of the unit vector (Figure 3 left), will allow us to get the orientation angle of the surface (Figure 3 right).

$$\overrightarrow{v}=\left(A,B,C\right)$$

(3)

$$\text{ β}=arcTan\left(\raisebox{1ex}{$A$}\!\left/ \!\raisebox{-1ex}{$B$}\right.\right)$$

(4)

where $\overrightarrow{v}$ is the vector normal to the plane; β is the angle between the horizontal projection of the normal vector and the cartesian Y-axis.

Figure 3. (Left) Quadrant determination from the components of the vector normal to the plane. (Right) Absolute orientation of the roof, denoted as “Or”, from the angle between the projection of the vector on the horizontal plane and the cartesian Y-axis.

Figure 3. (Left) Quadrant determination from the components of the vector normal to the plane. (Right) Absolute orientation of the roof, denoted as “Or”, from the angle between the projection of the vector on the horizontal plane and the cartesian Y-axis.

The tilt of the surface is calculated by a simple trigonometric process in a right-angled triangle where the normal vector is considered the hypotenuse and its projection on the horizontal plane (Z = 0) and C value the adjacent and opposite legs respectively (Figure 4) (Equations (5–7)).

Figure 4. Tilt angle from the values of the normal vector of the plane.

Figure 4. Tilt angle from the values of the normal vector of the plane.

$$d=\sqrt{A^2+B^2}$$

(5)

$$\text{α}=arcTan\left(\raisebox{1ex}{$C$}\!\left/ \!\raisebox{-1ex}{$d$}\right.\right)$$

(6)

$$Tilt=90°-\text{α}$$

(7)

where d is the module of the vector $\overrightarrow{v}$ projected on the horizontal plane (Z = 0); α is the angle between the horizontal plane and the normal vector of the surface; Tilt represents the tilt of the plane.

2.2.4. Geometric Analysis and Classification

Once the different planes of the roofs are detected and segmented, their areas, tilts and orientations are analyzed in order to perform their geometric classification. The roofs with an area smaller than required for the installation of solar panels and those with North orientation are discarded due to their inadequacy. The remaining roofs are classified in different groups according to their theoretical productivity as a function of their tilt and orientation, also taking into account the possibilities of either integrating the solar panels in the roof or installing them in configurable supports. The CTE (Technical Edification Code) sets the South orientation as the optimal position for these elements, with a tilt equal to the latitude where they are installed. However, the limits on the tilt can be computed according to the minimum efficiency allowed for the orientation of the surface using the method explained by the IDAE (Institute for Diversification and Saving of Energy) (Equations (8) and (9)) [9].

If $Tilt>15°$

$$Solar energy losses\left(\%\right)=100·\left[1.2·{10}^{-4}·{\left(Tilt-\text{φ}+10\right)}^2+3.5·{10}^{-5}·O{r}^2\right]$$

(8)

If $Tilt<15°$

$$Solar energy losses\left(\%\right)=100·\left[1.2·{10}^{-4}·{\left(Tilt-\text{φ}+10\right)}^2\right]$$

(9)

where $Or$ is the Orientation of the plane and $\text{φ}$ its latitude.

These limits allow the evaluation of the suitability of the surface under study and the computation of the ideal geometry of the support platforms if needed, taking into account that productivity losses cannot exceed the 20% if solar panels are installed directly over the roofs, consequently keeping the angle of solar incidence of the roof. However, productivity losses should be under 10% using the general method which implies the installation of solar panels in supports to modify the angle of solar incidence regarding the tilt of the roof.

2.2.5. Thermographic Refinement of Surfaces

Once the geometric classification is performed, the thermographic data enables the location of elements that constitute an obstacle for the installation of solar panels in order to avoid protrusions or shadows that could reduce their productivity, produced by both the elements of the roof surface and by nearby buildings. This identification of obstacles allows the analysis to find the optimal location within each roof. The existence of obstacles and anomalies in the roofs involves the presence of different materials and surfaces with different emissivity values, so the temperature detected by the thermographic images will be different even if the solar radiation received was the same. In addition, the existence of obstacles that prevent direct sunlight will be manifested by changes in the temperatures of the roof surfaces. These facts allow the location of those surfaces of the roofs affected by anomalies through the performance of a statistical study of the mean and the standard deviation of temperatures. A point will be considered as an obstacle when the difference between its temperature value and the mean temperature of the roof is higher than the standard deviation of the temperatures of the surface that is being analyzed. Considering the fact that the shadow produced by an obstacle will move following the path of the Sun, a perimeter defined by the user around each obstacle will be considered for further analysis. Although a self-occlusion analysis would be appropriate for the determination of the perimeter, the performance of this processing for each obstacle in large study areas would be a high computationally demanding task. Experimental analysis performed by the authors establishes 0.5 m as a recommendable value.

2.2.6. Location of the Most Suitable Zones

The existence of obstacles, detected by the thermographic refinement step, prevents the installation of solar panels covering the whole surface of those roofs classified by the geometric analysis as having the optimal geometry (orientation and tilt) for this purpose. Therefore, the development of a procedure to locate the optimal zones for panel installation inside optimal surfaces (Figure 5) is necessary. This task is addressed by analyzing the spatial distribution of those 3D points belonging to the optimal surfaces and not considered as obstacles. This approach is focused on four steps individually applied to each surface. (i) Projection of the point cloud to the horizontal plane in order to evaluate the surfaces in a 2D environment, simplifying the process; (ii) Extraction of those points that describe the perimeter of the surface to evaluate (concave hull); (iii) Computation of the largest empty rectangles (no concave hull points inside); (iv) Re-projection of the largest empty rectangles to 3D. The concave hull can be defined as a set of points that enclose a concave region and define the perimeter of an unorganized set of points allowing any angle between consecutive edges. The area of the defined shape should be minimized without distorting the appearance of the point cloud

Figure 5. Procedure to locate optimal zones inside optimal surfaces. (Left) 3D surface selected (red rectangle) over the roof. (Right) From top to bottom: (a) surface under evaluation, without obstacles, projected to the horizontal plane; (b) concave hull that defines the perimeter of the surface; (c) largest empty rectangles inside the concave hull and (d) evaluated surface inliers of the largest empty rectangles re-projected to 3D.

Figure 5. Procedure to locate optimal zones inside optimal surfaces. (Left) 3D surface selected (red rectangle) over the roof. (Right) From top to bottom: (a) surface under evaluation, without obstacles, projected to the horizontal plane; (b) concave hull that defines the perimeter of the surface; (c) largest empty rectangles inside the concave hull and (d) evaluated surface inliers of the largest empty rectangles re-projected to 3D.

The extraction of the points that describe the perimeter of the surface to evaluate is performed through the computation of their concave hull using an implementation of the “alpha shape” computational geometry approach based on the Delaunay triangulation [26]. The computation of the largest empty rectangles is performed through an iterative implementation presented in [27], restricting the orientation of the edges of the rectangles to the same orientation of the evaluated surface and its perpendicular vector on the horizontal plane. Finally, the largest empty rectangles are re-projected to the 3D space obtaining the georeferenced location of the optimal zones for the installation of solar panels in each roof, and their area.

2.2.7. Estimation of the Solar Irradiation

To introduce this section it will be useful to explain the differences between the terms “solar irradiance” and “solar irradiation”. Solar irradiance describes the instantaneous radiant flux per unit area that is being delivered to a surface, usually expressed in W/m². It varies depending on the location of the surface, time and date, and atmospheric conditions, among other factors. Solar irradiation, also known as insolation and typically expressed in kWh/m²/a, represents the amount of solar energy that can be collected on a surface per unit of area within a given time (i.e., solar irradiance integrated over time). The use of georeferenced point clouds at this point is crucial given that solar irradiation is heavily influenced by day length and the position of the Sun (solar ephemeris), obtaining very different results for different locations on Earth. Solar irradiation may also significantly differ between building roofs in the same zone, depending on the orientation of the roof: a more favorable angle of a surface to the Sun means better exposure and more solar energy. Two roof surfaces of the same size at the same location but different orientations (azimuth) and tilts, may drastically differ in their solar potential [28,29]. This solar irradiation value, complemented with the area of the optimal surface obtained in the previous step, allows the computation of the solar irradiation of a surface in kWh/a [6,30]. Therefore, we can say that solar irradiation on a roof surface is mainly derived of four essential geographic parameters (area, tilt, azimuth and geographic location) that may be defined as a function (Equation (10)):

$$I={{\displaystyle \int }}^{\text{}}\left(\left[\text{φ},\text{λ}\right],\text{β},\text{θ},A,\left[\text{ω}\right]\right)$$

(10)

where I is the solar irradiation on a roof; $\text{φ}$ is the latitude; λ is the longitude;β is the tilt of the surface; θ is the azimuth; A is the area and ω other components such as cloud cover.

Solar irradiance can be decomposed into three components [31,32]: direct irradiance, diffuse irradiance and reflected irradiance. These three components are important and significantly influence the total irradiance [33], so they have been analyzed separately in this approach. Direct and reflected irradiance and their adjustment for a tilted and oriented surface have been calculated as described in [34]. The estimation of the diffuse solar irradiance is more complex and must be adjusted to one of the several existing empirical models [35,36,37].

Therefore, solar irradiation can be estimated by integrating the solar irradiance over a period of time (Equation (11))

$$I={\sum }_{i=start}^{end}{E}_i+\text{Δ}{t}_i$$

(11)

where I is the solar irradiation; Δt_i is the length of the i^th time interval and E is the solar irradiance

The annual solar irradiation is calculated following the protocol used in the Solar3DCity [5,38] that includes the modeling of “real-sky” conditions through the integration of the historical EPW (Energy Plus Weather file format) weather data downloaded from the nearest weather station [39]. Solar ephemerides are from XEphem [40,41], which have been proven suitable for solar irradiation studies [42]. The computations use the empirical anisotropic irradiance model developed by [43], which was implemented in the solpy library [44]. The annual solar irradiation is then calculated for each roof surface by integrating the hourly irradiance values of the entire year. The computation of solar irradiation values can be extrapolated for every azimuth/tilt combination obtaining a function, usually called tilt-orientation-factors (TOF), which can be represented in a 3D diagram. This TOF function can be considered an additional product, since it represents the optimal tilt and orientation for a location, which is the main aim of several location-based studies [45,46].

3. Experimental Results

3.1. Study Case

The proposed methodology has been validated in a wide urban area in the city of Avila (Latitude 40°66'N, Longitude 4°70'W), with an extension of 237,250 m² in a rectangular shape of 365 m × 650 m, chosen due to the existence of roofs with different geometric characteristics, which is an interesting characteristic for testing the methodology.

Flight planning was performed considering the limitations of the thermographic sensor (Figure 6). Time between shots, in order to ensure properly overlapping, was stablished as 500 ms for the RGB sensor and 160 ms for the thermographic sensor, both for a flight speed of 50 km/h. Due to the speed flight, it was necessary to minimize the shutter opening time in order to avoid the motion blurring effect. This requirement needed several tests to find the optimal ISO, aperture and shutter speed configuration to ensure the correct exposure of the images minimizing the effect of any external factor that might damage the survey. As result, 409 images for the RGB sensor and 5312 images for the thermographic sensor were captured covering the whole study area. The flight altitude over the ground selected was 160 m, resulting on a GSD of 11 cm and 2 cm for the thermographic and RGB cameras, respectively.

Figure 6. Flight planning: (Top) Areas captured by each image shot for the thermographic sensor. (Middle) Flight planning for the navigation of the aerial trike. (Bottom) GPS track after the aerial trike flight with the position of each image shot.

Figure 6. Flight planning: (Top) Areas captured by each image shot for the thermographic sensor. (Middle) Flight planning for the navigation of the aerial trike. (Bottom) GPS track after the aerial trike flight with the position of each image shot.

The full resolution RGB images were processed according to the photogrammetric and computer vision methodologies obtaining as result a point cloud of 24,858,863 points, implying an average resolution of 100 points/m² (≈1 point/GSD). The thermographic texture was projected over the point cloud obtaining a hybrid 3D point cloud (Figure 7) with thermographic texture mapped over the points of the roofs.

Figure 7. Hybrid 3D point cloud generated from images captured with the RGB camera integrating thermographic texture mapped over the roofs. Areas shown in detail in consecutive figures are remarked in red.

Figure 7. Hybrid 3D point cloud generated from images captured with the RGB camera integrating thermographic texture mapped over the roofs. Areas shown in detail in consecutive figures are remarked in red.

The survey was designed in order to fulfill the conditions established by PNOA (Spanish National Plan of Aerial Orthophotography) [47]. These conditions state the required accuracy to the GSD value for the X, Y and Z point coordinates. In order to check compliance with these requirements, a topographic GPS survey was performed, consisting of 26 check points homogeneously distributed over the whole study area (Figure 8).

Figure 8. Check points homogeneously distributed over the whole study area.

Figure 8. Check points homogeneously distributed over the whole study area.

This 3D point cloud with thermographic texture is the input of the algorithm developed. In the first step, ground, vegetation and façade points are removed using the pass through and conditional filters obtaining as result a point cloud of the roof with 8,930,030 points.

The point cloud of the roofs with thermographic texture is introduced into the segmentation process (Figure 9). The result of the Euclidean cluster segmentation is the extraction of 37 roofs that have been automatically segmented into 168 planar surfaces by the RANSAC algorithm and evaluated and classified regarding their geometrical suitability (location, area, orientation and tilt) for the installation of solar panels.

Figure 9. (a) Result of the extraction of each roof using the Euclidean cluster extraction algorithm. (b) Results of the extraction of each planar surface using the RANSAC algorithm. (c) Details of the previous point cloud segmented and classified by its suitability to install solar panels. (Green) Surfaces suitable to install solar panels without supports. (Yellow) Surfaces suitable to install solar panels using a support to modify the solar incidence angle. (Red) Surfaces not suitable for the installation of solar panels.

Figure 9. (a) Result of the extraction of each roof using the Euclidean cluster extraction algorithm. (b) Results of the extraction of each planar surface using the RANSAC algorithm. (c) Details of the previous point cloud segmented and classified by its suitability to install solar panels. (Green) Surfaces suitable to install solar panels without supports. (Yellow) Surfaces suitable to install solar panels using a support to modify the solar incidence angle. (Red) Surfaces not suitable for the installation of solar panels.

The radiometry of the thermographic images mapped on the 3D point cloud has been used as described in the methodology for the identification of possible obstacles (Figure 10). These obstacles have been taken into account for the location of the optimum zones for the installation of solar panels. This way, 67 zones (Figure 11) have been classified as optimal for the installation of solar panels, covering an area of 2230.77 m². Finally, all these data have been complemented with the estimation of the solar irradiation received by each optimal surface (Figure 12), quantifying an accumulated solar irradiation for the whole study area of 4.01 × 10⁶ kWh/a.

The geometric and irradiation results are represented individually for each roof (Table 4). The geometric results show the azimuth (°) and tilt (°) of each surface and the area (m²) of each optimal location. The irradiation results show the yearly solar irradiation for unit of area (kWh/m²/a), computed from the azimuth and tilt values supported with the irradiation models and the atmospheric data, and the yearly total solar irradiation (kWh/a) of each optimal surface from its area. The area and the irradiation results of each optimal location for the installation of solar panels inside each roof surface are represented consecutively. The results of each segmented surface are recorded showing the total calculation of area and solar irradiation of the optimal areas in the whole surface.

Figure 10. Detail of the point cloud segmented and classified after the thermographic analysis. (Black) Points removed by the statistical thermographic analysis. (Grey) Perimeter defined by a distance parameter around the obstacles detected to assist the optimal location of solar panels.

Figure 10. Detail of the point cloud segmented and classified after the thermographic analysis. (Black) Points removed by the statistical thermographic analysis. (Grey) Perimeter defined by a distance parameter around the obstacles detected to assist the optimal location of solar panels.

Figure 11. Georeferenced point cloud of the detail zones after the process. Optimal surfaces (rectangles) for the installation of solar panels are highlighted in blue. Roof ID’s are used in Table 4.

Figure 11. Georeferenced point cloud of the detail zones after the process. Optimal surfaces (rectangles) for the installation of solar panels are highlighted in blue. Roof ID’s are used in Table 4.

Figure 12. Tilt-orientation-factors (TOF) representation for Ávila (Spain) estimated by Solar 3D city.

Figure 12. Tilt-orientation-factors (TOF) representation for Ávila (Spain) estimated by Solar 3D city.

Table 4. Example of the results after the processing of a roof of the dataset (roof and numbers in Figure 8).

Surface	Tilt (°)	Azimuth (°)	Yearly Solar Irradiation by m2 (kWh/m2/a)	Area of Optimal Location (m2)	Yearly Total Solar Irradiation (kWh/a)
1	19.3568	346.0080	1295.011	--	--
2	19.4968	165.6463	1824.290	103.066	188,023
--	--	--	--	99.026	180,652
--	--	--	--	18.197	33,196.8
3	19.0947	156.1201	1651.910	22.938	37,891.8
4	19.3133	76.2060	1523.485	--	--
5	19.2929	346.1950	1295.834	--	--
6	19.5358	166.1320	1824.914	140.417	256,249.0
--	--	--	--	110.023	200,782.0
--	--	--	--	41.781	76,245.8
--	--	--	--	25.003	45,628.7
7	19.3667	156.0211	1652.508	38.410	63,472.9
8	19.2102	76.2400	1524.167	--	--
Total results for optimal locations of surface 2				220.289	401,871.8
Total results for optimal locations of surface 3				22.938	37,891.8
Total results for optimal locations of surface 6				317.224	578,905.5
Total results for optimal locations of surface 7				76.703	126,752.5
Total results for optimal locations of the roof				637.154	1,145,421.6

Table 4. Example of the results after the processing of a roof of the dataset (roof and numbers in Figure 8).

Surface	Tilt (°)	Azimuth (°)	Yearly Solar Irradiation by m2 (kWh/m2/a)	Area of Optimal Location (m2)	Yearly Total Solar Irradiation (kWh/a)
1	19.3568	346.0080	1295.011	--	--
2	19.4968	165.6463	1824.290	103.066	188,023
--	--	--	--	99.026	180,652
--	--	--	--	18.197	33,196.8
3	19.0947	156.1201	1651.910	22.938	37,891.8
4	19.3133	76.2060	1523.485	--	--
5	19.2929	346.1950	1295.834	--	--
6	19.5358	166.1320	1824.914	140.417	256,249.0
--	--	--	--	110.023	200,782.0
--	--	--	--	41.781	76,245.8
--	--	--	--	25.003	45,628.7
7	19.3667	156.0211	1652.508	38.410	63,472.9
8	19.2102	76.2400	1524.167	--	--
Total results for optimal locations of surface 2				220.289	401,871.8
Total results for optimal locations of surface 3				22.938	37,891.8
Total results for optimal locations of surface 6				317.224	578,905.5
Total results for optimal locations of surface 7				76.703	126,752.5
Total results for optimal locations of the roof				637.154	1,145,421.6

Those surfaces with North orientation have not been evaluated as not being candidates for the installation of solar panels by their geometry. For that reason, the “area of optimal location” and “yearly total solar irradiation” values of these surfaces are not available.

In order to evaluate the weaknesses and the reliability of the process, the roof planar surfaces of the study area have been analysed by visual inspection. Through this inspection 216 planar surfaces have been counted in the whole area under study. The difference of 48 planes existing between the reality and the result of the processing is distributed as follows. 42 planar surfaces have been correctly extracted by the algorithm but their evaluation has been discarded due to their minimum size that would make impossible the installation of solar panels. The 6 remaining surfaces match with roofs where the angle between their planar surfaces is next to 180° (Figure 13). In this case, RANSAC algorithm is not able to segment the roof in different planar surfaces obtaining as result of the interpolation a single horizontal plane that does not match with the reality of the roof. Surfaces whose tilt comes close to horizontal plane can be classified as non-optimal for installation of solar panels. For these reasons, it could be considered that the proposed approach is efficient enough for the location of the optimal zones to install solar panels, since it only misses 2.8% of the number of roofs of interest.

Figure 13. (a) Diagram of the RANSAC interpolation between two roof surfaces with an intersection angle next to 180°. (b) Top view of the evaluated roof. (c) Top view of the RANSAC algorithm results.

Figure 13. (a) Diagram of the RANSAC interpolation between two roof surfaces with an intersection angle next to 180°. (b) Top view of the evaluated roof. (c) Top view of the RANSAC algorithm results.

However, it must be emphasized that this algorithm only allows the automatic extraction of those surfaces which geometric approach can be interpreted and represented by a plane. Both curve and complex surfaces, where the installation of solar panels might be possible if complex support structures are used, are discarded.

3.2. Computing Efficiency Analysis

The machine used to perform the computation was a Microsoft Windows 8.1 workstation with 32 GB RAM, a 3.40 GHz Intel Core i5-3570K processor and an Nvidia Quadro 2000 GPU. The 3D dense point cloud generation was the most computationally demanding process investing a total of 13 h 20 min for the whole data set.

Regarding the algorithm developed for the automatic location of the best places for the installation of solar panels and the estimation of their incident solar irradiation, the execution time performed through a single thread algorithm was 26 min for the whole study area. However, multithreading computation is available, which would reduce the computation time proportionally to the number of concurrent running threads.

4. Conclusions

This article presents a methodology for the automatic processing of 3D point clouds with thermographic information for the automatic plane segmentation of the roof surfaces and their classification according to their theoretical productivity derived from their geometric characteristics (orientation, tilt and area). A refinement according to the thermographic radiometry allows the location of anomalies or obstacles that could prevent the installation of solar panels or reduce their productivity. This way, the location of the optimal zones for the installation of solar panels inside each surface is determined, avoiding obstacles detected in the thermographic refinement and allow the computation of the solar irradiation received on these zones. In particular, the hybrid product obtained provides complete thermographic and metric information of the different roof surfaces, which enables better detection, spatial location and interpretation of the suitability of each surface to harbor solar panels than traditional methods. In addition, the high level of automation of the procedure allows the evaluation of wide urban areas in a fast and accurate way, without the need to consult the technical documentation of each building.

The results of the application of the procedure are geo-referenced point clouds of the surfaces of the roofs with their geometric information (area, tilt, orientation), classified by their theoretical productivity regarding solar energy catchment. The computation has considered those elements that could reduce the productivity of the solar panels, as well as a perimeter zone potentially affected by the influence of the obstacles along the day. What is more, the procedure calculates the optimal tilt of the supports if they were necessary. In addition, results are combined in order to locate the optimal zones to install the solar panels inside each roof surface and complemented with the computation of the received solar irradiation. The latter, together with the technical characteristics of the panels to install, make possible the estimation of the productivity of the solar panels.

Thanks to that, the geometric information of the 3D point clouds has been preserved during all processing steps, enabling the performance of measurements directly on the point clouds of the segmented planes. As a result, the software developed could be used as a decision-making tool for those issues related with the dimension or type of solar panels that could be installed, calculating the actual productivity of the selected components.

In addition, the methodology proposed is valid for processing datasets captured with different airborne sensors such as LiDAR or any other RGB and thermographic sensors transported by any manned or unmanned aerial vehicle. This fact, supported with the existence of open access (public and free) LiDAR data [48] throughout the national territory, provides to this methodology and the developed software a direct applicability with minimum cost.

Regarding the efficiency of the approach developed, the methodology is applicable to the processing of entire towns or cities. Even assuming that the entire process of 3D reconstruction and the post-processing of point clouds for very large areas could imply some days of processing, the time required to perform the in-situ visual inspection (supported with technical documentation of the object) of large areas by human operators to achieve a similar result is not viable. It should also be noted that a human operator currently performs a subjective decision of the suitability of a roof to harbor solar panels (without generation of metric products and objective evaluation of obstacles). For this last reason, even in the case that a significant number of operators is available to perform the in-situ inspection in a “reasonable” period of time, the proposed approach is also an important breakthrough, especially since only an aerial trike pilot and an expert operator are required to perform the survey and to oversee the processing and analyze the results, respectively.

The main drawback of the methodology proposed is the possibility of finding particular homogeneous surfaces on the roofs where either the photogrammetric approach or the LiDAR measurement fails. These could lead to the appearance of areas without points or too noisy that prevent the segmentation process.

This project opens new trends for future work both from a sensorial and methodological point of view. Concerning the first, the evolution of multimodal matching techniques which can be applied to the matching of RGB and thermographic images would allow the automatic registration between both data sets, avoiding the tedious manual identification of homologous points between them, in those cases where the accurate positioning of both cameras is not possible. From the methodological point of view, the integration of the Sun path enables the most accurate determination of the location of the shadow zones that could reduce the productivity of the solar panels. The procedure considers shadows produced both by obstacles located on the same roof and by adjacent buildings. Improving the latter through a self-occlusion analysis directly from the 3D geometric information, together with the incorporation of the possible surrounding orography occlusion effects will be the next milestone towards the increase of the reliability of the decision-making processes using as input the processing results obtained from this methodology.

Regarding the possibility to take advantage of the large amount of information derived from this methodology, it would be interesting to advance in the automatic generation of BIM (Building Information Modelling) and CIM (City Information Modelling), using gbXML or CityGML languages, respectively, for the representation of the 3D point cloud geometry of the segmented roofs and the inclusion of the solar irradiation estimation. Progress toward these systems will allow to work with lighter and more agile information that we can include as an additional layer on a GIS enabling the integration of this information with other data sources (cadastral information, demographic information, urban parameters, etc.) to perform urban energy management tasks.