Experimental Fire Measurement with UAV Multimodal Stereovision

Abstract

In wildfire research, systems that are able to estimate the geometric characteristics of fire, in order to understand and model the behavior of this spreading and dangerous phenomenon, are required. Over the past decade, there has been a growing interest in the use of computer vision and image processing technologies. The majority of these works have considered multiple mono-camera systems, merging the information obtained from each camera. Recent studies have introduced the use of stereovision in this field; for example, a framework with multiple ground stereo pairs of cameras has been developed to measure fires spreading for about 10 meters. This work proposes an unmanned aerial vehicle multimodal stereovision framework which allows for estimation of the geometric characteristics of fires propagating over long distances. The vision system is composed of two cameras operating simultaneously in the visible and infrared spectral bands. The main result of this work is the development of a portable drone system which is able to obtain georeferenced stereoscopic multimodal images associated with a method for the estimation of fire geometric characteristics. The performance of the proposed system is tested through various experiments, which reveal its efficiency and potential for use in monitoring wildfires.

1. Introduction

Every year, wildfires cause ecological, economic, and human disasters [1]. The term “wildfires”— also known as “forest fires” or “wildland fires”, is commonly used to refer to unwanted fires that burn forests and wildlands [2,3]. Annually, more than 340 million hectares of vegetation are burned on our planet by these fires [4]. Climate and weather are some of the major factors increasing wildfire hazard and risk [5]. In the near future, global temperatures are expected to be warmer than current levels, severe droughts to be more frequent, and fire seasons to be longer in many ecosystems. Other anthropogenic factors have contributed to the increase in fire risk worldwide [2,6]; for instance, anarchic extension of the urbanization in wildland–urban interfaces (WUIs) or fire exclusion policies, which contribute to fuel accumulation, can potentially increase the impacts of wildfires. With the combination of these factors, wildfires may occur more frequently. This evolving scenario requires strategies that are able to reduce the risk of fires and thereby decrease their economic, environmental, and social impacts. Therefore, management, prevention, and control actions are performed. The effectiveness of these types of actions is related to the knowledge of the phenomena that occur during the outbreak and spread of fire. Wildfire research has been conducted for more than 50 years, in order to understand these phenomena, to develop propagation and behavior models, to predict the evolution of wildfires [7,8,9,10,11] and the heat they transfer [12,13,14], and to develop decision management systems for fire fighting and land-use planning [15,16,17,18,19].

Experimental data of fire spread across vegetal fuels in laboratory and field conditions are needed to better understand the phenomenon and improve and/or validate models. Fire front characteristics such as position, rate of spread, base area, height, length, width, depth, inclination, shape, surface, and volume are needed. Three-dimensional (3D) information are as complete as can be expected. Thus, for example, all the points of a non-linear fire front could be monitored, which indicate fire front deformation, which can occur due to terrain topology, wind, or vegetation. The energy transferred from a fire to a target depends on the distance between the target, as well as the fire and the orientation of the fire surface in relation to the position of the target [20]. Obtaining the 3D shape of a fire and having, at each instant, the distance from the fire to the target and the surface of the flame that is oriented toward the target allows for the estimation of this energy. In recent decades, research activity in the fire community has focused on developing methods based on computer vision to measure the geometric characteristics of propagating fire. The step of detection of fire pixels in an image is essential in the process of measuring fire by vision, as it determines the accuracy with which the characteristics of the phenomenon can be estimated. The visible domain is the reference spectrum used by researchers working on wildfires, due to the simplicity of use of visible cameras and their affordable price. However, the detection of fire pixels in an image representing the visible spectrum is challenging and it appears [21] that the performance of 11 state-of-the-art fire color segmentation algorithms [22,23,24,25,26,27,28,29,30,31,32] (benchmarked using a data set containing more than 500 wildfire images [33]) are dependent on the features of the fire, including, for instance, color, texture, presence of smoke, and luminosity of the background. The use of infrared images can help to overcome the problem of fires masked by smoke. Multiple works have been conducted in this area. Gouverneur et al. [34] found that the best spectral bands for fire detection (long-distance fire camera) were short wavelength infrared (SWIR: $1.4$–3 $\mathsf{\mu }$m), middle wavelength infrared (MWIR: 3–5 $\mathsf{\mu }$m), and long wavelength infrared (LWIR: 8–15 $\mathsf{\mu }$m). However, as the fire emits in several spectral bands in a non-uniform way, the fire areas obtained from different spectral images do not perfectly overlap. This is due to some hot gas zones not being captured by the visible spectrum [34,35]. Thus, infrared images alone cannot be used for measuring the geometric characteristics of fires.

Systems using multiple modalities have been proposed: Verstockt et al. [36] proposed the use of a system that combines LWIR and visible spectrum sensors, as well as a system that uses time of flight (TOF) and visible sensors [37]. In these latest studies, the experiments were limited to controlled environments. As the resolutions of the cameras used in these studies were low, they cannot be used for fire metrology.

For the development of a vision-based measurement system dedicated to fires, two kinds of frameworks have been developed: The first uses mono-cameras which are positioned to obtain complementary views [38,39,40,41], but is not applicable in an uncontrolled environment. Martínez-de-Dios et al. [42] developed a vision-based measurement system at the field scale using fixed cameras positioned at two viewpoints (frontal and lateral views) to obtain complementary information. Frontal images allow for the estimation of fire front position and the rate of spread, while lateral cameras are useful for extracting the maximum flame height, the flame inclination, and the fire base width. An infrared camera is also deployed in a lateral position, in order to obtain the fire location and the fire base width for situations with smoke present, which masks visual images. Real-world estimations of fire features are computed by calibrating the cameras using a terrain model, camera position, and artificial beacons and natural marks. The developed system was used to estimate the position of the most advanced point of the fire front and its width. Finally, a 3D view of the fire model is generated using the computed measurements. Other authors have proposed improvements of this method using visual and infrared cameras fixed on the ground or mounted onboard unmanned aerial systems with a differential global positioning system (GPS) and inertial measurement units (IMU), using data fusion techniques to integrate the results from the different cameras [43]. This framework can be used to estimate the location of the fire front, the rate of spread, and the maximum height of the flames in real time. The 3D fire shape is obtained by the concatenation of triangles, each of which is characterized by the estimated features. This system estimates part of the geometric characteristics needed by researchers working on forest fire propagation and behavior modeling. However, when using 2D information, 3D depth information cannot be obtained for all the points of a fire front and, therefore, certain characteristics cannot be estimated. Moreover, deployment can be challenging in areas that are difficult to access.

The second family of frameworks is based on the use of stereovision to compute 3D information [44]. This method obtains depth information from two simultaneous and staggered images of the same scene without the need for landmarks or artificial beacons. It has been applied at the laboratory scale to measure the geometric characteristics of a fire front [45,46]. A system was also developed for the semi-field scale [47]. The framework supports the use of multiple stereo pairs which are positioned to capture complementary views of the fire front during its propagation. Multimodal stereo pairs operating in both the visible and near-infrared spectral bands are used. Each of the cameras forming the stereo pair is rigidly fixed on a metallic bar with a one meter baseline. This baseline value, which is associated to the focal length that is used and with a camera–fire distance of 15 m, produces a theoretical precision of depth measurement of less than 0.3 m.

Registration is conducted using the captured views to build a complete 3D reconstruction of the fire front. This is achieved using multisensory fusion based on visual data, GPS positions, and IMU inertial data. This framework effectively estimates the geometric characteristics of a fire propagating in a field of about 10 m over time: position, rate of spread, base width, depth, perimeter, height, length, flame inclination angle, 3D shape, surface, view factor, and volume. However, it has certain limitations: First, it is necessary to anticipate the fire path, in order to optimally position the cameras. Second, as the theoretical precision of the depth measurement increases proportionally to the square of the camera–object distance [48], precision loss occurs when the fire moves away from the cameras. The use of a stereovision system carried by an unmanned aerial vehicle (UAV) can overcome these problems, as the drone can stay close to the fire throughout its evolution. The first challenge to be met when using a vison system whose position and orientation change at every moment is expressing the data in a global reference frame. The second is computing the equation of the local propagation plane of the fire and its main direction for each image acquisition instant, which are necessary for the estimation of the geometric characteristics.

This paper presents a UAV multimodal stereovision system for the measurement of the geometric characteristics of fires propagating over unlimited distances. It is based on our previous work in the area of 3D vision-based metrology of fires [47,49]. A vision device that is fixable on a UAV, composed of two cameras operating simultaneously in the visible and the infrared ranges, and able to produce georeferenced stereoscopic multimodal images is presented. The proposed framework permits the estimation of the 3D shape of the fire front, its position on the ground, rate of spread, surface, thickness, height, and inclination angle for any terrain configuration and without using a digital terrain model. To evaluate the performance of the system, tests were conducted with experimental fires spreading over several tens of meters on terrains with variable slope.

2. Materials and Methods

2.1. Multimodal Stereovision System

The stereovision system is composed by two FLIR${}^{\circledR }$ Duo Pro R 640 cameras (FLIR Systems Inc, Wilsonville, OR, USA) [50] rigidly fixed on a carbon fiber draping axis with an inter-camera distance of 0.85 m, which is attached to a Matrice 600 Pro DJI UAV (DJI, Shenzhen, China) (Figure 1).

The camera combines an FLIR${}^{\circledR }$ Tau 2 thermal sensor and a 4K sensor, such that images in the visible and LWIR spectra are taken simultaneously. The visible sensor has a focal length of 8 mm with a pixel resolution of $4000\times 3000$ pixels and a field of view of ${56}^{\circ }\times {45}^{\circ }$. The infrared sensor has a focal length of 13 mm with a pixel resolution of $640\times 512$ and a field of view of ${45}^{\circ }\times {37}^{\circ }$. The maximum frame rate of the camera is 30 Hz but, for researchers working on fire behavior modeling, the frequency of 1/4 frames per second (fps) is generally considered adequate and, so, we decided to set it to this value. Simultaneous image acquisition by the two FLIR${}^{\circledR }$ Duo Pro R cameras is triggered using an external 50 fps pulse width modulation (PWM) signal generated by a Raspberry Pi 2B+ (Newark Corporation, Chicago, IL, USA) computer, which is also used to store the data; the images are processed offline. A 3S 500 mAh LiPo battery and a power bank are used to power the cameras and the computer, respectively; they are fixed on the drone chassis, near its center of gravity. The total weight of the vision system, integrating the cameras, batteries, and the Raspberry computer, is 2.278 kg.

The theoretical precision of depth measurement ($\Delta z$) in a stereo system is linked to the precision of the disparity measurement ($\Delta d$) by:

$$\Delta z=|{\displaystyle \frac{z^2}{Bf}}|\Delta d,$$

(1)

where $\Delta d$ is the disparity in pixels, z is the real distance between the object and the stereovision system in meters, B is the baseline of the stereovision system in meters, and f is the focal length of the cameras in pixels. Considering $B=0.85$ m, $\Delta d=1$ and $z=15$ m, the theoretical precision of the depth measurement ($\Delta z$) of the stereovision system composed by the visible sensors is 0.4 m. This value is 1.7 m if the distance between the UAV and the fire is 30 m.

The stereovision system was calibrated outdoors using a $1\times 1$ m checkerboard formed of black and white square patterns. The white squares were covered with aluminium to obtain information in the visible and IR domains. Calibration was performed using Bouguet’s MATLAB${}^{\circledR }$ (MathWorks, Natick, MA, USA) calibration toolbox [51], which permits the extraction of intrinsic and extrinsic parameters that are used in the stereo image rectification process.

The FLIR${}^{\circledR }$ Duo Pro R camera integrates an onboard GPS/compass sensor, IMU board, and altitude sensors. These sensors are used to obtain the position and orientation of the vision system at each instant and generate georeferenced images.

2.2. Methodology

A diagram of the proposed methodology is shown in Figure 2. The first step of the process consists of detecting the fire pixels, which is conducted in both the LWIR spectrum and the visible range. Then, the visible picture is analyzed such that the most important feature points can be implemented in the stereo matching algorithm. From the corresponding points, a triangulation procedure permits their 3D reconstruction. At each image acquisition time, the field local plane equation is estimated using the lower 3D fire points. The 3D fire points are then projected in several frames to produce results expressed in a reference frame that is intrinsic to the fire front. The final step consists of reconstruction of the 3D fire model and estimation of the geometric characteristics of the fire using the transformed 3D points.

2.2.1. Multimodal Fire Pixel Detection

At each image acquisition instant, one visible spectrum image and one LWIR spectrum image are obtained by each Duo Pro R camera. Figure 3 shows an example of fire images obtained simultaneously. A method combining information obtained in the infrared image and in the visible image is used to detect the fire pixels in the visible image.

The MATLAB toolbox code developed by Balcilar [52] with small changes is used to produce superimposed images. The multimodal procedure is performed in two steps. In the first step, the LWIR image is processed with the the Otsu threshold selection method [53] to find the location of the highest intensity pixels corresponding to the fire pixels. Only the pixels situated in the visible image at the same position as those detected in the LWIR image are considered to detect fire pixels.

A fire pixel in the pre-selected area of the visible image is detected as follows: The effectiveness of 11 state-of-the-art fire color segmentation algorithms [22,23,24,25,26,27,28,29,30,31,32] is visually evaluated on the first image of an image sequence associated to a propagating fire with a graphical interface. A previously described method [31] was adapted to use only the second step of this procedure (the first step corresponding to the pixel pre-selection procedure). The procedure that achieved the best pixel detection is then used for all images in the sequence.

2.2.2. Feature Detection, Matching, and Triangulation

Features in the fire areas are detected, such that they can be matched and used in a triangulation method to obtain 3D points. From these 3D data, the geometric characteristics of the fire can be estimated.

Features are detected in the fire areas of images by processing these zones successively with the Harris detection algorithm [54] and the speeded-up robust features (SURF) procedure [55]. The first method is applied with a box filter of $5\times 5$ pixels. The second one uses six scale levels with the following filter size dimensions: $27\times 27$, $51\times 51$, $75\times 75$, $99\times 99$, $123\times 123$, and $147\times 147$. The points obtained by each procedure are added and duplicate points are deleted.

The zero mean normalized sum of squared differences (ZNSSD) method [56] is used in the feature matching procedure to measure descriptor similarities. Figure 4 shows 1803 matched points obtained from the features detected in the stereoscopic images, with an example shown in Figure 3.

3D fire points are obtained by triangulation, as described in [44], using the coordinates of the matched points and the intrinsic and extrinsic parameters of the stereovision device. Their tridimensional coordinates are given relative to the center of the left camera of the stereovision system. The farthest points from the main cloud are not considered if one of the two conditions presented in Equation (2) is true; in which case, the point is identified as an outlier and is eliminated.

$$\left\{\begin{array}{c}|d_1-{\mu }_{d1}|\ge 3{\sigma }_{d1},\hfill \\ |d_2-{\mu }_{d2}|\ge 3{\sigma }_{d2},\hfill \end{array}\right.,$$

(2)

where ${\mu }_{d1}$ and ${\sigma }_{d1}$ are the mean and standard deviation of the distance $d_1$ between this point and its four neighbors, respectively; while ${\mu }_{d2}$ and ${\sigma }_{d2}$ are the mean and standard deviation of the distance $d_2$ between this point and the camera frame origin, respectively.

Figure 5 shows the 3D fire points for the fire presented in Figure 3.

2.2.3. Fire Local Propagation Plane and Principal Direction Estimation

Fire Local Propagation Plane

As there is no a priori knowledge about the plane on which the fire spread occurs, the equation of the local propagation plane must be computed in order to estimate information such as the position of the fire on the ground and its height. First, at each image acquisition time, the equation of the local propagation plane is obtained by a least squares method using the lower 3D fire points. Second, the 3D points belonging to planes obtained at successive instants and with slope variations of less than $3^{\circ }$ are gathered and used to compute a novel equation of the local plane passing through them. This procedure improves the estimation of the local propagation plane equation. Figure 6 presents the estimated topology of the ground on which the fire shown in Figure 3 propagated. At each instant, the local plane is characterized by its longitudinal angle ($ϵ$) and its lateral angle ($\theta$).

Principal Direction

The principal direction of a fire is needed for the computation of certain characteristics, such as its inclination, width, and length. It is variable during the spread of the fire and depends on the wind and the ground slope. It is given by the vector whose extremities are the barycenters of two successive sets of ground fire points. For this, at time t, the set of points obtained at the time t and those obtained at the instant $t-1$ are used.

2.2.4. 3D Fire Points Transformations

At each instant of image acquisition, the vision system carried by the drone has position and orientation differing from those of the previous moment. As the 3D points obtained by stereovision are expressed in a frame positioned in the vision device (camera frame) and as this frame is moving over time, it is necessary to project all the 3D points into a global reference frame, in order to produce results that show the temporal evolution of the fire geometric characteristics, such as the position of the front line.

The origin $O_C$ of the camera frame is the optical center of the left visible camera, the x-axis ($X_C$) corresponds to the axis from the left camera towards the right camera, the z-axis ($Z_C$) is perpendicular to $X_C$ and directed forward from the cameras, and the y-axis ($Y_C$) is the axis perpendicular to the other two axes such that the resulting triad is right-handed, as shown in Figure 7.

In this work, we chose the GPS data ($O_G$) associated to the first image obtained by the stereovision system before the take-off of the drone as the origin of the global reference frame. The drone was positioned approximately in front of the propagation zone, such that the frame has its x-axis ($X_G$) parallel to the width side of the burn zone. Its z-axis ($Z_G$) corresponds to the depth of the burn zone, and the y-axis ($Y_G$) is such that the triad is left-handed (i.e., corresponding to the altitude), as shown in Figure 8.

Let $\alpha$, $\beta$, and $\gamma$ be the roll, pitch, and yaw angles estimated by the IMU situated on the vision device when the UAV is flying, respectively. Let ${\alpha }_0$, ${\beta }_0$, and ${\gamma }_0$ be the roll, pitch, and yaw angles estimated by the IMU situated on the vision device before the UAV takes off and associated to the global reference frame, respectively. Let $R_{\alpha }$ ($R_{\beta }$, and $R_{\gamma }$) be the homogeneous rotation matrices around the x, y, and z axes, respectively. Let T be the translation matrix defined using the coordinates N, E, and U of the position of the vision system in the camera frame—that is,

$$T=\left[\begin{array}{cccc}1& 0& 0& E\\ 0& 1& 0& N\\ 0& 0& 1& U\\ 0& 0& 0& 1\end{array}\right].$$

(3)

N, E, and U are computed by applying a transformation on the GPS coordinates of the camera frame in the global coordinate frame.

Let $S_{yz}$ be a homogeneous matrix performing a swap between the y-axis and the z-axis.

The transformation matrix is defined by:

$$M_{C⟶G}=R_{\gamma }(-{\gamma }_0)·R_{\beta }(-{\beta }_0)·R_{\alpha }\left({\alpha }_0\right)·S_{yz}·R_{\alpha }(-\alpha )·R_{\beta }\left(\beta \right)·R_{\gamma }\left(\gamma \right)·T·S_{yz}·R_{\alpha }\left(180\right).$$

Fire geometric characteristics such as height, length, and inclination angle must be estimated independent of the local propagation plane on which the fire is located. To do this, a transformation function is applied to the 3D fire points, to work as if the slope of the propagation plane was zero. A frame called the “slope frame” is considered; its origin $O_S$ is equal to $O_G$, its x-axis ($X_S$) is parallel to the ground plane lateral slope, the z-axis ($Z_S$) is parallel to the ground plane longitudinal slope, and the y-axis ($Y_S$) is parallel to the normal of the ground plane. This frame is considered at each moment of image acquisition (and 3D points calculation), where each local propagation plane (characterized by its average angles $ϵ$ and $\theta$) imposes an orientation of the S frame, as shown in Figure 9.

The coordinates of the 3D points expressed in the global reference frame are transformed using a matrix $M_{G⟶S}$. Let $ϵ$ and $\theta$ be the longitudinal and lateral angles of the local propagation plane obtained at a given acquisition image instant, respectively. Let $R_{ϵ}$ and $R_{\theta }$ be the homogeneous rotation matrices of the angle $ϵ$ around the $X_G$-axis and of the angle $\theta$ around the $Z_G$ axis, respectively. Then, the transformation matrix $M_{G⟶S}$ is defined by:

$$M_{G⟶S}=R_{ϵ}·R_{\theta }.$$

Finally, as the fire may change direction over time, in order to compute geometric characteristics such as width, flame inclination, and length, it is necessary to rotate the 3D points such that the depth axis of the frame used to express them corresponds to the instantaneous direction; the z-axis of the slope frame will be parallel to the instantaneous main direction of the fire at each instant. Let $\eta$ be the angle between the instantaneous fire direction and the z-axis of the slope frame; the 3D fire points are then rotated around the $Y_S$-axis by the angle $\eta$ (Figure 10).

2.2.5. Fire Geometric Characteristics Estimation

From the transformed 3D fire points, the geometric characteristics of the fire can be estimated. All of the points are used for the 3D reconstruction of the fire and the computation of its surface and view factor. Only the points that are on the ground are considered for the estimation of the front position, base area, width, and depth.

Shape and Volume

The Delaunay triangulation method is applied to the 3D fire points, which provides a set of tetrahedrons [57]. The 3D fire shape depends on the radius used. Tests were conducted by varying the radius between 0.3 and 0.4 m in steps of 0.01 m; the value that produced the best result was 0.35 m. The tetrahedrons for which the projection of their centers onto the segmented image do not match with fire pixels are eliminated. Figure 11 shows the 3D reconstruction of the fire shown in Figure 3. The sum of the volumes of the selected tetrahedrons is considered to be the volume of the fire.

Surface and View Factor

Considering the set of triangles forming the surface of the fire, it is possible to compute the surface of the fire that produces the heat flux in front of the fire and the fraction of the total energy emitted by the fire surface and received by a target, regardless of its position (also called the view factor).

To compute the fire surface, only the surfaces of the triangles that are not masked by others and which are oriented in the principal direction of the fire are added.

The view factor is estimated by considering all the fire triangles and the 3D coordinates of the target. For this, we used the method described in [58]. The view factor $F_{S⟶\mathrm{d}T}$ for the radiation between the whole flame surface S and the target area $\mathrm{d}T$ is defined as the fraction of the total energy emitted by all the elementary triangle surfaces $\mathrm{d}{S}_i$ and received by $\mathrm{d}T$. Let $r_i$ be the distance between the target area and the center of the $i\mathrm{th}$ triangle of surface $\mathrm{d}{S}_i$. Let ${\theta }_{S_i}$ and ${\theta }_{T_i}$ be the angle between $r_i$ and the normal of the elementary triangle surface and the angle between $r_i$ and the normal of the target surface, respectively. Therefore:

$$\mathrm{d}{F}_{S⟶\mathrm{d}T}={\displaystyle \frac{\mathrm{d}T}{S}}\sum _{\begin{array}{c}i=1\\ cos{\theta }_{S_i}>0\\ cos{\theta }_{T_i}>0\end{array}}^N{\displaystyle \frac{cos{\theta }_{S_i}cos{\theta }_{T_i}}{\pi r_i^2}}\mathrm{d}{S}_i.$$

Position, Rate of Spread and Depth

Figure 12 shows the transformed fire ground points of Figure 5.

A fire is delimited and localized by its front and back lines. To find them, the points are processed in sectors. For the experiments described in this paper, they were 15 cm wide, but this value can be scaled to the width of the fire, in order to consider different scenarios. In each sector, the most- and least-advanced points are selected. The front (back) line of the fire is obtained using a B-spline interpolation with a polynomial function passing through the most (least) advanced points, as shown in Figure 13.

Over time, the lines move and distort. This can be modeled by considering the velocity of each point of the lines. Researchers working on fire propagation modeling are interested in the rate of spread of characteristic points situated in the center and on the sides of the fire line. This can be estimated by computing the ratio of the distance between two equivalent points on two successive lines divided by the time interval between the two acquisition moments of the images from which the curves were calculated, as previously described in [58]. Considering two successive lines and a given point in the first line, its equivalent point is the intersection point between the normal of the first line passing through the given point and the second line.

The depth of the fire front is estimated by computing the distance between the mean point obtained from the most advanced points and the mean point estimated from the least advanced points.

Combustion Surface

When modeling wildfires, the fuel below a flame contributing to the combustion must be considered; for this purpose, the surface can be approximated by simple forms [59]. From the ground fire points, a polygon is obtained using the method described in [60] (Figure 14). This corresponds to the base of the fire and to the surface of the fuel that is in combustion. Its area is estimated by summing the surface of the triangles contained in the polygon.

Width, Height, Length and Inclination Angle

The fire width, height, length, and inclination angle are computed with 3D points transformed such that the depth axis of the frame used to express them corresponds to the instantaneous direction of the fire.

The fire width is estimated in three steps: First, the two points that have extreme x coordinates among the least-advanced fire ground points are identified. Second, points such that the x coordinate is not more than 15 cm from the latter are used to compute two mean points. Third, the width is computed as the Euclidean distance between the two mean points (Figure 15).

The distance between the base plane and each point of the upper part of the fire corresponds to the height of the point. To produce an average fire height, the mean point of the 3D points situated in the highest 30 cm of the fire is considered.

The length of the fire has been defined by researchers working on fire behavior modeling as the distance between the top of the fire front and the most advanced point of the fire base. In this work, the length is calculated as the Euclidean distance between the average point ($P_H$) of the 3D points located within the highest 30 cm of the flame and the average point ($P_F$) of the most advanced ground 3D fire points (Figure 16).

The fire inclination angle is estimated as the angle between the segment $\overline{P_H{P}_F}$ and the normal of the fire base plane.

3. Results

Due the unpredictable and non-reproducible behavior of fire, it is difficult to evaluate the uncertainty of the proposed framework.

An initial experiment was conducted using a parked car to evaluate the accuracy with which the system evaluates the position of an object and its dimensions. The UAV completed a full turn around the parked car to acquire pictures from each side of the car with a 10–15 m UAV–car distance. The position of the car was measured with a GPS sensor positioned on the roof of the vehicle at the base of the antenna of the car.

Figure 17 shows the GPS position of the drone during the test and the position of the GPS of the car.

Only visible images were processed and, as the method used for fire pixel detection was not usable, features were selected manually in the images of the four sides of the car. As the goal of this test was to evaluate the efficiency of the framework for estimating the position and the dimensions of an object and not its exact shape, it was not necessary to add the car marks to the homogeneous parts of the car to increase the number of features (Figure 18).

Figure 19 shows the 3D car points obtained separately from the four complementary car views. From each set of points, a surface was computed using the Delaunay method, as shown in Figure 20. Figure 21 presents the complete 3D reconstruction considering all 3D car points in the same set.

We can observe that the overall shape of the vehicle was compatible with the image shown in Figure 18. From the 3D car shape, we estimated the width, height, and length of the car. To estimate the car position, the 3D points of the car antenna were estimated for each car side.

Table 1. Comparison between real and estimated measurements of the car.

	Length (m)	Width (m)	Height (m)	Position (lat${}^{\circ }$, lon${}^{\circ }$)
Real	3.99	1.64	1.50	(42.2999911${}^{\circ }$, 9.1755291${}^{\circ }$)
Estimated	3.96	1.62	1.48	(42.2999923${}^{\circ }$, 9.1755259${}^{\circ }$)
Error	0.7%	1.2%	1.1%	0.26 m

presents a comparison between the data estimated by stereovision and the real data (dimensions of the car given by the manufacturer and position of the car obtained with a GPS sensor).

A second experiment was conducted on an outdoor pseudo-static fire. Wood wool was placed on an area of $3\times 5$ m. The form of the fire changed, but not its position (Figure 22).

At each instant, from the estimated fire points situated on the ground, the front line was obtained. As an example, Figure 23 shows the 3D points computed from the fire image acquired at $t=100$ s and Figure 24 shows the fire front lines estimated at $t=92$, 96, and 100 s. The form of the lines changed over time but remained in the same position, in keeping with reality.

Finally, several tests were conducted outdoors in controlled fire burn areas. All fires were set on a semi-field scale on an area of $5\times 10$ m consisting of an initial flat zone of 2 m and a second zone with an incline at ${20}^{\circ }$, which was covered with wood wool (Figure 25).

The drone flew at a height of 10 m, maintaining a distance of 15 m from the fire zone. The inclination of the stereovision system was set to $-3^{\circ }$ downwards, in order to obtain the best shooting angle at 15 m from the fire. The fire was set along the short side of the rectangle, in order to produce propagation over the entire length of the fuel area along the long side of the rectangle.

Figure 26 shows the Cartesian positions of the stereovision system during the experiment.

Figure 27 shows the ground 3D fire points obtained for the complete spreading and the estimated propagation plane. The framework modeled the ground in two parts: the first had a longitudinal inclination equal to $0^{\circ }$ and the second had a longitudinal inclination equal to ${21}^{\circ }$. These values were in accordance with the real characteristics of the ground.

Figure 28 shows the fire after a few minutes of propagation. It appears that the fire split into several parts and the frontal first (which is of interest to researchers working on fire behavior modeling) had its ground points at very different depth positions.

Only the frontal part of the fire was considered to produce results. It was identified as the part that corresponded to the biggest part of the fire and had the highest 3D points.

Figure 29 displays the segmented frontal fire area; 1130 matched points were detected in this zone.

Figure 30 shows the 3D points obtained from the matched points presented in Figure 29 and the estimated local propagation plane.

Figure 31 presents the computed 3D reconstruction of the frontal part of the fire. This shape was fully compatible with the image in Figure 28.

Figure 32 displays the temporal evolution of the fire position, showing the front lines only for certain moments.

Figure 33 shows the front lines obtained at moments closer together than those considered in Figure 32, with the order of the polynomial function being higher than that used to generate the lines in the previous figure. The lines are clearly smoother than those shown in Figure 32 and were used to identify the fire regimes. A regime is characterized by a constant rate of spread. In this example of fire propagation, three regimes (with relevant lines) were included in the rectangles. These were in accordance with the phenomena occurring over time: first, the fire started on a flat ground with little speed; then, the effect of the ground slope produced fire acceleration; finally, the direction of the wind changed and its intensity increased, resulting in more spaced lines with variation in their normal directions.

Figure 34 shows the temporal evolution of the fire width, height, and depth. The width varied between 2 and 5 m and was in accordance with the width of the ground and the evolution of the fire trajectory.

Figure 35 presents the temporal evolution of the maximum flame tilt angle. The curve presents two high tilt angle values, which reflect the strong flame inclinations that were observed during the experiment due to gusts of wind.

The fire considered in this example lasted 120 s and the images obtained in the first 116 s were exploited. The computation was performed using MATLAB 2019b${}^{\circledR }$ on a computer with an Intel(R) Core(TM) (Intel Corporation, Santa Clara, CA, USA) i7-8650U CPU @$1.9$ GHz processor and 16 GB of RAM memory. All results were obtained in 46 s (the time taken by the procedure to achieve the best pixel detection method was not counted in this duration).

4. Discussion

Wildfires are a phenomenon whose behavior depends on factors such as the fuel and its load, air humidity, wind, and the slope of the ground. Outdoor experiments cannot possibly reproduce actual fires and, so, obtaining ground truth data is extremely difficult. In this study, two experiments were designed to be as close as possible to ground truth. In the first, we used a parked car with known geometrical characteristics and a UAV flying 10–15 m around it. We found that the estimated dimensions and the position of the car were accurate, demonstrating the validity of the proposed model. The second experiment was conducted using a $3\times 5$ m area of vegetation, in which the fire did not propagate. The estimated position lines of the fire obtained over time were in line with the experimental situation.

The estimated data obtained during the monitoring of an experimental fire propagated on a $5\times 10$ m field were consistent with the expectations for this type of propagation; some data correlated to fuel platform dimensions or events such as wind. In future studies, we will use landmarks to provide position and height references for comparison with the measured characteristics.

In the framework presented in this paper, the images were stored on an SD card and processed offline. To use the system onsite in real time, works will be performed to equip the drone with an onboard computer and to reduce the execution time of the programs.

5. Conclusions

In this paper, we proposed a UAV multimodal stereovision framework to measure the geometric characteristics of experimental fires. It is composed of a vision device—integrating two multimodal cameras operating simultaneously in the LWIR and visible spectra—and a UAV. From georeferenced multimodal stereoscopic images, 3D fire points are obtained, from which fire geometric characteristics can be estimated. The framework is able to produce the temporal evolution of the geometric characteristics of a fire propagating over unlimited distance. This satisfies the need for a system that is able to estimate fire geometric characteristics, in order to understand and model wildfire behavior. In future work, the use of this system to monitor controlled burns will be explored.