1. Introduction
Airborne laser scanning, also termed airborne Light Detection and Ranging (LiDAR), is an active remote sensing technique for acquiring 3D geospatial data over the Earth’s surface [
1,
2]. A typical airborne LiDAR system consists of a GPS (Global Positioning System), an IMU (Inertial Measurement Unit), and a laser scanner, with which a point cloud dataset encoding 3D coordinate values under a given geographic coordinate system can be generated [
3]. The point cloud can be further processed to extract thematic information and geo-mapping products, such as manmade objects [
4], stand-alone plants [
5], DEM (Digital Elevation Model)/DTM (Digital Terrain Model) [
6], etc. However, there are still many challenges in terms of object detection, extraction, and reconstruction by using the LiDAR dataset alone, because the point cloud provided by a LiDAR system is unstructured, irregularly spaced, and lacks spectral and textural information. Thus, a commercial airborne LiDAR system usually integrates a high-resolution metric digital camera, from which high-resolution aerial images can be collected while collecting point cloud data. The individual characteristics of LiDAR point cloud and image data are considered complementary [
7]. They have been used to enhance the extraction of thematic information by fusing the two datasets for a variety of applications, such as buildings detection and reconstruction [
8,
9], land cover classification [
10,
11], road modeling [
12,
13], and tree species classification [
14,
15], to name but a few.
In photogrammetric applications, it is necessary to determine the geometric model of the sensing system before the collected images can be used for highly accurate measurement purposes. In traditional aerial photogrammetric mapping, the process begins with the determination of the IOEs (Interior Orientation Elements) and the EOEs (Exterior Orientation Elements) of the camera. IOEs are usually provided by the camera manufacturer [
16]. This means that IOEs can be viewed as known variables during the photogrammetric processing. EOEs can be processed in two steps (relative and absolute orientation), but simultaneous methods (such as bundle adjustments) are now available in the majority of software packages [
17]. A photogrammetric test field with highly redundant photo coverage such as 80% forward overlap and 60% side overlap and accurate ground control points (GCPs) are required in the simultaneous methods [
18,
19]. With the availability of the combination of GPS/IMU, direct georeferencing becomes possible because the EOEs can be derived from an integration of relative kinematic GPS positioning and IMU data by Kalman filtering, which is the case in an airborne LiDAR system integrated with a digital camera.
One of the prerequisites for direct georeferencing of images is the rigid connection between the camera and the IMU in order to keep a strict parallel between the image sensing frame and the IMU body frame, which is hard to achieve and may vary even within a given flight day [
20]. Moreover, as the origin of the camera frame cannot be coincident with the projection center of the camera, and the GPS antenna will be on the top of the aircraft, the attitude and positional relation between camera and IMU, known as boresight errors/misalignments, must be determined before direct georeferencing of images can be performed, which includes the determination of three rotational angles and three lever arms, as shown in
Figure 1. Level arms can be measured by traditional methods, such as direct measurement with ranging tools, close range photogrammetry [
21], and accuracy within one centimeter can be achieved [
22]. However, the measurements of the boresight misalignments are far more complicated compared to lever arm measurements because no direct methods exist. Conventionally, they are determined indirectly by using a reference block with known ground control points located within the project area or in a special test field, a process termed as system calibration, because it provides the calibration of other parameters such as focal length. Many research works have been conducted regarding direct georeferencing with the conventional method in the past two decades. Heier et al. [
23] showed the postprocessing steps of DMC (Digital Metric Camera) image data to generate virtual central perspective images and gave an overview of the entire DMC calibration. Skaloud et al. [
24] and Skaloud [
25] conducted a study on the method of GPS/IMU integration to provide exterior orientation elements for direct georeferencing of airborne imagery with more reliability and better accuracy. The operational aspects of airborne mapping with GPS/IMU were analyzed and strategies for minimizing the effect of the hardware integration errors on the process for direct georeferencing were proposed. Heipke et al. [
26] discussed the direct determination of the EOEs via the combination of GPS and IMU as a complete substitute for aerial triangulation. Jacobsen [
27] discussed the direct georeferencing based on restoring the geometric relations of images in a chosen object coordinate system, and the possibility of avoiding using control points by direct sensor orientation with the combination of GPS and IMU. Mostafa et al. [
28] argued that boresight misalignments calibration is one of the critical steps in direct georeferencing for geomapping purposes. They presented the experimental results of boresight misalignments calibration by using a software and checked the results with ground control points. Honkavaara [
29,
30] discussed block structures for calibration that significantly affected the cost and efficiency of the system calibration. The experiments indicated that boresight misalignments and the IOEs are the main factors influencing the final results. Jacobsen [
31] investigated the direct and integrated sensor orientation based on the combination of relative kinematic GPS and IMU. The investigation showed the advantages of using direct sensor orientation for image georeferencing without ground control points and independent of block or strip configurations. Filho et al. [
32] presented an in-flight calibration method for multi-head camera systems, and the applications of direct georeferencing were evaluated.
Though intensively adopted in practice, traditional system calibration shows the following drawbacks: Firstly, the environmental conditions such as temperature, humidity, etc. between the test field and mapping areas may dramatically differ. Therefore, the camera geometry during operation may also change relative to the situation in the test filed due to changes in environmental conditions [
33,
34]. Secondly, establishing a new test field for every mapping project and collecting large numbers of ground control points is expensive and sometimes impractical. On the other hand, airborne LiDAR systems deliver direct dense 3D measurements of object surface at a high rate of accuracy [
7,
16]. Moreover, continued improvements in the performance and accuracies of LiDAR systems in recent years have enabled the use of LiDAR data as a source of control information suitable for photogrammetric applications. Different methods have been tested and implemented for integrating LiDAR and photogrammetric data in performing aerial triangulation or determining the boresight misalignments for direct georeferencing, as will be reviewed in the following.
Delara et al. [
35] presented a method to perform the bundle block adjustment using aerial images and laser scanner data. In the method, LiDAR control points were extracted from LiDAR intensity images for determining the exterior orientation elements of a low-cost digital camera. Habib et al. [
36,
37] utilized linear features derived from LiDAR data as control information for image georeferencing. However, a large number of linear features with good spatial distribution are needed to achieve high accuracy. Kwak et al. [
38] proposed using the centroid of the plane roof surface of a building as control information for estimating exterior orientation elements of aerial imagery and registering the aerial imagery relative to the aerial LiDAR data. In the method, the centroid of the plane roof is extracted from aerial imagery by using the Canny Edge Detector and from aerial LiDAR data by using Local Maxima Filtering. Liu et al. [
39] presented a method for utilizing LiDAR intensity images to collect high accuracy ground coordinates of GCPs for aerial triangulation process. Yastikli et al. [
40] investigated the feasibility of using LiDAR data for in situ calibration of the digital camera. In addition, the determination of attitude and positional relationship between digital camera and IMU was also discussed. Mitishita et al. [
41] presented a method of georeferencing photogrammetric images using LiDAR data. The method applied the centroids of rectangular building roofs as control points in the photogrammetric procedure. Ding et al. [
42] utilized the vertical vanishing point in an aerial image and the corner points of the roof edge from the point cloud to estimate the pitch and roll of the cameras rotation angles. Based on Ding’s study, Wang and Neumann [
43] introduced a new feature 3CS (three connected segments) to replace the vanishing point to optimize the method. Each 3CS has three segments connected into a chain. Wildan et al. [
44] utilized control points derived from LiDAR data to perform the aerial triangulation of a large photogrammetric block of analogue aerial photographs. According to the authors, the mapping has achieved the national standard of cartographic accuracy for the 1:50,000 scale mapping. Chen et al. [
45] proposed a new method for boresight misalignments calibration of the digital camera integrated in an airborne LiDAR system without ground control points. In the calibration, tie points in overlapping areas are selected manually, and the ground points corresponding to these points are calculated using a multi-baseline space intersection and DEM elevation constraints. Gneeniss [
46] and Gneeniss et al. [
16] conducted studies on cross-calibrate aerial digital cameras via the use of complementary LiDAR data. The effect of the number and spatial distribution of LiDAR control points to perform aerial triangulation of large photogrammetric blocks was investigated as well.
Direct georeferencing of images based on LiDAR point cloud was also provided by commercial software Terrasolid in the TMatch module. In this module, a filter is used firstly to obtain ground points from point cloud, while a large number of tie points of the images are manually selected. The optimal camera misalignment values (new heading, roll, and pitch) are calculated using the tie points from the overlapping images and their corresponding ground points from the LiDAR point cloud. However, image points and LiDAR-derived ground points are matched manually, where artificial errors are inevitable, and matching is impractical when the surveying area is large.
The objective of this study is to introduce a new automatic boresight misalignments calibration method for direct georeferencing of images collected by a digital camera integrated in an airborne LiDAR system. Because the three lever arms can be accurately measured, we focus on the determination of the three rotational angles by using the LiDAR point cloud as auxiliary data. In contrast to the methods presented in previous literature, the in situ camera calibration focuses on using VCPs (Virtual Control Points—defined in following section) and a small sub-block of images selected from the entire block covering the surveying area. The method establishes the error equation by minimizing the distances between the initially selected tie points in the image space and the image points corresponding to VCPs by space resection. The main advantages of the method can be summarized as follows: Firstly, no dedicated calibration test fields, or even ground control points, are needed. Secondly, the whole procedure is fully automatic, from the extraction of tie points to the calibration of the boresight misalignments. This is of particular importance when the airborne LiDAR system is employed to collect data for rapid response to natural disasters, such as earthquake relief efforts. Finally, the accuracy of the georeferenced images is high enough for many geospatial applications, as shown in the experimental results.