^{1}

^{*}

^{2}

^{2}

^{1}

^{2}

^{1}

^{1}

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

A land-based mobile mapping system (MMS) is flexible and useful for the acquisition of road environment geospatial information. It integrates a set of imaging sensors and a position and orientation system (POS). The positioning quality of such systems is highly dependent on the accuracy of the utilized POS. This limitation is the major drawback due to the elevated cost associated with high-end GPS/INS units, particularly the inertial system. The potential accuracy of the direct sensor orientation depends on the architecture and quality of the GPS/INS integration process as well as the validity of the system calibration (

Roads are perhaps the most important infrastructures for people’s quality of life. It is used not only for land vehicle transportation, but also for providing the routes for power lines, sewer channels, water supplies, as well as TV and telephone cables. Thus, an efficient and accurate approach for the collection and updating of the road environment information is of extreme importance to the government and public sectors. Traditionally, the acquisition of geographic and attribute information about the road environment, such as traffic signs, road boundaries, sewer manholes, fire hydrants, advertisement boards, and building boundaries, are commonly performed by topographic mapping from large scale aerial photos and/or site surveying. Due to the visual limitations of aerial photos, the demand for site surveying, which is labor intensive and inefficient, is still quite high. Therefore, the development of land-based mobile mapping systems (MMS) has been the focus of several research groups in order to reduce the required manpower and cost while maintaining the necessary accuracy and reliability.

An overview of mobile mapping technology and its applications can be found in [

In order to fully explore the potential accuracy of such systems and guarantee accurate multi-sensor integration, a careful system calibration must be carried out [

The two-step procedure for the estimation of the mounting parameters relating the cameras and the IMU body frame is based on comparing the cameras’ exterior orientation parameters (EOPs), which are determined through a conventional bundle adjustment (indirect geo-referencing) procedure, with the GPS/INS derived position and orientation information of the platform at the moments of exposure. Similarly, the estimation of the ROPs among the cameras can be established by comparing the cameras’ EOPs determined through an indirect geo-referencing procedure. Although such procedures are easy to implement, its reliability is highly dependent on the imaging configuration as well as the number and distribution of tie and control points since these factors control the accuracy of the estimated EOPs.

The single-step procedure, on the other hand, incorporates the system mounting parameters and the POS-based information in the bundle adjustment procedure. The commonly used single-step approach to determine the system mounting parameters is based on the expansion of traditional bundle adjustment procedures with constraint equations [

In this paper, a novel single-step procedure, which is more suitable for multi-camera systems, is introduced. The proposed method utilizes the concept of modified collinearity equations, which has already been used by some authors in integrated sensor orientation (ISO) procedures involving single camera systems [

This paper starts by outlining the architecture of the designed medium-cost land-based MMS. Then, a discussion of the photogrammetric system calibration is presented. First, the procedure for calibrating the cameras is described followed by a discussion of the proposed mounting parameters calibration. Experimental results are presented next to test the feasibility of the proposed photogrammetric system calibration and to test the performance of the designed system. Finally, the paper presents some conclusions and recommendations for future work.

For the proposed land-based MMS, a reinforced aluminum frame is designed and fixed on top of a van (

Since a medium-cost land-based MMS is required in this research, a tactical grade MEMS GPS/INS integrated POS system is adopted (C-MIGITS© III from BEI SDID and a NovAtel© ProPak-V3 GPS receiver). The positional accuracy of such POS system after post-processing, in case of no GPS outage and using kinematic GPS data collection, is about 10 cm for the horizontal direction and 15 cm for the vertical, and the accuracy of the integrated GPS/INS attitude is 0.05° for the roll and pitch and 0.1° for the heading [

This research adopts DGPS positioning with a base station and a tightly-coupled scheme that integrates the IMU and GPS measurements to provide a seamless POS-based solution [

Three Basler Scouts and two AVT Stingray CCD digital cameras are installed in the proposed MMS. The specifications for these two types of cameras are similar; they both have a pixel size of 4.4 μm and an array dimension of 1,624 × 1,234 pixels. However, the used lenses have different focal lengths,

For a multi-sensor mobile mapping system, synchronization errors among the sensors will introduce significant position and attitude errors [

Direct sensor orientation can be performed in two different ways: (i) integrated sensor orientation (ISO) and (ii) direct geo-referencing [

There are several factors that might affect the performance of the direct sensor orientation. For instance, the quality of photogrammetric system calibration (

The purpose of camera calibration is to mathematically describe the internal geometry of the imaging system, particularly after the light ray passes through the camera’s perspective center. In order to determine such internal characteristics, a self-calibrating bundle adjustment with additional parameters is performed [

In _{O},Y_{O},Z_{O}) are the coordinates of the camera’s perspective center, (X_{A},Y_{A},Z_{A}) are the coordinates of the ground point (A), (x_{p},y_{p}) are the principal point coordinates, c is the camera’s principal distance, and (x_{a},y_{a}) are the image coordinates of (a). The camera’s attitude parameters (ω, ϕ, k) are embedded in the rotation matrix elements (r_{11}∼r_{33}) Finally, Δx and Δy are the image coordinates displacements introduced by the distortions. The mathematical model of the distortions is introduced in _{1},K,_{2}K_{3}), the de-centering lens distortion coefficients (P_{1},P_{2}), and in-plane (differential scale and non-orthogonality) distortion coefficients (b_{1},b_{2}). The out-of-plane (image plane un-flatness) distortion is not significant for digital cameras; thus, they are ignored in the adopted camera distortion model [_{a}–x_{p}), ȳ = (x_{a}–y_{p}), and
_{0}) value, which is a measure of the quality of fit between the observed image coordinates and the predicted image coordinates using the estimated parameters (_{0} is reduced significantly, for example more than 0.1 pixels—the expected accuracy of image coordinate measurement, the added parameter is considered significant. Otherwise, the added parameter can be ignored. The second approach is based on checking the correlation coefficient among the parameters and the ratio between the estimated value and its standard deviation (σ), namely significance index. If two additional parameters have high correlation coefficient, e.g., more than 0.9, then the one having the smallest significance index can be ignored. However, if the smallest significance index is larger than a pre-specified threshold, the parameter can still be considered significant. The threshold for the significance index is determined experimentally based on the results from the first approach.

As already mentioned, in multi-camera systems, the mounting parameters comprise two sets of ROPs: the ROPs among the cameras as well as the ROPs between the cameras and the navigation sensors. There are two main approaches for the determination of such parameters: two-step and single-step procedures. The proposed single-step procedure in this paper, which can be used for the estimation of the two sets of ROPs, and the traditional two-step procedures are explained in the following subsections.

The single-step estimation of the lever-arm offsets and boresight angles (

^{th}

^{th}

^{th}^{ci}^{ci}

By rearranging the terms in

P: is the

The ISO is implemented through a general Least Squares Adjustment (LSA) procedure, ^{th}^{th}^{th}^{th}^{th}^{th}

Another advantage of the proposed single-step procedure is the possibility of using the same implementation to enforce the Relative Orientation Constraints among the different cameras of a multi-camera system in an indirect geo-referencing procedure (^{th}

In summary, the mounting parameters relating the cameras to the IMU body frame can be directly estimated through the proposed ISO single-step procedure, which utilizes

The two-step procedure for estimating the lever-arm offsets and boresight angles of the cameras w.r.t. the IMU body frame is based on comparing the GPS/INS derived position and orientation (

Similarly, the ROPs of the cameras w.r.t. a reference camera can be determined by comparing the cameras EOPs (

An alternative two-step procedure for the estimation of the cameras’ mounting parameters w.r.t. the IMU body frame would be the utilization of the outcome from the indirect geo-referencing with ROC, introduced in the previous section, instead of the EOPs determined through the conventional indirect geo-referencing procedure. More specifically, this alternative procedure would compare the GPS/INS derived position and orientation with the EOPs of the reference camera (

It should be noted that the derived ROPs in

The mounting parameters for a GPS/INS-assisted multi-camera system refer to two groups of parameters: (1) the ROPs among the different cameras,

The estimation of the first group of ROPs can be established using either one of the following approaches:

Using the ISO in an indirect geo-referencing mode, which is denoted as indirect geo-referencing with ROC (10), one can directly derive an estimate of the ROPs among the cameras.

Using the conventional indirect geo-referencing, one can derive the EOP of the images captured by the different cameras. The derived EOPs are then used to derive time-dependent estimates of the ROPs according to the formulations in

On the other hand, the estimation of the second group of ROPs can be done using either one of the following approaches.

Using the ISO with prior GPS/INS position and orientation information—as explained in (5), one can directly derive an estimate of the mounting parameters relating the cameras to the IMU body frame.

Using the conventional indirect geo-referencing procedure, one can derive the EOPs of the images captured by the different cameras. The derived EOPs are then used to derive time-dependent estimates of the mounting parameters relating the IMU body frame and the different cameras according to the formulations in

Using the indirect geo-referencing procedure with ROC—as explained in

The experimental results section will provide a comparative analysis of the performance of these different mounting parameters calibration procedures.

In this study, the utilized CMigit-III IMU has an East-North-Up (ENU) local navigation coordinate system. The GPS/INS integrated position solution refers to the WGS84 latitude, longitude, and ellipsoidal height, while the orientation is provided as navigation angles, _{i}) at the corresponding location for a given time,

The rotation matrix relating the IMU body frame and the local navigation frame should be modified to express the rotational relationship between the IMU body frame and the photogrammetric local mapping frame (M-frame) [_{i}, λ_{i}) at a given time,

In _{i} and λ_{i} are the WGS84 latitude and longitude of the IMU body frame at a given time. Finally, the rotation matrix between the IMU body frame and the ECEF frame is modified to express the rotational relationship between the IMU body frame and the photogrammetric local mapping frame. In this work, the photogrammetric local mapping frame is defined as a topo-centric frame, denoted as N_{0}-frame, with its origin defined within the mapped area (ϕ_{0}, λ_{0}). Therefore, the rotation matrix from the IMU body frame to the photogrammetric local M-frame can be determined by

In a similar fashion, the ground coordinates of control points have to be transformed from the WGS84 longitude, latitude, and ellipsoidal height to the photogrammetric local mapping frame. After such transformation, the GPS/INS position and orientation information can be utilized together with the ground coordinates of the control points in the ISO procedure.

In this section, experimental results are presented to demonstrate the feasibility of the developed medium-cost land-based MMS and test the validity of the proposed photogrammetric system calibration. First, the camera calibration results are reported. Then, a comparative analysis between the two-step procedures and the proposed single-step procedure for the estimation of the mounting parameters is performed. The estimated camera and mounting parameters are incorporated in a direct geo-referencing procedure (space intersection) using an independent dataset to compare the different methods and evaluate the system performance.

The calibration process has been conducted according to the described method in Section 3.1. The five digital cameras AVT-0, AVT-1, Basler-2, Basler-3, and Basler-4 are calibrated independently. The average distance from the camera to the center of the round table is about 2.5 m. Each camera has a total of 20 images taken with around 80° convergence angle providing a strong imaging geometry. _{0} denotes the square root of the a-posteriori variance factor, which is a measure of the magnitude of the image residuals (_{0} values are quite acceptable and commensurate with the expected automated image-coordinate measurement accuracy using the retro-reflective targets. The relative accuracy in

The dataset used for the mounting parameters calibration was acquired over an established test field with 67 surveyed targets.

We can also observe in _{o})^{2} when comparing the single step procedure with either the traditional two-step procedure or the two-step procedure while enforcing the ROC.

The estimated lever-arm offsets and boresight angles relative to the IMU using the different calibration methods are then used in a direct geo-referencing procedure (space intersection) for an independent dataset (the nine remaining epochs of the acquired dataset) to perform a comparative analysis and to evaluate the performance of the designed system. The direct geo-referencing results (

In this paper, the implementation and accuracy analysis of a medium-cost land-based MMS have been demonstrated. The paper started by outlining the architecture of the proposed MMS. Then, a discussion of the photogrammetric system calibration was presented. First, the procedure for calibrating the cameras was described followed by a discussion of the mounting parameters calibration. A novel single-step procedure for mounting parameters calibration has been presented. The contributions of the proposed method can be summarized as follows: (i) The modified collinearity equations, which have been implemented in previous work for single camera systems only, is expanded in this research work to handle multi-camera systems; (ii) In contrast to the commonly-used additional constraints, the proposed method is much simpler,

Experimental results using real data have shown a significant improvement in the precision of the estimated mounting parameters (especially, the boresight angles) and the object space reconstruction (50 cm reduction in the RMSE values and bias improvement on each axis) when utilizing the proposed single-step procedure. Moreover, the proposed procedure has shown an improved estimation accuracy of the ROPs among the cameras when compared to the estimated ROPs from a two-step procedure. The single-step procedure provides more accurate results for the ROPs among the cameras due to the fact that the relative orientation constraint is explicitly enforced.

Future works will focus on more testing using simulated and real datasets to verify the performance of the proposed system/methods as well as investigating the optimum imaging and control configurations for reliable estimation of the mounting parameters. Also, future implementation will be extended to include previously estimated ROPs among the cameras as prior information when estimating the ROPs between the cameras and the IMU body frame in the developed single-step procedure. In other words, previously estimated relative orientation parameters among the cameras will be included as additional constraints during the single-step estimation of the mounting parameters relating the IMU body frame and involved cameras.

The authors would like to acknowledge the National Science Council of Taiwan (Project # NSC 97-2221-E-006-216), the Natural Sciences and Engineering Research Council of Canada (Discovery and Strategic Project Grants), and the Canadian GEOmatics for Informed DEisions (GEOIDE) NCE network (PIV-SII72 and PIV-17) for their financial support of this research work.

The proposed mobile mapping system.

Image acquisition scheme for camera calibration.

The distribution of acquired images, surveyed targets/control points (in blue), and tie points (in black) together with the intersecting light rays for some of the control points.

Quality analysis of the camera calibration.

_{0} |
0.15 | 0.16 | 0.11 | 0.12 | 0.11 |

Relative Accuracy | 1:17,600 | 1:14,400 | 1:24,100 | 1:22,900 | 1:25,800 |

Estimated ROPs between camera “0” (AVT-0) and the other cameras.

Two-step Indirect | Camera “1” (AVT-1) | 0.92777 ±285.5 | −0.38012 ±100.1 | −2.00209 ±85.2 | −0.03 ±0.01 | −1.47 ±0.01 | 0.06 ±0.01 |

Camera “2” (Basler-2) | −41.65608 ±144.7 | −0.05911 ±140.8 | −1.05843 ±198.3 | −0.02 ±0.01 | −1.49 ±0.01 | 0.62 ±0.01 | |

Camera “3” (Basler-3) | −88.95329 ±235.1 | 1.98176 ±237.6 | −0.69070 ±200.0 | −0.04 ±0.01 | −1.48 ±0.02 | 1.71 ±0.02 | |

Camera “4” (Basler-4) | −128.10177 ±321.9 | 0.52740 ±130.3 | −0.33972 ±85.9 | −0.06 ±0.01 | −1.48 ±0.01 | 2.47 ±0.01 | |

| |||||||

Single-Step Indirect Geo-ref. with ROC | Camera “1” (AVT-1) | 0.93444 ±14.6 | −0.40842 ±17.1 | −2.00061 ±20.0 | −0.03 ±0.0013 | −1.48 ±0.0019 | 0.06 ±0.0014 |

Camera “2” (Basler-2) | −41.66469 ±17.5 | −0.09493 ±23.3 | −1.06639 ±31.4 | −0.03 ±0.0017 | −1.50 ±0.0022 | 0.63 ±0.0024 | |

Camera “3” (Basler-3) | −88.91613 ±25.0 | 1.95771 ±43.2 | −0.69984 ±36.8 | −0.04 ±0.0021 | −1.49 ±0.0026 | 1.72 ±0.0031 | |

Camera “4” (Basler-4) | −128.10779 ±25.1 | 0.54875 ±52.1 | −0.32753 ±38.0 | −0.05 ±0.0021 | −1.48 ±0.0028 | 2.47 ±0.0035 |

Estimated lever-arm offsets and boresight angles between each camera and the IMU body frame, using different geo-referencing methods.

Two-step (Indirect Georef.) _{o})^{2}^{2} |
Camera “0” (AVT-0) | −0.97284 ±535.7 | −0.26904 ±4478.0 | 1.37450 ±5441.1 | 0.08 ±0.06 | 0.49 ±0.02 | −1.57 ±0.02 |

Camera “1” (AVT-1) | −0.03595 ±698.0 | −0.62728 ±4473.9 | −0.62241 ±5433.6 | 0.08 ±0.07 | −0.98 ±0.02 | −1.49 ±0.02 | |

Camera “2” (Basler-2) | −42.62160 ±600.5 | −1.17411 ±5287.6 | −0.21013 ±4784.4 | 0.09 ±0.06 | −0.99 ±0.03 | −0.93 ±0.02 | |

Camera “3” (Basler-3) | −89.92737 ±713.5 | 0.60325 ±5524.1 | −0.93508 ±4580.1 | 0.06 ±0.04 | −0.96 ±0.02 | 0.16 ±0.02 | |

Camera “4” (Basler-4) | −129.08182 ±805.1 | −0.38674 ±4761.7 | −1.40071 ±5167.7 | 0.04 ±0.04 | −0.95 ±0.02 | 0.92 ±0.02 | |

| |||||||

Two-step (Indirect Geo-ref. with ROC) _{0})^{2}^{2} |
Camera “0” (AVT-0) | −0.96123 ±658.8 | −0.27439 ±4502.9 | 1.38869 ±5430.5 | 0.07 ±0.06 | 0.49 ±0.02 | −1.57 ±0.02 |

Camera “1” (AVT-1) | −0.01722 ±641.3 | −0.65988 ±4491.6 | −0.60753 ±5440.2 | 0.08 ±0.07 | −0.98 ±0.02 | −1.49 ±0.02 | |

Camera “2” (Basler-2) | −42.61830 ±658.0 | −1.22283 ±5241.0 | −0.21120 ±4722.3 | 0.08 ±0.06 | −0.99 ±0.02 | −0.92 ±0.02 | |

Camera “3” (Basler-3) | −89.87900 ±766.2 | 0.56395 ±5439.1 | −0.94801 ±4492.6 | 0.06 ±0.04 | −0.97 ±0.02 | 0.17 ±0.02 | |

Camera “4” (Basler-4) | −129.07587 ±762.3 | −0.37444 ±4787.0 | −1.40052 ±5181.4 | 0.04 ±0.04 | −0.95 ±0.02 | 0.92 ±0.02 | |

| |||||||

Single-step (ISO) _{0})^{2}^{2} |
Camera “0” (AVT-0) | −0.90343 ±454.4 | 0.05174 ±125.7 | 1.28972 ±119.1 | 0.07 ±0.12 | 0.50 ±0.10 | −1.55 ±0.10 |

Camera “1” (AVT-1) | 0.06634 ±454.7 | −0.31522 ±125.1 | −0.70938 ±120.3 | 0.08 ±0.12 | −0.98 ±0.10 | −1.48 ±0.10 | |

Camera “2” (Basler-2) | −42.53492 ±454.7 | −0.92732 ±128.6 | 0.00765 ±119.3 | 0.08 ±0.12 | −0.99 ±0.10 | −0.92 ±0.10 | |

Camera “3” (Basler-3) | −89.83526 ±455.9 | 0.55968 ±131.7 | −0.53241 ±117.4 | 0.06 ±0.12 | −0.96 ±0.10 | 0.17 ±0.10 | |

Camera “4” (Basler-4) | −129.0088 ±456.0 | −0.64709 ±129.3 | −1.07301 ±119.4 | 0.05 ±0.12 | −0.94 ±0.10 | 0.93 ±0.10 |

Direct geo-referencing RMSE analysis. (unit: m).

Two-step Indirect | 0.71 |
0.68 |
1.32 |
1.65 |

Two step Indirect with ROC | 0.73 |
0.70 |
1.36 |
1.70 |

Single-step ISO | 0.47 |
0.60 |
0.80 |
1.10 |