^{1}

^{*}

^{1}

^{1}

^{1}

^{†}

^{1}

^{2}

Current address: Centro de Ciências Exatas e Tecnologia (CCET), Universidade Federal de Mato Grosso do Sul (UFMS), Campus Universitário, Campo Grande-MS, 79070-900, Brazil

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (

Image acquisition systems based on multi-head arrangement of digital cameras are attractive alternatives enabling a larger imaging area when compared to a single frame camera. The calibration of this kind of system can be performed in several steps or by using simultaneous bundle adjustment with relative orientation stability constraints. The paper will address the details of the steps of the proposed approach for system calibration, image rectification, registration and fusion. Experiments with terrestrial and aerial images acquired with two Fuji FinePix S3Pro cameras were performed. The experiments focused on the assessment of the results of self-calibrating bundle adjustment with and without relative orientation constraints and the effects to the registration and fusion when generating virtual images. The experiments have shown that the images can be accurately rectified and registered with the proposed approach, achieving residuals smaller than one pixel.

Professional digital cameras have a favorable cost/benefit ratio when compared to high-end digital photogrammetric cameras and are also much more flexible for use on different platforms and aircrafts. As a consequence, some companies are using professional medium format cameras in mapping projects, mainly in developing countries [

One alternative to augment the coverage area is using two (or more) synchronized oblique cameras. The simultaneously acquired images from the multiple heads can be processed as oblique strips [

Airborne Remote Sensing technology also uses similar methods to generate multispectral images from multiple cameras. Hunt

Ritchie

Chao

Schoonmaker

Hakala

Laliberte

One study on the use of images taken with a single RGB camera onboard of an UAV for monitoring soil erosion was presented by D’Oleire-Oltmanns

Grenzdörffer

Yang [

Holtkamp and Goshtasby [

Existing techniques for rigorous determination of images parameters aiming at virtual image generation from multiple frames have several steps, requiring laboratory calibration and direct measurement of perspective center coordinates of each camera and the indirect determination of the mounting angles using a bundle block adjustment [

An alternative is the simultaneous calibration of two or more cameras using self-calibrating bundle adjustment imposing additional constrains. Assuming that the relative position of the cameras is stable during the image acquisition mission it is possible to consider the constraints that the relative rotation matrix and the base components between the cameras heads are stable. The main advantage of this approach is that it can be achieved with an ordinary terrestrial calibration field and all parameters are simultaneously determined, avoiding specialized direct measurements.

The approach proposed in this paper is to generate larger virtual images from dual head cameras following four main steps: (1) dual head system calibration with Relative Orientation Parameters (ROP) constraints; (2) image rectification; (3) image registration; and (4) fusion and brightness adjustment to generate a virtual image.

Integrating medium format cameras to produce high-resolution multispectral images is a recognized trend, with several well known systems which adopted distinct approaches [

The generation of the virtual image from oblique frames can be done in a sequential process, with several steps, as presented by Zeitler ^{®} Digital Mapping Camera (DMC^{®}), which has four panchromatic and four multispectral heads. The first step is the laboratory geometric and radiometric calibration of each camera head individually. The positions of each camera perspective centers within the cone are directly measured, but the mounting angles cannot be measured with the required accuracy. These mounting angles are estimated in a bundle adjustment step, known as platform calibration. This bundle adjustment uses tie points extracted in the overlapping areas of the four panchromatic images by image matching techniques and with the IOP of each head being determined in the laboratory calibration. Transformation parameters are then computed to map from each single image to the virtual image and these images are projected to generate a panchromatic virtual image. Finally the four multispectral images are fused with the high resolution virtual panchromatic image. This process is accurate but requires laboratory facilities to perform the first steps.

Depending on the camera configuration and accuracy requirements of the application, the process of virtual image generation can be based on sets of two dimensional projective transformations, as presented by Holtkamp and Goshtasby [

The approach presented in this paper is another option, and it is based on the parameters estimated in a bundle adjustment with relative orientation constraints [

Camera calibration aims to determine a set of IOP (usually, focal length, principal point coordinates and lens distortion coefficients) [_{f}_{f}_{ij}_{0}, _{0}, _{0} are the coordinates of the camera perspective center (PC); _{0}, _{0} are the principal point coordinates; _{i}_{i}

Using this method, the exterior orientation parameters (EOP), IOP and object coordinates of photogrammetric points are simultaneously estimated by the LSM from image observations and using certain additional constraints. Self-calibrating bundle adjustment, which requires at least seven constraints to define the object reference frame, can also be used without any control points [_{0}) are not separable and the system becomes singular or ill-conditioned. In addition to these correlations, the coordinates of the principal point are highly correlated with the perspective center coordinates (_{0} and _{0}; _{0} and _{0}). To cope with these dependencies, several methods have been proposed, such as the mixed range method [

Stereo or multi-head calibration usually involve a two-step calibration: in the first step, the IOP are determined; in a second step, the EOP of pairs are indirectly computed by bundle adjustment, and finally, the ROP are derived [

Several previous papers on the topic of stereo camera system calibration considered the use of relative orientation constraints. He _{lm}^{(i)} being the elements of the relative rotation matrix for an image pair (_{lm}^{(k)} the relative rotation matrix for an image pair (

King [

El-Sheimy [

Tommaselli

Blázquez and Colomina [

Tommaselli

The basic mathematical models for calibration of the dual-head system are the collinearity equations (

The Relative Orientation (RO) matrix can be calculated as a function of the rotation matrix of both cameras by using:
_{RO}_{C1}_{C2}

Considering _{RO}^{(t)} as the ^{2(t)} as the squared distance between the cameras perspective centers, for the instant _{RO}^{(t+1)} and ^{2(t+1)}, it can be assumed that the RO matrix and the distance between the perspective centers are stable, but admitting some random variations. Based on these assumptions, the following equations can be written:

Considering _{ic}

The base component elements, relative to camera 1, can be derived from the EOP with

The base components can also be considered stable during the acquisition, leading to three equations that can be used as the Base Components Stability Constraints (BCSC), instead of just one base length equation (

Thus, for two pairs of images collected at consecutive stations, the RO constraints can be written using Equations (

The mathematical models corresponding to the self-calibrating bundle adjustment and the mentioned constraints were implemented in C/C++ language on the CMC (Calibration of Multiple Cameras) program, that uses the Least Squares combined model with constraints [

Collinearity equations combine adjusted observations (_{a}_{a}_{a}_{a}

Assuming that the additional constraints equations take the form _{c}^{a}_{c}_{a}^{a}_{c}_{c}_{c}_{c}

The vector of corrections _{0}, which is updated iteratively, is given by:
^{T}^{T}^{−1}^{T}^{T}^{−1}_{C}^{T}_{C}Q_{C}B_{C}^{T}^{−1}_{C}^{T}_{C}Q_{C}B_{C}^{T}^{−1}_{C}_{C}

_{C}

If the value of a parameter is known with a certain accuracy, for example, the coordinates of control points, then a weighted constraint can be applied similarly with Equations (

The approach used in this paper to generate larger images from dual head oblique cameras follows four main steps as previously presented in [

Self-calibrating bundle adjustment is performed with a minimum set of seven constraints, which are defined by the coordinates of three neighbor points and the RO constraints. The distance between two of these points must be accurately measured to define the scale of the photogrammetric network. After estimating the IOP, EOP and object coordinates of all photogrammetric points, a quality check is performed with distances between these points. This approach eliminates the need for accurate surveying of control points, which is difficult to achieve with the required accuracy. In the proposed approach the IOP estimated were the camera focal length, coordinates of the principal point and radial and decentering distortion parameters (Conrady-Brown model). Since that the affine distortion for the cameras used in this work is not significant, these parameters were not considered.

The second step requires the rectification of the images with respect to a common reference system, using the EOP and the IOP computed in the calibration step. The derivation of the EOP to be used for rectification was done empirically using the ground data calibration. From the existing pairs of EOP one was selected because the resulting fused image was near parallel to the calibration field.

Firstly, the dimensions and the corners of the rectified image are defined, by using the inverse collinearity equations. Then, the pixel size is defined and the relations of the rectified image with the tilted image are computed with the collinearity equations. The RGB values of each pixel on the rectified image are interpolated in the projected position in the tilted image. The value used for the projection plane is the focal length of camera 1 (

The third step is the registration of the rectified images using tie points located in the overlap area with subpixel precision using area based matching, refined with Least Squares Matching (LSM). Ideally, the coordinates of these points should be the same, but owing to different camera locations and uncertainties in the EOP and IOP, discrepancies are unavoidable. The average values of discrepancies can be introduced as translations in row and columns to generate a virtual image by resampling the rectified right image (step 4). The standard deviations of these discrepancies can be used to assess the overall quality of the fusion process and standard deviations smaller than 1–2 pixels can be obtained without significant discrepancies in the seam-line.

When the standard deviations of the discrepancies in tie points coordinates are higher than a predefined threshold (e.g., 2 pixels) a scale factor can be computed from two corresponding tie points in the limits of the overlap area. This scale factor is used to compute a new projection plane distance and the right image is rectified again. The registration process is repeated to compute new discrepancies in the tie points coordinates and to check their standard deviations.

The fourth step is the images fusion, when virtual images are generated (

Two Fuji FinePix S3Pro RGB cameras, with a nominal focal length of 28 mm, were used in the experiments (

Firstly, the system was calibrated in a terrestrial test field consisting of a wall with signalized targets (

Forty images, from five distinct camera stations, were acquired and analyzed. The image coordinates of circular targets in the selected images were extracted with subpixel accuracy using an interactive tool that computes the centroid after automatic threshold estimation. In each exposure station, eight images were captured (four for each camera), with the dual-mount rotated by 0°, 90°, 270° and 180°. After eliminating images with weak point distribution, 37 images were used: 18 images taken with camera 1 and 19 with camera 2. From this set of images acquired by the dual system, 8 image pairs were acquired at the same instant. For these 8 pairs the RO constraints equations were introduced in the system of linearized equations. The remaining images were treated as isolated frames and their EOPs were also computed in the solution along with the IOP of cameras 1 and 2 and ground coordinates of photogrammetric points.

In the experiments reported in previous papers [

A further set of distances (131) between signalized targets (

To assess the proposed methodology with real data, seven experiments were carried out, without and with different weights for the RO constraints. The experiments were carried out with RRMSC (Relative Rotation Matrix Stability Constraints—

In _{0} and _{0}) for both cameras are presented for each experiment. It can be seen that the similar estimated standard deviations were achieved in all experiments, except when the cameras were calibrated independently (Experiment A1).

The base components were then computed from the estimated EOP (

The relative rotation matrices for the same 6 image pairs for all experiments were also computed with

The second part of the experiments were performed with aerial images taken with the same dual cameras arrangement with flying height of 1,520 m and a GSD (Ground Sample Distance) of 24 cm (see

Firstly, image pairs were rectified using those IOP estimated in the self-calibration process with terrestrial data, for each group of experiments. Then, tie points were located in the overlap area of pairs of the rectified images using area based correspondence methods (minimum of 20 points for each pair). The average values of discrepancies and their standard deviations are then computed for each images pair. In _{c}) and 1 pixels in rows (σ_{r}). It is possible to note that the matching of the rectified image pairs when using parameters generated by self-calibration with RO constraints is better, mainly in experiments C and D (in which angular variations of 1″ and 10″ were considered, respectively). The effects of varying the weight in the base components constraints were not assessed in these experiments.

The distances between tie points in the overlap area were used to compute the scale factor and also to generated new rectified images for the right camera. The process of measuring tie points and compute discrepancies and their standard deviations was repeated and the average values of the standard deviations are shown in

Virtual images generated for all experiments were then used in a bundle block adjustment and the results were assessed with independent check points. From the set of virtual images generated, three images were selected (

The RMSE in check points coordinates were around 1 GSD (X and Y) and 2 GSD (Z) for the experiments D and G. The values for the RMSE in Z were higher in the other experiments (around 3 GSD). In general, it can be concluded that the proposed process works successfully, achieving results similar to a conventional frame camera with a single sensor.

In this paper, a set of techniques for dual head camera calibration and virtual images generation were presented and experimentally assessed. Experiments were performed with Fuji FinePix S3Pro RGB cameras. The experiments have shown that the images can be accurately rectified and registered with the proposed approach with residuals smaller than one pixel, and they can be used for photogrammetric projects. The calibration step was assessed with distinct strategies, without and with constraints considering the stability of Relative Orientation between cameras. In comparison with the approach presented in previous papers, some improvements related to the constraints in the base components, rather than on the base length constraint and also the use of self-calibration with independent distances check were introduced. An infrared channel can also be registered to the image, by using a third camera, as was shown in [

The advantage of the proposed approach is that an ordinary calibration field can be used and no specialized facilities are required. The same approach can be used in other applications, like generation of panoramas, a suggestion that can be assessed in future work. Additionally, the proposed approach is suitable to be used with lightweight cameras installed in UAVs, improving the area covered in each shot, and also enabling the integration of other cameras with additional spectral channels, that can be important and useful in several airborne remote sensing applications.

The focus of this work was the assessment of the influence of the RO constraints for the virtual image generation. However, it is important to mention that some further implementations are required to achieve a fully automatic processing chain for operational purposes.

Another suggestion for future work is the comparison of the proposed technique with the modified collinearity equations in which the EOPs of one camera are replaced by a function of the ROP and the EOP of the master camera.

The authors would like to acknowledge the support of FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) with Grant No. 07/58040-7. The authors are also thankful to CNPq for supporting the project with Grants 472322/04-4, 481047/04-2, 478782/09-8 and 305111/10-8.

Resulting rectified images of dual cameras: (

Dual head system with two Fuji S3 Pro cameras.

(

Root Mean Squared Error (RMSE) of the check distances.

Estimated standard deviations of _{0} and _{0} for both cameras.

Standard deviations of the computed base components.

Standard deviations of rotation elements of the Relative Rotation matrix computed from estimated exterior orientation parameters (EOP).

(

Average values for the standard deviations of discrepancies in tie points coordinates of 5 rectified image pairs with different sets of Interior Orientation Parameters (IOP) and Relative Orientation Parameters (ROP).

Average values for the standard deviations of discrepancies in tie points coordinates of five rectified image pairs with different sets of IOP and ROP, after scale change in the right image.

Distribution of ground control Points (triangles) and check points (circles) in the experiment with bundle adjustment.

RMSE in the check points coordinates obtained in a bundle adjustment with three virtual images generated with parameters obtained in the experiments: (

Technical details of the camera used.

Sensor | CCD − 23.0 × 15.5 mm |

Number of pixels | 4,256 × 2,848 (12 MP) |

Pixel size (mm) | 0.0054 |

Focal length (mm) | 28.4 |

Characteristics of the seven experiments with real data.

RO Constraints | Single camera calib. | N | Y | Y | Y | Y | Y |

Variation of the RO angular elements | - | - | 1″ | 10″ | 15″ | 30″ | 1′ |

Variation of the base components (mm) | - | - | 1 | 1 | 1 | 1 | 1 |