This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (
Realtime image georeferencing is essential to the prompt generation of spatial information such as orthoimages from the image sequence acquired by an airborne multisensor system. It is mostly based on direct georeferencing using a GPS/INS system, but its accuracy is limited by the quality of the GPS/INS data. More accurate results can be acquired using traditional aerial triangulation (AT) combined with GPS/INS data, which can be performed only as a postprocessing method due to intense computational requirements. In this study, we propose a sequential AT algorithm that can produce accurate results comparable to those from the simultaneous AT algorithm in real time. Whenever a new image is added, the proposed algorithm rapidly performs AT with minimal computation at the current stage using the computational results from the previous stage. The experimental results show that the georeferencing of an image sequence at any stage took less than 0.1 s and its accuracy was determined within ± 5 cm on the estimated ground points, which is comparable to the results of simultaneous AT. This algorithm may be used for applications requiring realtime image georeferencing such as disaster monitoring and imagebased navigation.
Realtime acquisition of spatial data such as DSMs (digital surface models) or orthoimages is needed to provide appropriate and prompt countermeasures to situations such as natural disasters or accidents. For example, by monitoring the areas of forest fires and floods using the spatial data acquired by an airborne multisensor system, we can observe the situation, assess ongoing damage, effectively decide how to evacuate people and restore damaged areas.
Disaster monitoring systems based on airborne realtime acquisition of spatial data mostly consist of an aerial segment for data acquisition of the target areas and a ground segment for data processing and delivery [
The orthorectification requires a DSM over the target areas and the intrinsic and extrinsic parameters of the camera. In a realtime situation, the DSM can be promptly generated from the laser scanner data. The intrinsic parameters can be determined during camera calibration on the ground before the flight and are assumed to be constant during the flight. Most problems occur when determining the extrinsic parameters in real time, which requires knowing the position and attitude of the camera at the time of exposure for each image. This process is referred to as image georeferencing in this paper, which attempts to establish the geometric relationship between an image or image sequences and the absolute ground coordinate system.
Image georeferencing in both photogrammetry and computer vision fields may be categorized into two groups,
An existing promising method to determine camera parameters is aerial triangulation (AT) based on the bundle block adjustment [
The three sequential estimation algorithms used to determine the camera parameters that are mostly used in photogrammetry or computer vision fields are: (1) the Kalman filter, which updates the inverse of the normal matrix [
Although the sequential algorithm based on Givens transformations is efficient for estimating the unknown camera parameters, it cannot provide the variancecovariance matrix of the estimates without reverse computation. This is because it changes the structure of the normal matrix by application of the Givens transformation. However, the variancecovariance matrix should be produced in real time so that we can compute the correlation of a new image with the previous images in the sequence. With any efficient sequential algorithm, the processing time at least gradually increases with the number of images. To limit the processing time to a certain threshold such as the image acquisition period, we should discard some images that are significantly less correlated with the new image. In this study, we thus attempt to develop a new sequential aerial triangulation algorithm that can estimate and update not only the camera parameters but also their variancecovariance matrix in real time whenever a new image is added to an image sequence. Using this algorithm, we can update the inverse of the normal matrix with minimal computation while maintaining the original structure of the normal matrix. In this paper, we introduce the new sequential aerial triangulation algorithm, summarize the experimental results, and present some conclusions and future research.
To establish realtime image georeferencing, all inputs such as controlled information and tie points between adjacent images must be determined in real time whenever an image is newly captured. The controlled information about ground truth is acquired by surveying ground control points or from the GPS/INS sensors. Acquisition of the ground control points requires laborintensive operations that preclude realtime processing. Therefore, we assume that image georeferencing is performed without ground control features in which only the initial camera parameters are provided from the GPS/INS sensor. It is also assumed that a sufficient number of tie points among the successive images are collected from a robust realtime image matching algorithm [
Under these assumptions, we have to perform aerial triangulation in real time for realtime image georeferencing. Aerial triangulation is a sequential estimation problem where we have to update existing parameters whenever new observations and even new parameters are added. Therefore, we propose a sequential aerial triangulation approach in which we will not only estimate the camera parameters of the new images, but also update those of the existing images whenever a new image is added.
During realtime image acquisition, a new image is continuously being added into an image sequence. The addition of an image involves the addition of the following new parameters: six extrinsic parameters of the camera, also called the exterior orientation parameters (EOP), and 3
For the sequential AT approach, we classify the observations into three types,
In the initial stage, a small number of images are acquired, and the traditional simultaneous aerial triangulation algorithms based on bundle block adjustment is applied. Here, the EOP of the images and the coordinates of ground points (GP) corresponding to the tie points are the parameters to be estimated. The collinearity equations for all the tie points are used as the observation equations. The initial EOP provided from the GPS/INS system is used for stochastic constraints. The observation equations with the stochastic constraints can be expressed as
The observation equations can be rewritten as
By applying the least squares principle, we can derive the normal equations as follows:
With the subblock representations of the normal matrix and the right side, the normal equation can be rewritten as
The inverse of the normal matrix is then represented as
For this computation, we need to calculate the inverses of the matrices,
With the results of the initial stage, we can progress toward the combined stage. At the combined stage, we have a set of new images and the newly identified ground points corresponding to the tie points either in new images only or in new and existing images together. The parameter vectors for the EOP of the new images and the newly identified ground point coordinates are denoted as
The new observation equations in
The normal equations resulting from the application of the least squares principle to the observation equations are expressed as
During an estimation process based on the least squares principle, the most timeconsuming step is the computation of the inverse of the normal matrix. Since the size of the normal matrix is the sum of the number of existing and new parameters (
The inverse of the normal matrix can be written as
The main component in the inverse of the normal matrix is
Since we already computed
Such a situation can be relieved by using an efficient sequential update formula derived from
The number of parameters to be updated increases linearly whenever a new image is added into an image sequence, although we use the computation results from the previous stage to efficiently compute the inverse of the normal matrix at the current stage. If we have to acquire an image sequence for a long time, it would be impossible to perform AT in real time due to the extremely large size of the parameter vector. We thus need to maintain a constant parameter vector size for realtime processing. For this purpose, we may update the parameters associated with a certain number of recent images, for example, the latest 50 images. However, such a constant number is set by intuition; it does not originate from the underlying principle of the estimation process. In this study, the correlation between parameters is employed to reasonably limit the size of the parameter vector. It is obvious that some images in the beginning of the sequence must have almost no correlation with a new image if image acquisition continues for a long time. Therefore, we do not update the parameters associated with the images in the beginning of the sequence in such a case. To exclude parameters associated with the images in the beginning of the sequence, we determine the correlation coefficient between the parameters of the current stage and those of the previous stage. If the correlation coefficients between the previous parameters and current parameters are larger than a threshold, only the previous parameters should be updated.
The correlation coefficient is an efficient measure of the correlation between two variables. In general, the correlation coefficient between two variables can be computed using the covariance between the two variables and the standard deviation of each variable. A variancecovariance matrix, derived from an adjustment computation, consists of the variance of a parameter and the covariance between parameters. Our sequential AT process not only estimates the parameters, EOP and GP, but also offers a cofactor matrix (Q) including the variance of the estimated EOP/GP and the covariance between the estimated EOP/GP at every stage. In our proposed algorithm, as shown in
We have two classes of parameters,
To facilitate the efficient implementation of this algorithm, there is no further examination of the correlation coefficient between the EOPs of the next images and the EOPs of the last image after an image highly correlated with the last image is detected from the beginning of the sequence. For example, if the correlation coefficient between the EOPs of the third image and those of the last image is more than the threshold for the first time, all of the EOPs from the third image to the last image should be updated, even though the correlation coefficient of the EOPs for the fifth and the last image is less than the threshold (
After the images associated with the EOPs included in the adjustment are selected, GPs appearing only in the images associated with EOPs excluded from the adjustment also have to be eliminated in the adjustment computation. Some GPs may appear only once in the images associated with the EOPs included in the adjustment. They should also be deleted from the adjustment computation because they can no longer be considered to be derived from tie points. As shown in

Calculate the correlation coefficient between EOPs of a previous image and the new image. 

Exclude the EOPs of a previous image from the parameter vector. 

Count the number of images that appear per previous GP 

Include the GP in the parameter vector 
Experiments were conducted with simulated data sets to evaluate our proposed AT algorithm. We implemented three types of aerial triangulation processes, the sequential AT with highly correlated images (SeqR), the sequential AT with full images (SeqF), and the simultaneous AT (Sim) using Matlab (ver. R2008b), and tested them in the computing environment described in
Emergency monitoring will definitely benefit from realtime image georeferencing technologies. In an emergency situation, we need a feasible platform such as an unmanned aerial vehicle (UAV). One of the advantages of utilizing an UAV is that it can fly over dangerous regions without a human operator. In addition, it enables us to acquire sensory data with high spatial resolution since it can fly at relatively low altitude. Thus, we assumed a multisensor system based on a closerange UAV to simulate experimental data. We assumed that the system is equipped with a mediumformat digital camera and a mediumgrade GPS/INS. These sensors were carefully selected among the actual sensors available in current markets and their specifications were used for the simulation parameters. Under reasonable assumptions for flight and sensor parameters and a terrain model, we determined the attitude and position of the camera at the time of exposure of each image and the tie points in the adjacent images. The main simulation parameters are summarized in
The simulation procedure is shown in detail in
There were a total of 5,812 simulated images points and 304 ground points, and an average of 14.4 conjugate points between adjacent images. The overlap ratio between two subsequent images was almost 94%. One ground point appeared in 20 images on average except at the fringes of the project area. The results of the simulation are summarized in
By applying the simulated data to the three different AT processes, we can estimate the EOPs and GPs whenever a new image is acquired. Since the input data are a simulated set, we know the true values for all the unknowns and compute the RMSE by comparing them with the estimated ones.
The red, blue and pink lines represent the RMSE of the estimates from the sequential AT with only highly correlated images (SeqR), the sequential AT with full images (SeqF), and the simultaneous AT (Sim), respectively. The results from all types of AT are evidently better than the initial approximation the results from direct georeferencing. In addition, the results from each AT method are very similar. We see that the red, blue and pink lines almost coincide. The RMSEs of the estimated EOPs from each method are about 0.18 m and 0.05°. This is nearly a 50% improvement in accuracy compared with the results of direct georeferencing. Indirect georeferencing technologies based on aerial triangulation can compensate for position/attitude sensor performance such as in GPS/INS.
To compare the results from these methods, we compute the standard deviation of the differences between the ground point coordinates estimated using the three different methods, as shown in
The goal of sequential AT is to determine the EOP of all the images in almost real time whenever a new image is acquired. Therefore, it is necessary to prove that sequential AT is more efficient than simultaneous AT in terms of processing time.
The processing time of simultaneous AT increases proportionally to the square of the number of images as shown in
The processing time of sequential AT with full images also increases slightly with the number of images as shown in
Sequential AT with highly correlated images can be useful for realtime processing for a long image sequence. In such a sequence, it is obvious that some images in the beginning of the sequence must have almost no correlation with a new image. Hence, we choose not to update the parameters associated with the images in the beginning of the sequence. Here, the question is how many images in the beginning of the sequence can be safely excluded from the adjustment. Instead of setting a particular number of images to be excluded, we adaptively select the images that would have an almost negligible impact on the adjustment by determining their correlation with the new image. Such images with EOPs that have a low correlation coefficient with the EOPs of the new image can be reasonably excluded from the adjustment.
In the experiment, we exclude the images with correlation coefficients less than 0.1. With this threshold, the size of the parameter vector is maintained at about 400 (
We acquired the real data using an airborne multisensory system composed of an UAV and sensors such as a digital camera, a laser scanner, and GPS/INS. The main specifications of the sensors are summarized in
We obtained 113 images with a GSD (Ground Sampling Distance) of 3 cm and performed automatic image matching using a commercial digital photogrammetric workstation to generate conjugate points, the input data for AT. A total of 1,488 conjugate points corresponding to 304 ground points are produced with an average of 13 conjugate points between adjacent images. The coverage of the images and the distribution of the ground points are shown in
We applied our sequential AT algorithms to the real data and analyzed the results in terms of the accuracy and processing time. To verify the accuracy, we compared the estimates from our AT methods (SeqR and SeqF) with those from the conventional AT method (Sim). Since we do not know the true values for the unknowns in the experiments using real data, we used the Sim results as the reference data instead. The Sim results have been recognized as the most accurate, although Sim cannot be performed in real time.
The AT results are verified in terms of EOP positions, EOP attitudes, and GPs. The RMS values of the differences between three AT results are shown in
The processing times of each AT method according to the number of images being acquired are shown in
This research proposes a new sequential AT algorithm to perform realtime georeferencing of image sequences acquired by an airborne multisensor system. Although the traditional AT algorithm can produce very accurate results, it cannot be employed for realtime georeferencing due to its computation time, which dramatically increases with the number of images. The proposed sequential AT algorithm can produce accurate results comparable to those from the simultaneous AT with a computation time maintained within a constant time frame. This algorithm can be controlled such that it has a computation time shorter than the image acquisition time, which supports realtime georeferencing. Rapid computation is possible since only the minimum computation at the current stage is performed, using the computational results from the previous stage whenever a new image is added. Moreover, the exclusion of an image based on the correlation between the existing and new parameters in the algorithm can minimize the processing time.
The accuracy and processing speed were verified by applying this algorithm to a simulated data set. The experimental results show that the georeferencing of an image sequence is possible in less than 0.1 s whenever a new image is acquired every 0.5 s. The accuracy of the sequential AT results is comparable to that from the simultaneous AT results, where the differences between both results are very small, within ±3 cm in terms of the ground point coordinates. In addition, the proposed algorithm was applied to a real data set acquired by an airborne multisensory system and the results confirm that it works efficiently with the real data as well as the simulated data.
Consequently, it is expected that our sequential AT algorithm can be effectively employed for various applications requiring realtime image georeferencing such as disaster monitoring and imagebased navigation. In the near future, this sequential AT algorithm will be integrated with a realtime image matching algorithm for realtime image georeferencing. Finally, this approach will be applied to a variety of real data sets and also verified with respect to accuracy and processing speed.
Realtime image georeferencing.
Exclusion process of the previous parameters.
Simulation procedures.
RMSE of estimated EOP (position).
RMSE of estimated EOP (attitude).
RMSE of estimated ground point coordinates.
Standard deviation of estimated ground point coordinates.
Processing time of the three AT methods.
Dimension of the inverse matrices to be computed.
Processing time of the three AT methods (zoomed in).
Size of parameter vectors.
Test site and flight trajectory.
Airborne multisensory system during data acquisition over the test site.
Image coverage and ground points.
RMS values of EOP differences (position).
RMS values of EOP differences (attitude).
RMS values of GP differences.
Processing time of each AT method.
Implementation and test environments.
Operating System  Microsoft Window XP SP3 
CPU  Intel(R) Core(TM)2 Duo CPU 
E7500 @ 2.93 GHz, 2.93 GHz  
RAM  3.00 GB 
Main simulation parameters.
Flight configuration  height  200  m 
speed  10  m/s  
no. strips  1  
length of a strip  2,000  m  
Camera  focal length  17  mm 
pixel size  3.45 × 3.45  
detector dimensions  2,456 × 2,058  pixels  
frame rate  2  images/s  
GPS/INS  position error  0.3  m 
attitude error  0.1  degree 
Simulation results.
Images per strip  384 
Ground points  304 
Image points  5,812 
Image points per ground point  20 
Tie points between adjacent images  14 
Ground Sampling Distance (GSD)  3 cm 
Main specifications of the sensors.
Camera  medium format 
effective pixels : 4,872 × 3,248  
focal length : 50 mm  
frame rate : 1 fps  
 
GPS/INS  tactical grade 
position accuracy : 0.3 m  
attitude accuracy : 0.1°  
data rate : 20 Hz 