The construction of geometrical outdoor models, which consist of millions of points and patches, can be carried out with relatively efficiency and accuracy by using current 3D sensors, particularly with phase-shift and time of flight scanners. Colour information can also be acquired with high resolution by using appropriate digital cameras and lenses, which lead to a wide variety of parameters that can be set by hand.

However, the combination of a set of 3D clouds of points and colour images to obtain a photorealistic model is still an area of research which has a number of unsolved problems. The first of these relates to the experimental setup itself. Some scanners have an on-board camera with determined technical characteristics, and others permit the attachment of an external camera that has to be fitted in a specific position. Thus, in the field of digitalization, geometrical information and colour information are usually captured at the same time, under the illumination conditions that exist at the very moment of the 3D data acquisition. This makes the fusion of both types of information highly prone to errors. Under these circumstances, the process of geometrical and colour data acquisition is subdued in terms of both space and time: in terms of space because the sensor and the camera take up established positions during the data acquisition stage, and in terms of time because the geometrical information and the colour information are sensed simultaneously.

Consequently, mismatches commonly occur when combining texture images with 3D geometrical data either due to parallax effects that depend on the baseline between the laser scanner and the optical centre of the camera, or to differences between several colour images taken at different times and, therefore, under different illumination conditions.

The second problem arises when attempting to register and merge the set of data acquired from different positions in a unique representation model. Geometrical registration and 3D data integration problems, of course, were solved a number of years ago. There are numerous approaches in the literature related to registration, many of them based on the original well-known algorithm by Besl and McKay known as ICP (Iterative Closest Point Algorithm) [1]. Zhang presents [2] an important variant of the original algorithm. Xiao et al. [3] modify the ICP and utilize both regional surface properties and shape rigidity constraint to align a partial object surface and its corresponding complete surface. In addition to ICP, some other methods have been proposed for 3D registration. For example, in [4] genetic algorithms are used for registration of 3D data with low overlap and which may include substantial noise. Extensive comparison of registration methods can be seen in [5,6].

For the resolution of the integration problem, several methods have also been proposed. The most important classic ones are those based on “zippering” [7] or volumetric [8] algorithms. In [9] a faster variant of zippering algorithm is presented. Santos et al. propose in [10] a high fidelity merging algorithm that it is used for digital art preservation. Nevertheless, the integration of colour information taken from different viewpoints is an issue that still requires further research.

The problem in texture fusion is mainly caused by the colour disparity between two images. This is brought about by several reasons: First, changes in the angle between the observer direction and the normal of the reflecting surface and, second, variations in illumination conditions in the environments where they were captured. The third reason concerns the camera parameters, which may change from one scan to the next.

Many researchers have proposed solutions based on pairwise colour correction, colour mixing algorithms and image illumination change detection. When the scene is an artificially illuminated interior the difficulty for texture fusion is lessened, and acceptable results are obtained in most of the cases. In some cases, the light of the scene is controlled, so it is assumed that the scene is scanned under constant illumination. This assumption greatly simplifies the colour integration method which consists merely of searching for optimal blending functions [11–14]. For non-controlled scenarios, researchers have come up with only partial solutions. Most of these solutions are based either on global approaches, in which the complete colour image from one position of the scanner is processed (and therefore, modified), or on local strategies, in which only the colours at certain points of the image are corrected. In [15] the colours are inserted by detecting featured points in the image and defining the overlapped regions. Park et al. [16] developed a method that simultaneously fills in holes and associates (not acquired) colour. The study in [17] deals with several images of the scene, which are taken under different light conditions, and corrects discontinuities in the overlapped zones. Troccoli et al. [18] compute a relighting operator by analysing the overlapped area of a pair of images: the source image to be relighted and a target image. They then apply this operator over the non-overlapping region of the source, making it consistent with the target image. In [19], Dellepiane et al. present a colour correction space which is capable of providing a space-dependent colour correction for each pixel of the image. It estimates the flash light position and permits identification and removal of annoying artifacts, such as highlights and shadows. Since many of the developed methodologies for the mapping of colour information on 3D models also include the possibility of discarding parts of the input images or of selectively assigning a weight to contributing pixels, these authors state that their proposal is an interesting addition to such methods

Drastic approaches can be also found. For instance Sun et al. [20] remove, in overlapped regions, the less confident triangles and consider only the texture of the selected patches, avoiding any colour fusion procedure. Bernardini et al. [21] adjust the colours by global colour registration, imposing a restriction that distributes error in the results equally on the model. Finally, Xu et al. [22] integrate texture maps of large structures by decomposing the colour into two frequency bands and combining them.

The other line of research focuses on the obtention of a surface reflectance map of the scene. In this, the 3D model is illumination-free and can be rendered under any illumination conditions. Some interesting methods of obtaining a reflectance function are presented in [23] and [24]. These proposals were developed under laboratory conditions in which light positions are assumed to be known. Later, Debevec et al. [25] estimated the spatially-varying surface reflectance of complex scenes under natural illumination conditions. This approach uses a novel lighting measurement apparatus to record the full dynamic range of both sunlit and cloudy natural illumination conditions. Finally, in [26] the authors present a model of natural illumination as a combination of point source plus ambient component. They are able to compute the actual reflectance and to remove any effects of illumination in the images.

In spite of researchers' efforts over the last decade, the generated models are frequently incomplete or have neither sufficient accuracy nor quality. In particular, when dealing with outdoor scenarios the process becomes much harder. The colour images are severely affected by weather variability (rain and humidity) as well as the time of day at which they are taken. Furthermore, differing orientation and placement of the camera with respect to the surface must once again be taken into account.

This paper addresses the problem of how to build complete photorealistic models by uncoupling the time and particular circumstances in which 3D data and colour of the scene are captured. This issue is tackled from two different perspectives. Firstly, using a complete model (geometry plus texture) obtained after several data acquisition sessions, our aim is to modify the original texture provided on site by the scanner on-board camera with the colour information extracted from external images taken at particular times of day, times of the year (season) or under more suitable environmental conditions. We discuss this in Section 2. Secondly, we aim to keep the processes of geometrical data acquisition and colour acquisition apart, so that they become independent. Thus, the acquisition of colour information with external high-resolution cameras could be executed at chosen times, separately from the data acquisition for the geometrical modelling process, which is not sensitive to time or weather variations. This task therefore involves assigning colour to the geometrical model once it is finished, rather than modifying an existing colour model on it. Section 3 of this paper explains the procedure carried out to do this.

A further aspect to bear in mind with regard to our study relates to the fact that our algorithms increase the automation level in the overall geometry and colour merging process. In particular, we aim to automate two sub-processes: the colour fusion algorithm for several views by using a previous analysis of the global colour of the scene, and the search for the best view in the stage of making the match between the model and the external images. The automation of these phases is intended to go beyond manual and user dependent procedures. Despite this, the current state of the method presented in this paper requires user intervention in some steps. For instance, the user selects the set of external images to be used and can also vary certain specific parameters at the colour integration stage, such as window sizes, masks, thresholds, etc. These aspects are dealt with in Section 4.

Finally, the experimental section shows the results obtained in large outdoor scenarios. Specifically, we present the work carried out in two Roman archaeological sites that date back to the first century A.D., the theatre of Segobriga and the Fori Porticus of Emerita Augusta, both in Spain.

Let us assume a 3D model M of an object obtained from the information acquired with a scanner equipped with an internal low-resolution camera. The rough information provided by the sensor is composed of the 3D coordinates of the points of the object and their corresponding colors in RGB format. A meshing algorithm then yields a patch representation, including vertices and normal vectors. We specifically used the volumetric approach published in [8] to generate the partial meshes. The view registration and geometric integration problem is solved using the k–d tree algorithm published by Zhang in [2]. Given the model M (V, F, N, C) where:

Our goal is to modify C by using a new set of images taken with external high-quality cameras and to obtain a photo-realistic model M′ (V, F, N, C).We achieve this after the sequence of steps drawn in the sketch shown in Figure 1.

In the first step, we choose two images, I₁ and I₂. I₁ can be either the image provided by the camera of the scanner itself or an ortho-photo of M from a viewpoint close to the viewpoint of the external high-quality camera when taking I₂. The following step consists of selecting control points in both images. In the software we have created to execute colour modification, these can be marked automatically or by hand. If the automatic option is selected, the algorithm picks several pixels equally separated in the image I₂ and looks for their corresponding ones in I₁. Then, for every pair of marked pixels, an iterative algorithm is run so that the position of each selected pixel in I₁ is recalculated, and the correspondence error between pixels is minimized. The procedure for the iterative algorithm can be summarized as follows: for a given pair of pixels, a window W₂ centred in the corresponding pixel from I₂ is set. At the same time, a smaller window W₁ is selected for the pixel in I₁. By means of a cross-correlation algorithm all the possible superimpositions of W₁ onto W₂ are obtained. The cross-correlation factor R is calculated by following Equation (1).

(1) R = ∑ x ∑ y ( P 1 , x y − I ¯ 1 ) ( P 2 , x y − I ¯ 2 ) ( ∑ x ∑ y ( P 1 , x y − I ¯ 1 ) 2 ) ( ∑ x ∑ y ( P 2 , x y − I ¯ 2 ) 2 )where P₁,_xy is the pixel (x,y) in the image I₁, P₂,_xy is the pixel (x,y) in I₂ and Ī₁ and Ī₂ are the arithmetic means of the components of I₁ and I₂, respectively. As is known, the superimposition that gives the cross-correlation factor closest to 1 corrects the correspondence of pixels between I₁ and I₂. Figure 2 shows an example detailing several stages of the pixels correspondence process.

The next step consists of correcting the original image I₁. The correspondence between n pixels from I₁ and I₂ can be expressed by means of a homogeneous transformation A in Equation (2): (2) P e = A P mwhere P_e and P_m are the coordinates of the pixels in I₁ and I₂, respectively.

Note that the alignment between images I₁ and I₂ is not an affine homography because the scene is not exactly the same for both cases. I₁ and I₂ are taken for different scenes and seen under different optical properties. Thus, I₁ corresponds to the image of a 3D model and is generated under graphical characteristics. In this case, the scene is a 3D model. Nevertheless, I₂ is the real image captured by the camera, which has its own optical properties, and the scene is now the real environment.

For a general case: (3) P e = ( λ 1 x e 1 λ 1 x e 2 ⋯ λ n x e n λ 1 y e 1 λ 1 y e 2 ⋯ λ 1 y e n λ 1 λ 2 ⋯ λ n ) , A = ( a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 ) , P m = ( x m 1 x m 2 ⋯ x m n y m 1 y m 2 ⋯ y m n 1 1 ⋯ 1 ) If we chose a₃₃ = 1, then: λ₁=a₃₁x_m1+a₃₂y_m1+1 λ₂=a₃₁x_m2+a₃₂y_m2+1 ⋮ λ_n=a₃₁x_mn+a₃₂y_mn+1

Taking these n related points as a starting point, Equation (2) leads to the expression: (4) H A v = X pand matrices X_p and H are obtained from P_e and P_m in the following way: (5) X p = ( x e 1 y e 1 x e 2 y e 2 x e 3 y e 3 ⋯ ⋯ x e n y e n ) , H = ( x m 1 y m 1 1 0 0 0 − x m 1 x e 1 − y m 1 x e 1 0 0 0 x m 1 y m 1 1 − x m 1 y e 1 − y m 1 y e 1 x m 2 y m 2 1 0 0 0 − x m 2 x e 2 − y m 2 x e 2 0 0 0 x m 2 y m 2 1 − x m 2 y e 2 − y m 2 y e 2 … … … … … … … … … … … … … … … … x m n y m n 1 0 0 0 − x m n x e n − y m n x e n 0 0 0 x m n y m n 1 − x m n y e n − y m n y e n ) , A v = ( a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 )

Then we solve the Equation (4) and obtain the transformation matrix between the two images I₁ and I₂: (6) A v = ( H T H ) − 1 H T X p

Once we have A, the final corrected image, I 1 ′ will be the one obtained after modifying the RGB components with the RGB values of the corresponding pixels in the image I₂. Let us denote C as the vector which contains the three colour components, C = [R G B]^t. For each pixel of I₁ the colour is modify as follows: (7) C ( P k ′ ) = C ( P k ) P k = A P k ′ λ k , ∀ k

P_k and P k ′ being the coordinates of the k-th pixel in I₂ and I 1 ′, respectively, and C(P_k) being the colour information stored in the k-th pixel of I₂ and C ( P k ′ ) the colour information assigned to the k-th pixel in I 1 ′.

In those cases when the image deformations are not considered, that is, when dealing with a geometric transformation, the following equalities are verified: (8) λ 1 = λ 2 = ⋯ = λ k = ⋯ = λ n = 1 ; a 31 = a 32 = 0 ; a 33 = 1

At the end of this process, a global correction algorithm is applied to the initial image I₁. Thus, the transformation that relates the colour components of image I₁ to the colour components of image I 1 ′ can be modelled by a linear transformation as follows: (9) C I 1 ′ = Z C I 1and the values of the elements of Z can be computed using Equation (10): (10) Z = ( C I 1 ′ C I 1 T ) ( C I 1 ′ C I 1 T ) − 1

Note that this section is developed taking one view of the original coloured model and therefore Z represents a colour transformation for this particular view. Nevertheless, in the case of uniform colour models, transformation Z could be used to modify the colour of any partial view of the model. This circumstance frequently occurs in heritage pieces, like marble statues or stone monuments. Formally: (11) C M ′ = Z C M

Figure 3 shows an example in which the colour of the whole model of the piece has been modified.

Let us now suppose that we have a geometrical model M (V, F, N) of a large scenario with no integrated texture and a set of high resolution images taken without any restrictions with an external camera. Restrictions concern any aspect related to the image (size, accuracy, resolution…) and the time at which the images were taken. With the collection of images and the geometry of the scene, a procedure to obtain a final complete model that comprises colour and geometry has been developed.

It is well known that colour assignation can be carried out on-line using a camera and always under an accuracy camera-scanner calibration. As mentioned in Section 1, this involves certain inconveniences in colour integration post-processing. By decoupling geometry and colour with respect to the moment at which the data are taken, our aim is to integrate the texture in the geometry in a controlled off-line process.

Our procedure consists of three main stages:

At this first stage, two images are chosen: an ortho-photo of the 3D geometrical model, which will be denoted as I₁, and an external image acquired without any restrictions, I₂. The viewpoint that provides the image I₁ must be defined in such a way that the resulting projected view can be compared to the external image I₂, as shown in Figure 4. Several transformation matrixes must be stored when projecting the model to assure the reverse transformation from the coloured ortho-image to the 3D model, i.e., the matrix that controls the position of the viewpoint which the user views the model from, the projection transformation matrix, which defines the kind of perspective applied to the 3D model (either orthogonal or orthographical), and the model rotation/translation matrix that stores the rotation and/or translation undergone by the 3D model to be placed in a view close to the external image view. It is also necessary to record the size (in pixels) of the window in which the 3D model is viewed.

When the colour assignation for a specific viewpoint is finished, we have a coloured model M (V, F, N, C′) in which only a part of its vertices has colour information. In order to have the whole model coloured, we need to assign the texture provided by the set of external images to the rest of the vertices. As was mentioned before, the images are taken without restrictions about the point of view and they may or may not overlap.

We carry out an iterative procedure that assigns colour to some of the nodes as new images are matched onto the geometric model. Let us assume a particular step t in the iterative process. At this point, we denote T̃_t as the matrix generated throughout the earlier t-1 iterations which stores the colour information extracted from t-1 external images. A new external image can provide texture for some of the vertices that have not yet been assigned, leading to a matrix T_t+1 that must be combined with T̃_t to produce a new matrix T̃_t+1 (Figure 7). This process is generically denoted by the symbol ⊗ in Equation (13)

In a sense, matrix T̃_t is a matrix accumulated after t-1 iterations (see Figure 7). Let us suppose that we have the mesh model without colour in iteration number 0. The colour fusion process is defined depending on the overlapping between the new view and the current status of the model. Thus if T̃_t y T_t+1 have no nodes in common, Equation (13) then becomes T̃_t+1 = T̃_t ∪.T_t+1 Nevertheless, if T̃_t y T_t+1 have common nodes the colour merging then becomes a more complex process. We can distinguish between global and local approaches.

Depending on the kind of scene we are working on, we apply either one of the two processes described below or both of them. We will denote the two processes using symbols ➀ and ➁ in Equations (14) and (15): (14) T ˜ t + 1 = T ˜ t 1 T t + 1 (15) T ˜ t + 1 = T ˜ t 2 T t + 1

The chart in Figure 8 illustrates both processes. The process ➀ is illustrated in Figure 8(a). In order to make the notation more readable from here on we will assume that the superscript “c” in the following equations will signify the overlapped part of matrices T̃_t and T_t+1 and superscript “c̄” will be the non-overlapped part. Thus T ˜ t c and T ˜ t + 1 c are the sub matrixes for the common nodes in matrices T̃_t and T_t+1.

Process ➀ consists of three steps. First, a transformation is applied over T t + 1 c, and T_t+1 turns on T′_t+1. This transformation is obtained by using the colour relationship between T ˜ t c and T t + 1 c, which can be modelled by means of transformation Z in Equation (16): (16) T ˜ t c = Z T t + 1 c

The solution for Z is given by the equation: (17) Z = T ˜ t c [ T t + 1 c ] T [ T t + 1 c ⋅ [ T t + 1 c ] T ] − 1

Thus, matrix T_t+1 is transformed after: (18) T t + 1 ′ = Z ⋅ T t + 1

After this, we calculate the average values between T ˜ t c and T t + 1 c ′ and add the non-common part T t + 1 c ¯ ′, generating the partial matrix T t + 1 a (Equation (19)). T′_t+1 then turns on T″_t+1.

(19) T t + 1 a = { [ T ˜ t c + T t + 1 c ′ 2 ] ∪ T t + 1 c ¯ ′ }

Finally, T t + 1 a is inserted in T̃_t and T̃_t+1 is generated. We denote the insertion operator by means of the symbol ⊕ in Equation (20): (20) T ˜ t + 1 = T ˜ t ⊕ T t + 1 a

Process ➁ is illustrated in Figure 8(b,c). It consists of making a colour substitution of T ˜ t c by using a weighted expression of the colour. The weight factors are based on the observation angles for the nodes of which the colour values are about to be modified.

We start by defining a weighted observation angle for each node of the model which is seen in previous scans. This parameter will be used if we want to take into account local properties in the model. Formally, we define the weighted observation angle as follows.

Let us suppose N be a node of T̃_t and T_t+1 with assigned colours C̃_t(N) and C_t+1(N), respectively. Let θ_t+1(N) be the observation angles of N in the step t, where θ t + 1 ( N ) = ( n → , l → ^ ) , l → = N R → t + 1 , n → being the normal at N and R_t+1 being the camera position in iteration t. Finally, let θ̃_t(N) be a weighted observation angle which is computed after t-1 iterations. The weighted observation angle at iteration t is calculated as follows: (21) θ ˜ t + 1 ( N ) = θ t + 1 ( N ) ⋅ w t + 1 + θ ˜ t ( N ) ⋅ w ˜ t ( N ) w t + 1 ( N ) + w ˜ t ( N )in which weights w_t+1 are based on angles θ_t+1: (22) w t + 1 ( N ) = { 1 , θ t + 1 ( N ) ≤ a θ t + 1 − b a − b , a < θ t + 1 ( N ) < b 0 , θ t + 1 ≥ band w̃_t is based on previous weights in the following manner: (23) w ˜ t ( N ) = ∑ k = 1 t − 1 w k t

In Equation (22)a and b are two thresholds imposed to the observation angles.

With this algorithm, matrix T̃_t is modified taking into account the colour and the observation angle for each common node in T̃_t and T_t+1. The colour updating for a node N in T̃_t is given by the expression: (24) C ˜ t + 1 ( N ) = C t + 1 ( N ) ⋅ w t + 1 ( N ) + C ˜ t ( N ) ⋅ w ˜ t ( N ) w t + 1 ( N ) + w ˜ t ( N )

Where C̃_t(N) is the colour of N in T̃_t and C_t+1 is the colour of N in T_t+1, the weights w_t+1 and w̃_t are calculated with Equations (22) and (23) from the observation angle s θ_t+1(N) and θ̃_t(N), respectively.

In conclusion, the process of colour merging, depending on the type of scene we are dealing with, can consist of:

In order to illustrate the worthiness of our proposal, we present in this section the results for the reconstruction of two Roman archaeological sites from to the first century A.D., the Roman Theatre of Segobriga and the Fori Porticus of Emerita Augusta (Figure 9). They are both good prototypes of exterior scenarios, huge outdoor spaces with a high degree of difficulty due to the multitude of occlusions that arise when acquiring the surfaces.

For the acquisition of the geometry, we used both a Faro LS 880 laser scanner and a Faro Photon 80 sensor to provide a panoramic view of the scene. Azimuth and elevation angles range over [0, 360°] and [−65°, 90°], respectively, and the maximum data density of the scanner is 40.110 × 17.223 points. Obviously when the data is used in outdoor scenes most of the potential data are missed because they correspond to the sky and only a part of the data are useful. The colour information has been obtained with two Nikon D200 cameras with Nikon AF DX Fisheye lens, one for each scanner. This system composes a single panoramic image from 10 photos, 10.2 Mpixels each. The system takes 1 hour and 49 minutes to scan a panoramic sample plus 8 minutes to take the sequence of photos.

Building accurate and complete (geometry+texture) 3D models of large heritage indoor/outdoor scenes from scanners is still a challenging and unresolved issue on which many researchers are currently working [27,28]. In the last decade technology relating to 3D sensors has significantly improved in many respects and the devices are now more flexible, more precise and less costly than in the past. Consequently, we believe that better reverse engineering results will be obtained in this area. Processing of information, nevertheless, may be inefficient if the scene is digitalized under unsuitable conditions. This frequently occurs in outdoor scenarios in which the illumination and the environment conditions of the scene cannot be controlled. This paper proposes to improve the existing modelling techniques by decoupling the colour integration and geometry reconstruction stages. Colour assignation in particular is carried out as an independent and controlled process. The method has basically two parts: colour assignation for one shot of the scanner and colour fusion of multiple views.

Colour assignation focuses firstly on modifying low-quality coloured 3D models by using sets of images of the scene that have been taken at different times and circumstances than those of the geometrical information capture phase. In this approach a colour transformation is obtained after matching the model's orthoimages with external images. The second line is addressed to directly assigning external images to the complete geometric model. Both approaches are quite simple and can easily be implemented in practice.

The merging colour algorithm is proposed in the last part of the paper. This stage is carried out under an iterative process in which the next external image is automatically integrated into the ‘as-built’ model. The result is a complete coloured model that can be refined in the last step.

The proposed method has been widely tested in large outdoor scenes with very good results; the present paper shows the results obtained at two popular heritage sites in Spain. We have achieved high-quality photorealistic models in all tests we have performed. Comparison with former models is also illustrated at the end of the paper.