True² Orthoimage Map Generation

Abstract

Digital/true orthoimage maps (D/TOMs) are one of the most important forms of national spatial data infrastructure (NSDI). The traditional generation of D/TOM is to orthorectify an aerial image into its upright and correct position by deleting displacements on and distortions of imagery. This results in the generated D/TOM having no building façade texture when the D/TOM superimposes on the digital building model (DBM). This phenomenon is no longer tolerated for certain applications, such as micro-climate investigation. For this reason, this paper presents the generation of a true² orthoimage map (T²OM), which is radically different from the traditional D/TOM. The basic idea for the T²OM generation of a single building is to orthorectify the DBM-based building roof from up to down, the building façade from front to back, from back to front, from left side to right side, and from right side to left side, as well as complete a digital terrain model (DTM)-based T²OM, of which a superpixel is proposed to store building ID, texture ID, the elevation of each pixel, and gray information. Two study areas are applied to verify the methods. The experimental results demonstrate that the T²OM not only maintains the traditional characteristics of D/TOM, but also displays building façade texture and three-dimensional (3D) coordinates (XYZ) measurable at any point, and the accuracy of 3D measurement on a T²OM can achieve 0.025 m (0.3 pixel).

1. Introduction

Digital orthophotomaps (DOMs) are a critical component of national spatial data infrastructure (NSDI) [1,2,3,4]. DOMs (1) serve as a geospatial foundation upon which to add detail and attach attribute information; (2) provide a base on which to accurately register and compile other themes of data; and (3) orient and link the results of an application to the landscape [5,6]. Especially, a highly detailed DOM is capable of serving as a source for locating the features to be mapped and measured [7].

Many investigations have demonstrated that the generation of high-resolution urban DOMs using the existing procedures and algorithms, proposed in 1990 by the USA National Digital Orthophoto Program (NDOP), has encountered many problems, such as incomplete orthorectification, occlusion, ghost image, shadow, etc. A comprehensive discussion regarding these problems can be found in [8,9]. Thus, the generation of so-called true orthophoto maps (TOMs) has become obligatory, and has been researched by many studies at the end of the 20th century and the beginning of the 21st century, such as [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39].

However, when a TOM superimposes onto a corresponding digital terrain model (DTM) in a flat and/or hilly area, the terrestrial texture is clearly visible (Figure 1a), but when a TOM superimposes onto a digital building model (DBM), the detailed façade textures of a building are not visible (Figure 1b). The impacts of these problems have significantly influenced the usefulness of TOMs in industries such as micro-climate monitoring, micro-environment analysis, and cellphone transmission station distribution, since incomplete building façade information cannot be tolerated for these applications. Therefore, this paper proposes a true² orthoimage map (T²OM) generation method which can provide three-dimensional (3D) and detailed textures of a building’s roof and facades.

2. Related Works

The study of TOM began in the 1980s and has continued for decades. A successful and complete TOM generation method consists of an orthorectification algorithm, occlusion detection and compensation, shadow detection and recovery, a seamless mosaic, etc. The review of related works is, therefore, classified accordingly as follows.

Occlusion detection and compensation: Occlusion detection is one of the most major components of TOM generation. Amhar et al. [10] and Schickler and Thorpe [11] considered the hidden effects introduced by abrupt changes of surface height (e.g., buildings and bridges). Jauregui et al. [20] presented a procedure for orthorectifying aerial photographs to produce and update terrain surface maps. Vassilopoulou et al. [40] used IKONOS images to generate orthoimages for monitoring volcanic hazards on Nisyros Island, Greece, and Siachalou [21] used IKONOS images to generate the urban orthoimage. Cameron et al. [41] analyzed orthorectified aerial photographs to measure changes in the native pinewood of Scotland, and Passini and Jacobsen [42] analyzed the accuracy of orthoimages from very-high-resolution imagery. Biason et al. [22] further explored the automatic generation of true orthoimages. Piatti and Lerma [43] address the problem of image orthorectification through photogrammetric simulation and its generation based on digital elevation/building/surface models, as well as internal and external orientation parameters of image sensors (i.e., digital cameras). This method appears to be able to create the high-resolution 3D models needed for accurate orthophotos. Zhou et al. [29] proposed a new urban orthophoto occlusion detection method, which first establishes a model describing the relationship between each ghost image and the corresponding building occlusion boundary, and then applies an algorithm that uses building displacement to identify the occlusion region in the ghost image. The method can effectively avoid pseudo-occlusion detection and the drawback of simultaneous occlusion detection and orthophoto generation, providing a key technique for DBM-based T²OM. Yoo and Lee [44] proposed a facet-based method for generating realistic orthophotos of building surface facets. The method identifies occluded areas based on the unit surface of the building and uses multiple images and high-detail digital building model data (i.e., DBM) to recover the occluded areas from each other. Oliveira et al. [45] proposed a new occlusion detection method for true radiographic image mosaic generation. The proposed method uses irregularly spaced point clouds to identify occluded regions, avoiding the interpolation process as an initial step in occlusion detection and thus avoiding the insertion of additional errors in the surface representation. Zhou and Sha [46] proposed a method to simultaneously detect building roof and ground shadows using the DBM as an overlay model. The method determines the solar zenith angle and solar altitude angle by selecting the geographic information of the corner points on the shadow boundary in the aerial image, and then displays the actual shadow area determined in the DBM on the ghost image. The method is independent of ground reflectivity and illumination conditions, and provides technical support for producing high-quality true projection images. Marsetič [47] proposed a method to automatically generate true orthophotos from optical super-resolution satellite images. The automatic workflow consists of five modules, starting with the extraction of ground control points, then the geometric processing of image patches, occlusion detection, orthorectification, and finally generating a real orthophoto. The quality of the true radiographic images produced by this method depends on the accuracy of the geometric correction and the number of images. The occlusion compensation for TOM generation, Skarlatos [13] and Greenfeld [15] demonstrated that building occlusions significantly influenced not only image quality but also the accuracy of the orthoimages. Rau et al. [18] treated enhancements in image radiometry, demonstrating a suitable enhancement technique to restore information within building shadow areas. Sheng et al. [48] used a model-based method to reconstruct a true orthophoto model (CSM) to replace the DEM to generate a true photogram of the forest scene, which mainly focused on the efforts of occlusion and distortion caused by trees in the forest area. Zhou et al. [9] compensated by conjugating blocks of orthoimages, i.e., by refilling the masked area adjacent to the orthoimages. Zhou et al. [29] used adjacent overlapping “slave” orthophotos to fill the occluded region using the filling method proposed by Zhou et al. [9] to compensate for the occluded region in the “master” orthophoto. With such occlusion compensation, a complete true orthophoto can be created for the study area.

For shadow detection and recovery, many efforts have been made. For example, Leone and Distante [49] performed image shadow detection by improving the classification, segmentation and localization of detected objects, which improved the effect of shadow detection. Makarau et al. [50] proposed an alternative robust method for shadow detection. The method is to adaptively calculate the parameters of a specific scene and allow one to use many different sensors and images obtained under different lighting conditions, improving the accuracy of shadow detection. Tiwari et al. [51] propose an improved algorithm to obtain rough shadows by changing the ratio of intensity to hue and then performing shadow compensation using local thresholding. The experimental results show that the method is more suitable for the shadow detection of low-intensity and medium-intensity images, and the shadow compensation algorithm is suitable for all test images.

For the mosaic of multiple TOMs. Many researchers have made many efforts for the generation of high-quality TOM through the improvement of TOM mosaicking, such as [11,17,26,36,37,38,52,53]. Their studies resulted in clearer features in the shadows and more continuous and natural grayscale of the filled areas and surrounding images. For example, Pan and Wang [53] adopted a multi-scale processing strategy which can automatically locate the specific positions of the splicing lines and transition areas and improve the image quality after mosaicking. Gharibi and Habib [36] proposed a weighted averaging method to mitigate seam line effects and spectral differences that may occur in true orthophoto mosaics.

Despite many previous efforts, traditional TOM only provides 2D (XY) coordinates and building roofs’ texture information, while the 3D attributes and the textures of the building facades cannot be provided at all. For this reason, the generation of T²OM is presented in this paper. The organization of this paper is arranged as follows: The principle of T²OM is presented in Section 3, and Section 4 presents the experimental results and analysis. The conclusions are drawn in Section 5.

3. Principles of True² Orthoimage Map (T²OM) Generation

T²OM is defined as a DOM that can provide measurable 3D (XYZ) coordinates and textures for both the roof and the facade of a building. This means that the T²OM not only has traditional TOM characteristics, but also provides the 3D geometric information (X,Y,Z) and the textures for the facade of a building. The method of generating a T²OM includes the four basic steps below:

(1)

DBM-based single-building T²OM generation, which consists of orthorectifying both the building roof and building facades: a concept, named “superpixel” is proposed the for storage of building texture, building ID, etc. information.

(2)

DBM-based multiple-building T²OM generation: merging the DBM-based single-building T²OM, including organization of the building ID, building façade, building corner coordinates, etc.

(3)

DTM-based T²OM generation for the orthorectification of gentle and continuously elevated hilly areas.

(4)

DTM- and DBM-based T²OM merging, which is for merging DTM- and DBM-based T²OM for the creation of an entire T²OM.

3.1. Generation of a DBM-Based Single-Building T²OM

In order to clearly describe the process for the generation of T²OM for buildings, a single building is first taken as an example (see Figure 2), presuming that the DBM for the single building and the exterior/interior orientational parameters (EOPs/IOPs) for an image are known. The steps for the generation of a T²OM for a single building are below.

Step 1: DBM-based building roof orthorectification, which consists of:

(1)

Determining the size of the T²OM: The resulting DBM-based single-building T²OM is expressed as a raster image with pixels arranged in rows and columns. Since the resulting orthoimage is orthorectified from raster image input (called original image) using the DBM data, the size of the output image is defined [9] as

$$\begin{array}{l}{X}_0=\max \left\{\min \right\{X_D\},\min \{X_I\left\}\right\},Y_0=\max \left\{\min \right\{Y_D\},\min \{Y_I\left\}\right\}\\ X_1=\min \left\{\max \right\{X_D\},\max \{X_I\left\}\right\},Y_1=\min \left\{\max \right\{Y_D\},\max \{Y_I\left\}\right\}\end{array}$$

(1)

where X₀ and Y₀ are the coordinates of the lower-left corner of the output image; X₁ and Y₁ are the coordinates of the upper-right corner of the output image; X_D and Y_D are the X and Y coordinates of the DBM; X_I and Y_I are the X and Y coordinates of the original image; and max and min denote maximum and minimum of the elements in the blanket. All of the coordinates here refer to the geodetic coordinate system required in the resulting T²OM.

(2)

Computing the X, Y coordinate of each pixel: In Figure 2, P (I, J) is a given point pixel on the roof of the T²OM building roof, and their raster rows and columns can be transformed to the coordinates of the output T²OM, i.e.,

$$\begin{array}{l}X=X_0+I\times P_x\\ Y=Y_0+J\times P_y\end{array}$$

(2)

where X and Y represent the coordinates of pixels; X₀ and Y₀ are the coordinates of the lower-left corner of the output roof T²OM; P_X and P_y are the sizes of the pixels in the X and Y directions, respectively; and I and J are the rows and columns of P points, respectively.

(3)

Computing the Z coordinate of P (I, J): In order to perform orthorectification, we also need to know the Z coordinates of the pixel P (I, J) in the output roof T²OM and this is obtained from DBM. However, DBM data only have vector coordinates at corner points. Therefore, it is necessary to interpolate an elevation to the roof pixel of the building. As shown in Figure 3, the elevation (height) is obtained only for pixels with corner points (blue pixels in Figure 3), while the other pixels (orange in Figure 3) are calculated by:

$$h=a_{0}j-a_{1}i-a_2$$

(3)

where h is the raster height value; i, j is the row number of the raster; a₀, a₁ and a₂ are the equation weight values defined by:

$$Aj+Bi+Ch+D=0\left(C\ne 0\right)$$

(4)

$$h=-\frac{A}{C}j-\frac{B}{C}i-\frac{D}{C}$$

(5)

$$a_0=-\frac{A}{C},a_1=-\frac{B}{C},a_2=-\frac{D}{C}$$

(6)

where each triangular surface weight can be calculated from Equation (2) in the case of three known vertices of the triangular surface, and A, B and C are the equation weight values.

When the pixels on the roof of the building have an elevation, we have to convert the pixel’s geodetic coordinates back into the column and row by:

R_D = (X − X_0D)/P_D, C_D = (Y − Y_0D)/P_D

(7)

where P_D denotes the pixel size of the DBM image. X_0D and Y_0D are the lower left corner coordinates of the DBM image. K_D and L_D will not generally be exact integers. Thus, an interpolation must be performed to determine Z. Usually, a bilinear interpolation method of the following form is employed (see Figure 2c):

$$Z=\left\{\left[Z_1\Delta X+\left(1-\Delta X\right)Z_4\right]+\left[Z_2\Delta X+\left(1-\Delta X\right)Z_3\right]+\left[Z_1\Delta Y+\left(1-\Delta Y\right)Z_4\right]+\left[Z_2\Delta Y+\left(1-\Delta Y\right)Z_3\right]\right\}/4$$

(8)

where △X = R_D − R_m and △Y = C_D − C_m, in which R_m is the R_D rounded to its maximal integer and C_m is C_D rounded to its maximal integer. After this estimation, we then know the coordinate (X, Y, Z) of the pixel.

(4)

Computing the corresponding coordinate in the original image: In order to orthorectify the source image, the corresponding coordinate of the source image pixel in the output image is calculated by:

$$\begin{array}{l}{x}_I=x_{0I}-f\frac{a_1(X-X_s)+b_1(Y-Y_s)+c_1(Z-Z_s)}{a_3(X-X_s)+b_3(Y-Y_s)+c_3(Z-Z_s)}\\ y_I=y_{0I}-f\frac{a_2(X-X_s)+b_2(Y-Y_s)+c_2(Z-Z_s)}{a_3(X-X_s)+b_3(Y-Y_s)+c_3(Z-Z_s)}\end{array}$$

(9)

where x_I and y_I are the corresponding coordinates of the pixel P(X,Y) in the source image; X_s, Y_s and Z_s are the exposure stations; f is the focal length; ai = {a₁, a₂, a₃}, bi = {b₁, b₂, b₃} and ci = {c₁, c₂, c₃} are the elements of the rotation matrix, and are the functions of the three exterior orientation angles (ϕ, ω, κ). These elements have to be computed using at least three ground control points (GCP).

(5)

Assigning the gray value to pixels: Since the grid of pixels in the source image rarely matches the grid of the output orthoimage, a re-sampling of the pixels has to be performed in order to assign gray value to the pixels in the output image. The nearest neighbor is employed because it directly transfers the original data values without averaging them. The computational procedure is illustrated in Figure 2.

(6)

Storing the data for DBM-based building roof T²OM: As can be seen from the above, T²OM needs to store more information than the traditional TOM does, such as building roof texture, facade texture, and façade Z coordinates. For this reason, “superpixel” is presented and has the following characteristics (see Figure 4): (1) it inherits the original image gray information; (2) the gray value, elevation, building ID, and facade texture index ID are stored; (3) each pixel coordinate is directly interconnected with the building ID and façade texture ID.

The detailed descriptions of G, S, H, and ID in the superpixel are as follows:

(1)

G (I, J) stands for storage of the gray at i-th row and j-th column in the image coordinate system whose value is 0–255.

(2)

S (I, J) stands for storage of the building corner coordinate subdivision grid identification value, which occupies 8 bits. That is, by dividing a single pixel into 256 subdivision sequences, the accuracy of vector to grid data conversion is improved. For a given point P (X_P, Y_P), this can be expressed as (i_p, j_p) after the conversion of the vector to the grid. The lost information is $\left(X_P-X_0\right)-i_p\cdot \Delta X$. With this information, Sx can be calculated through Equation (8).

$$S_x=\lfloor \frac{\left(X_{{}_P}-X_0-\Delta x\cdot i_p\right)\cdot 16}{\Delta x}\rfloor$$

(10)

where $\Delta X$ is the image resolution and $\lfloor .\rfloor$ is the function of rounding down. (X₀, Y₀) is the top-left point of the image.

The calculation method used for S_y is the same as that used for S_x. At this time, S is expressed as two values. This makes the storage of S very difficult. Therefore, (S_x, S_y) is converted into a one-dimensional form by means of the Morton code [54].

(3)

H (I, J) stands for the storage of the building height or DTM height with a floating format.

(4)

ID stands for the storage of building ID. An ID can be used to call for the facade texture. A large city may have hundreds of buildings; therefore, 12 bits are designed to store 0 to 4095 buildings.

The T²OM data for the roof of the building generated by the above steps are stored in the superpixel and shown in

Table 1. The superpixel for the roof texture data-generated DBM-based building.

DBM-Based Pixel	Row	Column	Gray Value	Sub Ordinal	Height	BuildingID
BP1	I(BP1)	J(BP1)	G1	id(BP1) (r,c)	h(BP1)	B1
BP2	I(BP2)	J(BP2)	G2	0	h(BP2)	B1
…	…	…	…	…	…	…
BPi	I(BPi)	J(BPi)	Gi	id(BPi) (r,c)	h(BPi)	B1
BPi + 1	I(BPi + 1)	J(BPi + 1)	Gi + 1	0	h(BPi+1)	B1
…	…	…	…	…	…	….

.

Step 2: DBM-based orthorectification for a single-building facade

In order to obtain the façade texture and the 3D coordinates of a building, the orthorectification for building facade in four directions is also performed in T²OM, and the four directions are determined according to the minimum bounding box (a detailed description has been given by [55]). The basic idea for the orthorectification of a building façade is: four directions at 0° for the front façade (Figure 5a), 90° for the left façade (Figure 5b), 180° for the back façade (Figure 5c), and 270° for the right façade (Figure 5d) are orthorectified, respectively. For example, the co-linear equation for the 0° directional facade texture is adopted (see Figure 6a), i.e.,

$$\begin{array}{l} y_g=-f\frac{a_1\left(Y_G-Y_S\right)+b_1\left(Z_G-Z_S\right)+c_1\left(X_G-X_S\right)}{a_3\left(Y_G-Y_S\right)+b_3\left(Z_G-Z_S\right)+c_3\left(X_G-X_S\right)}+y_0\\ z_g=-f\frac{a_2\left(Y_G-Y_S\right)+b_2\left(Z_G-Z_S\right)+c_2\left(X_G-X_S\right)}{a_3\left(Y_G-Y_S\right)+b_3\left(Z_G-Z_S\right)+c_3\left(X_G-X_S\right)}+z_0\end{array}$$

(11)

The facade texture in 0° direction is orthorectified into the ZOY plane (see Figure 6a). Similarly, the collinear equation for the orthorectification of the other three directions at 90°, 180°, 270° can be orthorectified into the ZOX (see Figure 6b), YOZ (see Figure 6c), and YOX (see Figure 6d) planes, respectively.

The details of the orthorectification of a single building can be further described as follows: As shown in Figure 7, the buildings in Figure 7a,b are 25.8 m and 46.4 m, respectively, with an elevation resolution of 0.2 m. There are four planes (a₁b₁, b₁c₁, c₁d₁ and d₁a₁) for the building facade in Figure 7a, and the corresponding texture index data are 65, 66, 67 and 68, respectively. There are four planes (a₂b₂, b₂c₂, c₂d₂ and d₂a₂) for the building facade in Figure 7b, and the corresponding texture index data are 809, 810, 811 and 812, respectively (see

Table 2. The building wall texture superpixel data in Figure 7a.

Pixel	Gray (8 bit)	Hight (H, 12 bit)	TextureID (TID, 12 bit)	Notes
1	11000110	000100000010	000001000001	Gray = 198		Wall 1 H = 25.8 TID = 65
2	11010011	000100000010	000001000001	Gray = 211
3	11010000	000100000010	000001000001	Gray = 208
…	…	…	…	…
8	11010011	000100000010	000001000001	Gray = 211
9	11001000	000100000010	000001000010	Gray = 200		Wall 2 H = 25.8 TID = 66
10	11101000	000100000010	000001000010	Gray = 232
…	…	…	…	…
18	11011101	000100000010	000001000010	Gray = 221
19	11011101	000100000010	000001000011	Gray = 221		Wall 3 H = 25.8 TID = 67
20	11011000	000100000010	000001000011	Gray = 216
…	…	…	…	…
27	11011000	000100000010	000001000011	Gray = 216
28	11010001	000100000010	000001000100	Gray = 209		Wall 4 H = 25.8 TID = 68
29	11011010	000100000010	000001000100	Gray = 218
…	…	…	…	…
37	11011010	000100000010	000001000100	Gray = 218

and

Table 3. The building façade texture superpixel data in Figure 7b.

Pixel	Gray (8 bit)	Hight (H, 12 bit)	TextureID (TID, 12 bit)	Notes
1	11010101	000111010000	001100101001	Gray = 213		Wall 1 H = 46.4 TID = 809
2	11111101	000111010000	001100101001	Gray = 253
3	11110001	000111010000	001100101001	Gray = 241
…	…	…	…	…
9	11010100	000111010000	001100101001	Gray = 212
10	11010111	000111010000	001100101010	Gray = 215		Wall 2 H = 46.4 TID = 810
11	11010101	000111010000	001100101010	Gray = 213
…	…	…	…	…
22	11010101	000111010000	001100101010	Gray = 213
23	11011111	000111010000	001100101011	Gray = 223		Wall 3 H = 46.4 TID = 811
24	11010111	000111010000	001100101011	Gray = 215
…	…	…	…	…
30	11010111	000111010000	001100101011	Gray = 215
31	11010101	000111010000	001100101100	Gray = 213		Wall 4 H = 46.4 TID = 812
32	11010111	000111010000	001100101100	Gray = 215
…	…	…	…	…
47	11010101	000111010000	001100101100	Gray = 213

).

The buildings in Figure 7a,b have 37 and 47 façade superpixels, respectively, and the superpixels are stored for each pixel as shown in Table 2 and Table 3. The corresponding pixels, textures, and elevations of each facade of the building in Figure 7 can be obtained, and the detailed storage contents are shown in

Table 4. The building wall texture index in Figure 7a.

WallID	Wall Pixel Index	Texture Index	Hight
a1b1	1, 2, 3, 4, 5, 6, 7, 8	65	25.8
b1c1	9, 10, 11, 12, 13, 14, 15, 16, 17, 18	66	25.8
c1d1	19, 20, 21, 22, 23, 24, 25, 26, 27	67	25.8
d1a1	28, 29, 30, 31, 32, 33, 34, 35, 36, 37	68	25.8

and

Table 5. The building wall texture index in Figure 7b.

WallID	Wall Pixel Index	Texture Index	Hight
a2b2	1, 2, 3, 4, 5, 6, 7, 8, 9	809	46.4
b2c2	10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22	810	46.4
c2d2	23, 24, 25, 26, 27, 28, 29, 30	811	46.4
d2a2	31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47	812	46.4

.

3.2. Generation for DBM-Based Multiple-Building T²OMs

There are usually many buildings in a city. Therefore, the next step is to generate the multiple-building T²OM on the basis of the generation of a single-building T²OM. Part 2 in Figure 8 shows the process of generation for multiple-building T²OMs, in which each building is assigned a unique identification (BuindingID) and the building ID is used to control the display and hiding of each building (see

Table 6. Building relationships.

BuildingID	Type	RoofID	WallID	Properties	Others
B1	Volume	R11	W11, W12, W13…	Brick structure	…
B2	Volume	R12	W21, W22, W23…	reinforced concrete structure	…
B3	Volume	R13	W31, W32, W33…	steel structure	…
…	…	…	…	…	…

). In addition, the information of a single-building model is divided into top surface, elevation and bottom surface. Additionally, each roof and facade are assigned a separate identity (RoofID, WallID) (see

Table 7. The relationship of building roof textures.

RoofID	Type	TextureID	PointID	Others
R11	Polygon	TR11	PR11, PR12, PR13…	…
R12	Polygon	TR12	PR21, PR22, PR23…	…
R13	Polygon	TR13	PR31, PR32, PR33…	…
…	…	…	…	…

and

Table 8. The relationship of building wall textures.

WallID	Type	TextureID	PointID	Others
W11	Polygon	TW11	Pw11, Pw12, Pw13…	…
W12	Polygon	TW12	Pw21, Pw22, Pw23…	…
W13	Polygon	TW13	Pw31, Pw32, Pw33…	…
…	…	…	…	…

). The face IDs are associated with the building top surface table and building wall table, and each face point ID (Points) is recorded in the roof table and wall table, and the point IDs are associated with the building corner point information table (see

Table 9. Building vertex relationships.

PointID	X, Y, Z Coord.			Pixel Coord.		Others
Pw11	XW11	YW11	ZW11	IW11	JW11	…
Pw12	XW12	YW12	ZW12	IW12	JW12	…
Pw13	XW13	YW13	ZW13	IW13	JW13	…
…	…	…	…	…	…	…
PR11	XR11	YR11	ZR11	IR11	JR11	…
PR12	XR12	YR12	ZR12	IR12	JR12	…
PR13	XR13	YR13	ZR13	IR13	JR13	…
…	…	…	…	…	…	…

). The corner point information is expressed using multiple horizontal projection polygons, and each corner point is provided with 3D vector information in the corner point information table. Building textures are divided into top surface textures and wall textures in the table. In the building, the top surface table does not need to identify the top surface texture identity; its information can be associated with the 2D T²OM multiple-building data table in having the building identity (BuindingID) from which the top surface texture value and 3D coordinate information value of the building is obtained, and the real building top surface texture is obtained by rendering a single pixel at a single vector point in turn. For the wall texture of the building, the texture ID of the wall is associated with the texture data in the database. The texture name (TextureName), the address of the file uploaded in the computer (FileAddress), the data of the texture saved in binary (Binary), the format of the texture (Format), the size of the texture (Size) recorded in bytes (Byte) in the database, and the date when the texture was saved (Date) are recorded (see

Table 10. Data texture table.

TextureD	TextureName	FileAddress	Date	Fomat	Others
TW11	WTN11	WFA11	WD11	WF11	…
TW12	WTN12	WFA12	WD12	WF12	…
TW13	WTN13	WFA13	WD13	WF13	…
…	…	…	…	…	…
TR11	PTN11	PFA11	PD11	PF11	…
TR12	PTN12	PFA12	PD12	PF12	…
TR13	PTN13	PFA13	PD13	PF13	…
…	…	…	…	…	…

). Parts 1, 3 and 4 in Figure 8 are described in detail in Section 3.3 and Section 3.4.

3.3. Generation of DTM-Based T²OM

Part 1 in Figure 8 is DTM-based T²OM generation, which orthorectifies the displacement caused by terrestrial elevation, i.e., orthorectifies the terrains into an upright position in a given map coordination. Therefore, the digital differential orthorectification method is applied for this purpose. The details of this method can be found in [8,37]. Similarly, the given DTM data, data structure and data storage for each pixel in the DTM-based T²OM are similar to that of the DBM-based T²OM, but the building ID is assigned as “none”.

3.4. Merging DTM- and DBM-Based T²OMs

In view of the different structures of the DBM- and DTM-based T²OMs, an entire T²OM generation needs merging algorithms (see Figure 8). To do this, the logic operation <or> is performed with the superpixel ID of 0 or non-zero. In order to eliminate possible boundary confusion in the merging process, the following judgment conditions are executed: with the DTM-based T²OM as the base map (see Part 1 in Figure 8), when the same grid number appears in the DTM- and DBM-based T²OMs at the same time, only the DBM-based T²OM is retained (see Part 2 in Figure 8). This is because the building area determined by the horizontal projection polygon is not a regular rectangle or divided along the grid direction. Therefore, there are actually more grid elements located at the building boundary than at the real building boundary. Retaining the grid of the DBM-based T²OM will ensure the accuracy of the building location to the greatest extent. This is helpful for the 3D T²OM display. Merge the DTM-based and DBM-based true-squared orthophotos to obtain a near-true-squared image (see Part 3 in Figure 8). Linking the DBM model and wall textures to the building data enables the ability to display building façade textures and three-dimensional (3D) coordinates (XYZ) measurable at any point (see Part 4 in Figure 8).

4. Experiments and Analysis

Figure 9 is a flowchart for T²OM generation, divided into five parts. Part 1 shows two experimental datasets (high-resolution images, control points, orientation parameters, DBM and DTM) from Denver, CO, USA, and Nanning, China, which are used as input data. Part 2 shows DBM-based T²OM generation, which consists of orthorectifying both building roof and building facades, with which “superpixel” is used for the storage of building texture, building ID, etc. Part 3 shows DTM-based T²OM generation, which is for orthorectification of gentle and continuously elevated hilly areas. Part 4 shows the merging of DBM-based and DTM-based T²OMs as the output data, i.e., T²OM. Part 5 shows the accuracy evaluation for the generated T²OM using ground control points.

4.1. Metadata of T²OM

This experiment was implemented using the programming language C++. To store the T²OM in binary form, a file format fus needed to be designed to store superpixels due to the restriction of traditional bit-storing in computers.

Table 11. The file format of the T²OM fus.

File Section	Properties	Description
File flag block	m_FileProperty	Identifier “fus” (char type)
	m_Version	Version number (int type)
Image header information	m_UpleftCoordinateX	Image coordinate lower right X value (double type, units: meters)
	m_UpleftCoordinateY	Image coordinate lower right Y value (double type, units: meters)
	m_TMaxZ	The highest point in the DTM file (double type, in meters)
	m_TMinZ	The lowest point in the DTM file (double type, in meters)
	m_BMaxZ	Maximum building height in DBM (double type in meters)
	m_BMinZ	Minimum building height in DBM (double type in meters)
	m_IntervalX	Unit interval in X-axis direction (double type in meters)
	m_IntervalY	Unit interval in Y-axis direction (double type, units: meters)
	m_FileHigh	Image height (int type)
	m_FileWidth	Image width (int type)
	Z_Tresolution	Topographic data unit elevation level (double type, units: meters)
	Z_Bresolution	Building data unit elevation level (type double, in meters)
	Build_Num	Number of building objects elements (type int)
Image Pixels Information	T2OM_Grey	Pixel grey component (unsigned char type)
	T2OM_Ordinal	Subdivision grid order (unsigned char type)
	T2OM_Height	Elevation level high 8 bits (unsigned char type)
	T2OM_HI	Elevation level low 4 bits, logo high 4 bits (unsigned char type)
	T2OM_Index	Marker data low 8 bits (unsigned char type)

shows the entire file format, which consists of a file flag block, image header file information, and image pixel information. These three parts were written to the fus file in binary form.

In the superpixel data structure, information about the elevation level H and building identification ID need 12 bits of memory. However, there is no data type of this size in the computer. Therefore, 12 bits of information are saved by the bit operation. As shown in Figure 10, this method first opens up two unsigned short types of data (16 bit), TEM_Height and TEM_Index, and uses them to record H and ID, respectively. Then, the data are put into variables (T²OM_height, T²OM_hi and T²OM_index) through the bit operation. Finally, the information in the superpixel is combined, as shown on the right side of the figure.

4.2. T²OM Generation

In this section, this paper uses Dataset 1 and Dataset 2 to describe the generation process of generating T²OM and to verify the feasibility of the method in this paper.

4.2.1. Experimental Result with Dataset 1

The experimental Dataset 1 includes digital surface model data, aerial imagery data, and digital building model data. A brief description is given as follows:

(1)

DTM data: Figure 11a shows DTM data from Denver, CO, USA, which is represented as a height–depth map, where the darker the color is, the lower the height, and vice versa, because the topography of the city is relatively flat. Thus, the elevations shown on the ground are relatively similar (the colors shown are similar). The accuracy of plane surface coordinates and vertical coordinates are about 0.1 m and 0.2 m, respectively. The horizontal datum is GRS 1980, and the vertical datum is NAD83.

(2)

Aerial Image data: Figure 11b shows the original aerial image acquired using the RC30 aerial camera lens in Denver. The flight altitude in Denver is 1650 m higher than the mean ground elevation of the imaged area. Aerial photographs were initially recorded on film and then scanned into digital format at a pixel resolution of 25 μm.

(3)

DBM data: Figure 11c shows Denver DBM data, and these buildings with a ground resolution of about 25.4 cm per pixel were identified. Each building model contains building corner point information and elevation texture information.

Step 1. Generation of DBM-based T²OM

(1)

DBM-based building roof orthorectification

DBM-based roof orthorectification corrects only the displacements caused by the buildings and does not take into account the displacements caused by the terrain. Therefore, the generated DBM-based T²OM (see Figure 12) only corrects the building texture and not the texture of the terrain area, so the texture at the terrain is black (background value). Where (a) and (c) are two buildings in the T²OM, it can be seen that the roof textures of the buildings are obtained accurately.

(2)

DBM-based building façade orthorectification

In order to obtain the facade textures and 3D coordinates of the buildings, the orthorectification of the building facade textures was also performed in T²OM. The building facade textures for Dataset 1 were selected from the existing texture library and the same textures were used in all four directions, and the results of facade texture correction are shown in Figure 13. Figure 13a,b show the results of the façade texture correction for (a) and (c) in Figure 12, respectively, and the angles marked in Figure 13 are consistent with those in Figure 12.

Step 2. Generation of DTM-based T²OM

The correction of buildings needs to be followed by the orthorectification of non-building areas. The DTM-based differential correction of non-buildings is performed to obtain the corrected texture of the image. At the same time, superpixels are generated by overlaying each data information, and finally the DTM-based T²OM is obtained (see Figure 14). Because this part only corrects the terrain texture and not the building area texture, the texture at the building is black (background value). The DTM-based T²OM also has ghosts and shadows in the texture area, because Dataset 1 lacks complementary images, so occlusion detection, compensation for textures and shadow detection and compensation operations are not performed in this step.

Step 3. Merging DTM- and DBM-based T²OMs

Finally, the merging of the DBM with the DTM’s T²OM yields the result shown in Figure 15b, which fills the superpixels at the roof texture exactly to the area where the original background value is black. Figure 15a,c show enlarged views of two of the buildings.

4.2.2. Experimental Result with Dataset 2

The experimental Dataset 2 includes digital surface model data, aerial imagery data, and digital building model data. A brief description is given as follows.

(1)

DTM data: Figure 16a shows DTM data from Nanning, China, represented as a height–depth map, where the darker the color is, the lower the height, and vice versa, because the topography of the city is relatively flat. Thus, the elevations shown on the ground are relatively similar (the colors shown are similar). The accuracy of the plane surface coordinates and vertical coordinates are about 0.1 m and 0.2 m, respectively. The horizontal datum is GRS 1980, and the vertical datum is NAD83.

(2)

Aerial Image data: Figure 16b shows the original aerial image acquired using the CMOS lens in Nanning. The flight altitude in Nanning is 200 m higher than the average ground elevation of the imaging area.

(3)

DBM data: Figure 16c shows Nanning DBM data, and buildings with a ground resolution of about 25.4 cm per pixel were identified. Each building model contains building corner point information and elevation texture information.

Step 1. Generation of DBM-based T²OM

(1)

DBM-based building roof orthorectification

The DBM-based roof orthorectification corrects only the displacements caused by the buildings and does not take into account the displacements caused by the terrain. Therefore, the generated DBM-based T²OM (see Figure 17) only corrects the building texture and not the texture of the terrain area, so the texture of the terrain is black (background value). Four buildings of the T²OM are denoted as (a–d), and the roof textures of the buildings are obtained accurately.

(2)

DBM-based building façade orthorectification

In order to obtain the building facade texture and 3D coordinates, the building facade texture is also orthorectified in T²OM. The orthorectification results are shown in Figure 18. Figure 18a–d show the results of the façade texture correction for (a–d) in Figure 17, respectively, and the angles marked in Figure 18 are consistent with those in Figure 17.

Step 2. Generation of DTM-based T²OM

The correction of buildings needs to be followed by the orthorectification of non-building areas. The DTM-based differential correction of non-buildings is performed to obtain the corrected texture of the image. At the same time, superpixels are generated by overlaying each data information, and finally the DTM-based T²OM is obtained (see Figure 19). Because this part only corrects the terrain texture and not the building area texture, the texture of the building is black (background value). The DTM-based T²OM also has ghosting and shadows in the texture region, and the occlusion detection and compensation of textures and shadow detection and compensation operations in this paper are adopted from [9].

Step 3. Merging DTM- and DBM-based T²OMs

Finally, the merging of the DBM with the DTM’s T²OM yields the result shown in Figure 20, which fills the superpixels at the roof texture exactly to the area where the original background value is black. Figure 20a–d show enlarged views of four of the buildings.

4.2.3. T²OM 3D Measurement

With the 3D measurement function, the elevation of any point in the scene can be obtained. Figure 21 shows elevation information from a point on the selected facade. Figure 22 shows the color information, true 3D coordinates, and attribute information of each pixel in the acquired point. “Selective Hide” can also be used to display individual buildings, as shown in Figure 23. In addition, based on the fact that the horizontal projection polygon of a building can completely record building information, we can obtain complete information about corner points and facade through the “3D Building Information Display” function, as shown in Figure 24. The “Building Distance Measurement” function calculates the minimum and maximum distances between two buildings by using the building corner information, as shown in Figure 25.

4.3. Accuracy Evaluation and Analysis

In our method, T²OM generation is obtained by merging the DTM- and DBM-based T²OMs. There can be errors in multiple steps of the process. In addition, because the superpixel uses an elevation series instead of the double-form true elevation value, errors exist in the recording height of the superpixel. Based on the generated T²OM, this section evaluates the matching accuracy between the building horizontal projection polygons.

In order to ensure the accuracy of building location information in T²OM, it is necessary to evaluate the construction accuracy of the horizontal projection polygon. In Figure 26, the grid detected as the building edge is represented in blue. Building corners are represented in green. The horizontally projected polygon is displayed in red. As can be seen from the enlarged Figure 26e–g, the extracted blue edge is completely consistent with the green building corner. This proves that the accuracy of the building horizontal projection polygon construction is sufficient.

Four buildings identified in the T²OM data were randomly selected to evaluate the accuracy of information recording in T²OM, as shown in Figure 27. Firstly, roof corner points of each building above were extracted from the DBM. Then, the coordinates of the corner points in the DBM were compared with the grid number, subdivision, and elevation series recorded in the superpixels.

Table 12. 3D coordinate values before and after encoding.

Building	Xori	Yori	Zori	id(r,c)	h	Xbc	Ybc	Zbc
a	1286.850	1306.000	5551.700	14	1973	1286.840	1306.030	5551.600
	1346.850	1245.000	5551.700	14	1973	1346.840	1245.030	5551.600
	1355.000	1237.975	5551.700	241	1973	1355.030	1237.970	5551.600
	1356.000	1237.850	5551.700	209	1973	1356.030	1237.840	5551.600
b	1931.937	1424.000	5533.400	15	1876	1931.910	1424.030	5533.330
	1932.150	1440.000	5533.400	3	1876	1932.160	1440.030	5533.330
	1910.850	1462.000	5533.400	14	1876	1910.840	1462.030	5533.330
	1906.000	1464.150	5533.400	33	1876	1906.030	1464.160	5533.330
c	2990.850	1244.000	5462.500	14	1499	2990.840	1244.030	5462.330
	2912.850	1151.000	5462.500	14	1499	2912.840	1151.030	5462.330
	2911.850	1148.000	5462.500	14	1499	2911.840	1148.030	5462.330
	2964.850	1093.000	5462.500	14	1499	2964.840	1093.030	5462.330
d	3339.000	1882.850	5755.800	209	3057	3339.030	1882.840	5755.760
	3341.096	1883.000	5755.800	2	3057	3341.090	1883.030	5755.760
	3344.850	1881.000	5755.800	14	3057	3344.840	1881.030	5755.760
	3356.054	1864.000	5755.800	1	3057	3356.030	1864.030	5755.760

presents the 3D coordinates before and after coding, where x_ori, y_ori, and z_ori are the 3D coordinates before coding; S(r,c) and H are the subdivision and elevation series, respectively; and X_bc, Y_bc, and Z_bc are the 3D coordinates calculated using superpixels. Through calculations, the average errors of the X, Y and Z coordinate components were determined to be 0.017, 0.025, and 0.09 m, respectively. Compared with the resolution of the original data in plane coordinates of 0.1 m and the elevation of 0.2 m, it can be concluded that the coding method recorded by superpixels can greatly reduce the level of error generated in the process of converting vector data into grid data.

4.4. Discussion

From the above two sets of experimental results, it is concluded that our proposed T²OM can realize the switching display of two-dimensional flat TOMs and three-dimensional buildings, and use superpixels to save three-dimensional information so that the accuracy of three-dimensional measurement can be controlled within 0.0625 m.

In addition, it is feasible to expand the traditional pixel storage method to increase the amount of information expressed by a grid image. In our method, first, a large number of heterogeneous data are unified to achieve centralized management. This makes the reconstruction and display of a 3D model easier, because superpixels accurately store each location with elevation information. However, the proposed method still has defects. The main problem is that storage of the T²OM requires twice as much memory space as that of the TOM. This makes it difficult to store and transfer data, especially across large-scale urban areas. A solution is to compress the bit width of storage components, such as S, H and ID, by a statistical method. For example, in the compression of H, the height change curve for the whole study area can be counted, and the elevation series can be reduced from 4096 to 256. In addition, in areas without buildings, the removal of ID space to reduce memory consumption can be considered. Another defect that cannot be ignored is that the description of the geometric structure of building facades is not refined enough. On facades, there are balconies protruding from walls and windows recessed into walls. The geometric correction and line extraction of these facade structures is still difficult at present [56].

5. Conclusions

In light of the problem that traditional DOMs/TOMs only provide building roofs’ 2D (X,Y) attributes and gray information, and cannot provide 3D information or building facade textures at all, this paper proposes the generation of T²OM, which is radically different from the traditional generation of DOMs/TOMs, since the T²OM is able to provide three-dimensional (3D) and detailed textures of building roofs and facades.

The major innovation of this manuscript lies in the new method for the generation of T²OM, in which a data structure that can simultaneously store the 2D and 3D information of a building, building roof and building façade is developed. The proposed superpixel data structure takes the grid as the basic unit and successfully integrates a variety of data types by expanding the pixel storage space. The application of subdivision S and elevation series H greatly improves the accuracy of the 3D model. The proposed superpixel model is capable of promoting the fusion of multi-source heterogeneous data, so that a single image of data can display both 2D plane information and the 3D real scene. Moreover, the superpixel model can be applied to facade texture images, so that the 3D measurement of any point in a scene can be achieved. These contributions are valuable for large-scale urban DOM generation and applications.

Two sets of experimental results demonstrate that the proposed generation method of T²OM can maintain the traditional DOM/TOM characteristics, i.e., provide 2D XY coordinates and displaced building texture, but also provide the 3D XYZ coordinates of buildings’ roofs and facades. The accuracy of 3D measurement on a T²OM can achieve 0.025 m (0.3 pixel).

Nevertheless, the proposed method needs to be improved; for example, when the number of buildings in a city is large, if all of them are loaded into the memory according to the original texture data, the 3D display may occupy a large amount of memory and the refresh speed will be reduced. Therefore, memory loading according to the visible area and the compression of loaded memory are needed to reduce the memory occupancy and improve the refresh speed.