# 4D Building Reconstruction with Machine Learning and Historical Maps

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

#### Aims and Innovative Aspects

- -
- evaluation of multiple regressors to infer building heights from historical maps;
- -
- introduction of the geometric, neighbourhood, and categorical features usable when only digitised building footprints are available;
- -
- testing and evaluation of the proposed method on two different locations and multi-temporal historical maps; and
- -
- the realisation of 4D level of detail 1 (LoD1) building models for the geo-visualisation enhancement, creating a 4D cadastre, and spatial analysis purposes (e.g., volumetric density studies.

## 2. Related Works

## 3. Data and Case Studies

#### 3.1. Trento (Italy) Historic City Centre

#### 3.2. Bologna (Italy) Historic City Centre

## 4. Methodology

#### 4.1. Machine and Deep Learning for Regression Models

- (a).
- Ordinary Least Squares Linear regressor: this model minimises the residual sum of squares between the observed and target variables. It assumes a linear connection between outputs and predictor variables, and it is sensitive to random errors when variables are not independent [49].
- (b).
- Random Forest regressor: it is a supervised learning algorithm based on ensemble learning. Random Forest combines multiple decision trees (reducing the variance and overfitting) resulting in an averaged prediction of the individual classifiers. It also provides straightforward methods for the features’ importance analysis and selection [50].
- (c).
- CatBoost regressor: it uses gradient boosting on decision trees. The decision tree is used as a weak base learner, while gradient boosting iteratively fits a sequence of these trees [51].
- (d).
- Support Vector regressor with the Radius Basis Function (RBF) kernel: it produces a model depending only on a subset of the training data. The employed cost function ignores samples whose prediction is close to their target [52].
- (e).
- Multilayer Perceptron regressor: it is a neural network model where neurons are arranged in different layers, connected by differently weighted joints [53]. The model optimises the squared loss using, e.g., stochastic gradient descent.

#### 4.2. Predictors

- -
- Geometric predictors:
- (1).
- Area: defined as the building footprint area;
- (2).
- Perimeter: defined as the footprint perimeter;
- (3).
- NPI: the normalised perimeter index, an indicator of the polygon shape complexity. It is computed as the ratio of the perimeter of the equal-area circle and the perimeter of the shape (1):$$NPI=2{\displaystyle \sum}_{}^{}\frac{\pi A}{p}$$
- (4).
- Vertices: the number of vertices of a digitised polygon (Figure 6a);
- (5).
- Length MBR: the length of the minimum bounding rectangle (MBR) of a footprint;
- (6).
- Width MBR: the width of the minimum bounding rectangle (MBR) of a footprint;
- (7).
- Area MBR: the area of the minimum bounding rectangle (MBR) of a footprint;
- (8).
- Ratio: the ratio between the area of a footprint and the area of the corresponding minimum boundary rectangle (MBR).

- -
- Neighbourhood predictors:
- (9).
- Neighbours: defined as the number of adjacent polygons;
- (10).
- Distance: the distance of a polygon’s centroid from the nearest centroid (Figure 6b);
- (11).
- Density: the kernel density values (Figure 6c), considering four different estimation radii (50 m, 100 m, 150 m, 200 m), defined as (2):$$Density=\frac{1}{{\left(radius\right)}^{2}}{\displaystyle \sum}_{i=1}^{n}\sqrt{[\frac{3}{\pi}pop(1-{(\frac{dist}{radiu{s}^{2}})}^{2})}]$$

- -
- Positional and categorical predictors:
- (12).
- Position (X, Y): the planar position of each polygon centroid within the map;
- (13).
- Group: the aggregation of polygons in building blocks (Figure 6d). A “group” value is assigned to each polygon belonging to the same building block, while isolated buildings are grouped;
- (14).
- Class: defines a building of specific historical value, such as churches, palaces, castle, and tower;
- (15).
- Function: defines the civil or religious function of the buildings. In our cases, civil buildings were also grouped considering their approximative period of construction, derived by comparing multi-temporal historical maps;
- (16).
- Towers: includes a shape-based classification of the civil and religious towers, i.e., circular, rectangular, or octagonal shape. In our cases, we noticed that towers with similar shapes featured similar heights.

#### 4.3. Data Augmentation

#### 4.4. Heights Prediction Metrics

#### 4.4.1. Evaluation of the RMSE, MAE, and R^{2} on Randomly Split Training and Test Data

^{2}(coefficient of determination) (5). They are respectively defined as:

#### 4.4.2. Single-View Metrology from Historical Images

_{C}) and any other distance (H

_{U}) between two planes perpendicular to the reference direction (v

_{3}) can be derived. Two points B and T, lying on two planes P and P’ perpendicular to the reference direction v

_{3}, are represented in image space by points b and t lying on the two planes defined by the two vanishing points v

_{1}and v

_{2}. The image point lies at the intersection of the line joining the corresponding points C (the camera centre) and C’ with the vanishing line l

_{v}

_{1v2}. The point C lies on a plane at a distance H

_{C}from the reference plane P. Under this configuration, the image points b, t, c, and v

_{3}are aligned along the vertical reference direction, and therefore they define a cross-ratio. The ratio also holds in object space with points B, T, C’, and v

_{3}. Therefore, we can derive (6):

#### 4.5. Accuracy Aims

## 5. Results

#### 5.1. Regressors Evaluation

#### 5.2. Inferring Building Heights from a Historical Map—Trento Case Study

#### 5.3. Inferring Building Heights from a Historical Map—Bologna Case Study

## 6. Discussion

- (a).
- data preparation (i.e., the digitisation of historical maps) is demanding and time-consuming;
- (b).
- differences in the input data (i.e., different informative levels among the training data and the historical datasets) can affect the quality of the prediction;
- (c).
- the results can be influenced by the accuracy of the georeferenced maps, considering that positional attributes are included among the predictors; and
- (d).
- the method was tested on similar urban scenarios and using respective actual data as training. The applicability in different regional contexts and the prediction’s quality employing other cities’ training data need further investigations.

## 7. Conclusions

- (a).
- automatic methods and deep learning techniques for replacing the time-consuming digitisation procedure of historical maps;
- (b).
- the use of specific training data (e.g., prepared at a building-block level rather than using detailed cadastral maps) for historical datasets suffering from a low informative level;
- (c).
- the prediction response assigning a lower weight to positional attributes, for avoiding possible mismatches related to different-scale maps and georeferencing issues; and
- (d).
- the possible generalisation of the method, expanding the training set with data representative of different regional contexts, and applying the trained model in actual scenarios where no elevation data are available (e.g., remote areas).

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Döllner, J.; Kolbe, T.H.; Liecke, F.; Sgouros, T.; Teichmann, K. The virtual 3D city model of Berlin: Managing, integrating, and communicating complex urban information. In Proceedings of the 25th Urban Data Management Symposium, Aalborg, Denmark, 15–17 May 2006; pp. 15–17. [Google Scholar]
- Kersten, T.P.; Keller, F.; Sänger, J.; Schiewe, J. Automated Generation of an Historic 4D City Model of Hamburg and Its Visualisation with the GE Engine; Springer: Berlin/Heidelberg, Germany, 2012; Volume 7616. [Google Scholar]
- Singh, S.; Jain, K.; Mandla, V.R. Virtual 3D city modeling: Techniques and applications. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2013**, 73–91. [Google Scholar] [CrossRef] [Green Version] - Kocaman, S.; Akca, D.; Poli, D.; Remondino, F. 3D/4D City Modelling—From Sensors to Applications; Whittles Publishing: Scotland, UK, 2020; ISBN 978-184995-475-4. [Google Scholar]
- Biljecki, F.; Stoter, J.; Ledoux, H.; Zlatanova, S.; Çöltekin, A. Applications of 3D city models: State of the art review. ISPRS Int. J. Geo-Inf.
**2015**, 4, 2842–2889. [Google Scholar] [CrossRef] [Green Version] - Nocerino, E.; Menna, F.; Remondino, F. Multi-temporal analysis of landscapes and urban areas. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2012**, 39, B4. [Google Scholar] [CrossRef] [Green Version] - Haala, N.; Kada, M. An update on automatic 3D building reconstruction. ISPRS J. Photogramm. Remote Sens.
**2010**, 65, 570–580. [Google Scholar] [CrossRef] - Haala, N.; Rothermel, M.; Cavegn, S. Extracting 3D urban models from oblique aerial images. In Proceedings of the 2015 Joint Urban Remote Sensing Event (JURSE), Lausanne, Switzerland, 30 March—1 April 2015; pp. 1–4. [Google Scholar]
- Buyukdemircioglu, M.; Kocaman, S.; Isikdag, U. Semi-automatic 3D city model generation from large-format aerial images. ISPRS Int. J. Geo-Inf.
**2018**, 7, 339. [Google Scholar] [CrossRef] [Green Version] - Chen, R. The development of 3D city model and its applications in urban planning. In Proceedings of the 2011 19th International Conference on Geoinformatics, Shanghai, China, 24–26 June 2011; pp. 1–5. [Google Scholar]
- Ghassoun, Y.; Löwner, M.-O.; Weber, S. Exploring the benefits of 3D city models in the field of urban particles distribution modelling—A comparison of model results. In 3D Geoinformation Science; Springer: Berlin/Heidelberg, Germany, 2015; pp. 193–205. [Google Scholar]
- Willenborg, B.; Sindram, M.; Kolbe, T.H. Applications of 3D city models for a better understanding of the built environment. In Trends in Spatial Analysis and Modelling; Springer: Berlin/Heidelberg, Germany, 2018; pp. 167–191. [Google Scholar]
- Tomljenovic, I.; Höfle, B.; Tiede, D.; Blaschke, T. Building extraction from airborne laser scanning data: An analysis of the state of the art. Remote Sens.
**2015**, 7, 3826–3862. [Google Scholar] [CrossRef] [Green Version] - Yalcin, G.; Selcuk, O. 3D city modelling with Oblique Photogrammetry Method. Procedia Technol.
**2015**, 19, 424–431. [Google Scholar] [CrossRef] [Green Version] - Ali, I.; Khan, A.A.; Qureshi, S.; Umar, M.; Haase, D. 3D Geoinformation Science, Lecture Notes in Geoinformation and Cartographyitle; Breunig, M., Al-Doori, M., Butwilowski, E., Kuper, P.V., Benner, J., Haefele, K.H., Eds.; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
- Arroyo Ohori, K.; Ledoux, H.; Stoter, J. A dimension-independent extrusion algorithm using generalisedgeneralised maps. Int. J. Geogr. Inf. Sci.
**2015**, 29, 1166–1186. [Google Scholar] [CrossRef] - Smelik, R.M.; Tutenel, T.; Bidarra, R.; Benes, B. A survey on procedural modelling for virtual worlds. Comput. Graph. Forum
**2014**, 33, 31–50. [Google Scholar] [CrossRef] - Goetz, M.; Zipf, A. Towards defining a framework for the automatic derivation of 3D CityGML models from volunteered geographic information. Int. J. 3D Inf. Model.
**2012**, 1, 1–16. [Google Scholar] [CrossRef] - Biljecki, F.; Ledoux, H.; Stoter, J. An improved LOD specification for 3D building models. Comput. Environ. Urban Syst.
**2016**, 59, 25–37. [Google Scholar] [CrossRef] [Green Version] - Gröger, G.; Plümer, L. CityGML—Interoperable semantic 3D city models. ISPRS J. Photogramm. Remote Sens.
**2012**, 71, 12–33. [Google Scholar] [CrossRef] - Gröger, G.; Kolbe, T.H.; Nagel, C.; Häfele, K.H. OGC City Geography Markup Language (CityGML) Encoding Standard; Version 2.0.0, OGC 08-007r2; Open Geospatial Consortium: Wayland, MA, USA, 2012. [Google Scholar]
- Ledoux, H.; Ohori, K.A.; Kumar, K.; Dukai, B.; Labetski, A.; Vitalis, S. CityJSON: A compact and easy-to-use encoding of the CityGML data model. Open Geospat. Data Softw. Stand.
**2019**, 4, 4. [Google Scholar] [CrossRef] - Ledoux, H.; Meijers, M. Topologically consistent 3D city models obtained by extrusion. Int. J. Geogr. Inf. Sci.
**2011**, 25, 557–574. [Google Scholar] [CrossRef] [Green Version] - Shi, Y.; He, B. Creating Topologically Consistent 3D City Models of LOD+ with Extrusion. In Proceedings of the International Conference on Computer and Computing Technologies in Agriculture, Zhangjiajie, China, 19–21 October 2012; pp. 203–210. [Google Scholar]
- Fan, H.; Zipf, A. Modelling the world in 3D from VGI/Crowdsourced data. In European Handbook of Crowdsourced Geographic Information; Ubiquity Press: London, UK, 2016; pp. 435–466. [Google Scholar]
- Brasebin, M.; Perret, J.; Mustière, S.; Weber, C. A generic model to exploit urban regulation knowledge. ISPRS Int. J. Geo-Inf.
**2016**, 5, 14. [Google Scholar] [CrossRef] [Green Version] - Peeters, A. A GIS-based method for modeling urban-climate parameters using automated recognition of shadows cast by buildings. Comput. Environ. Urban Syst.
**2016**, 59, 107–115. [Google Scholar] [CrossRef] - Biljecki, F.; Ledoux, H.; Stoter, J. Generating 3D city models without elevation data. Comput. Environ. Urban Syst.
**2017**, 64, 1–18. [Google Scholar] [CrossRef] [Green Version] - Rook, M.; Biljecki, F.; Diakité, A.A. Towards Automatic Semantic Labelling of 3D City Models. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2016**, 4. [Google Scholar] [CrossRef] - Wichmann, A.; Agoub, A.; Kada, M. Roofn3D: Deep Learning Training Data for 3D Building Reconstruction. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2018**, 42, 1191–1198. [Google Scholar] [CrossRef] [Green Version] - Agoub, A.; Schmidt, V.; Kada, M. Generating 3D City Models Based on the Semantic Segmentation of Lidar Data Using Convolutional Neural Networks. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2019**, 4, 3–10. [Google Scholar] [CrossRef] [Green Version] - Tripodi, S.; Duan, L.; Trastour, F.; Poujad, V.; Laurore, L.; Tarabalka, Y. Deep learning-based extraction of building contours for large-scale 3D urban reconstruction. In Proceedings of the Image and Signal Processing for Remote Sensing XXV, Strasbourg, France, 9–11 September 2019; Volume 11155, p. 111550O. [Google Scholar]
- Lee, S.; Jung, S.; Lee, J. Prediction model based on an artificial neural network for user-based building energy consumption in South Korea. Energies
**2019**, 12, 608. [Google Scholar] [CrossRef] [Green Version] - Benavente-Peces, C.; Ibadah, N. Buildings Energy Efficiency Analysis and Classification Using Various Machine Learning Technique Classifiers. Energies
**2020**, 13, 3497. [Google Scholar] [CrossRef] - Mohammadiziazi, R.; Bilec, M.M. Application of machine learning for predicting building energy use at different temporal and spatial resolution under climate change in USA. Buildings
**2020**, 10, 139. [Google Scholar] [CrossRef] - Wu, Y.; Filippovska, Y.; Schmidt, V.; Kada, M. Application of Deep Learning for 3D building generalization. In Proceedings of the 29th International Cartographic Conference (ICC 2019), Tokyo, Japan, 15–20 July 2019. [Google Scholar]
- Tooke, T.R.; Coops, N.C.; Webster, J. Predicting building ages from LiDAR data with random forests for building energy modeling. Energy Build.
**2014**, 68, 603–610. [Google Scholar] [CrossRef] - Biljecki, F.; Sindram, M. Estimating building age with 3D GIS. In Proceedings of the 12th International 3D GeoInfo Conference 2017, Melbourne, Australia, 26–27 October 2017; pp. 17–24. [Google Scholar]
- Zeppelzauer, M.; Despotovic, M.; Sakeena, M.; Koch, D.; Döller, M. Automatic prediction of building age from photographs. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, Yokohama, Japan, 11–14 June 2018; pp. 126–134. [Google Scholar]
- Mahajan, N.; Patil, D.; Kotkar, A.; Wasnik, K. Prediction of Building Structure Age Using Machine Learning. Int. J. Adv. Res. Ideas Innov. Technol.
**2019**, 5, 232–234. [Google Scholar] - Mou, L.; Zhu, X.X. IM2HEIGHT: Height estimation from single monocular imagery via fully residual convolutional-deconvolutional network. arXiv
**2018**, arXiv:1802.10249. [Google Scholar] - Park, Y.; Guldmann, J.-M. Creating 3D city models with building footprints and LIDAR point cloud classification: A machine learning approach. Comput. Environ. Urban Syst.
**2019**, 75, 76–89. [Google Scholar] [CrossRef] - Liu, C.-J.; Krylov, V.A.; Kane, P.; Kavanagh, G.; Dahyot, R. IM2ELEVATION: Building Height Estimation from Single-View Aerial Imagery. Remote Sens.
**2020**, 12, 2719. [Google Scholar] [CrossRef] - Mahmud, J.; Price, T.; Bapat, A.; Frahm, J.-M. Boundary-Aware 3D Building Reconstruction from a Single Overhead Image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 441–451. [Google Scholar]
- Kapoor, A.; Larco, H.; Kiveris, R. Nostalgin: Extracting 3D City Models from Historical Image Data. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 4–8 August 2019; pp. 2565–2575. [Google Scholar]
- Anh, P.; Thanh, N.T.N.; Vu, C.T.; Ha, N.V.; Hung, B.Q. Preliminary Result of 3D City Modelling for Hanoi, Vietnam. In Proceedings of the 2018 5th NAFOSTED Conference on Information and Computer Science (NICS), Ho Chi Minh City, Vietnam, 23–24 November 2018; pp. 294–299. [Google Scholar]
- Choi, R.Y.; Coyner, A.S.; Kalpathy-Cramer, J.; Chiang, M.F.; Campbell, J.P. Introduction to machine learning, neural networks, and deep learning. Transl. Vis. Sci. Technol.
**2020**, 9, 14. [Google Scholar] - Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res.
**2011**, 12, 2825–2830. [Google Scholar] - Rong, S.; Bao-wen, Z. The research of regression model in machine learning field. In Proceedings of the MATEC Web of Conferences 2018 6th International Forum on Industrial Design (IFID 2018), Luoyang, China, 18–20 May 2018; Volume 176, p. 1033. [Google Scholar]
- Breiman, L. Random Forests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef] [Green Version] - Hancock, J.T.; Khoshgoftaar, T.M. CatBoost for big data: An interdisciplinary review. J. Big Data
**2020**, 7, 1–45. [Google Scholar] [CrossRef] - Awad, M.; Khanna, R. Support vector regression. In Efficient Learning Machines; Springer: Berlin/Heidelberg, Germany, 2015; pp. 67–80. [Google Scholar]
- Murtagh, F. Multilayer perceptrons for classification and regression. Neurocomputing
**1991**, 2, 183–197. [Google Scholar] [CrossRef] - Wong, S.C.; Gatt, A.; Stamatescu, V.; McDonnell, M.D. Understanding data augmentation for classification: When to warp? In Proceedings of the 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Gold Coast, Australia, 30 November–2 December 2016; pp. 1–6. [Google Scholar]
- Perez, L.; Wang, J. The effectiveness of data augmentation in image classification using deep learning. arXiv
**2017**, arXiv:1712.04621. [Google Scholar] - Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data
**2019**, 6, 60. [Google Scholar] [CrossRef] - Zheng, Q.; Yang, M.; Tian, X.; Jiang, N.; Wang, D. A Full Stage Data Augmentation Method in Deep Convolutional Neural Network for Natural Image Classification. Discret. Dyn. Nat. Soc.
**2020**, 2020, 4706576. [Google Scholar] [CrossRef] - Paschali, M.; Simson, W.; Roy, A.G.; Göbl, R.; Wachinger, C.; Navab, N. Manifold Exploring Data Augmentation with Geometric Transformations for Increased Performance and Robustness. In Proceedings of the International Conference on Information Processing in Medical Imaging, Hong Kong, China, 2–7 June 2019; pp. 517–529. [Google Scholar]
- Kim, E.K.; Lee, H.; Kim, J.Y.; Kim, S. Data Augmentation Method by Applying Color Perturbation of Inverse PSNR and Geometric Transformations for Object Recognition Based on Deep Learning. Appl. Sci.
**2020**, 10, 3755. [Google Scholar] [CrossRef] - Criminisi, A.; Reid, I.; Zisserman, A. Single view geometry. Int. J. Comput. Vis.
**2000**, 40, 123–148. [Google Scholar] [CrossRef] - Remondino, F. Recovering metric information from old monocular video sequences. In Proceedings of the 6th Conference on Optical 3D Measurement Techniques, Zürich, Switzerland, 22–25 September 2003. [Google Scholar]
- Semple, J.G.; Kneebone, G.T. Algebraic Projective Geometry; Oxford University Press: Oxford, UK, 1998. [Google Scholar]

**Figure 1.**The four digitised historical maps of Trento (1851, 1887, 1908, and 1936). Please note the different levels of detail of building footprints, e.g., between 1851 and 1887: in the latter case, footprints are bigger and include multiple buildings with regard to the other maps.

**Figure 2.**A view of the level of detail 1 (LoD1) buildings in Trento generated from the actual (2016) topographic data available as open data. Height values are referred to as the mean level of the pitched roofs.

**Figure 4.**A view of the LoD1 buildings in Bologna generated from the actual (2017) topographic data available as open data.

**Figure 6.**Examples of attributes (predictors) computed for the Trento 1851 dataset: number of vertices (

**a**); the distance of the polygon centroids from the nearest centroids (

**b**); kernel density value—radius 100 m (

**c**); groups (

**d**). In (

**a**) and (

**d**) each colour corresponds to the computed (

**a**) or assigned (

**d**) numerical value. In (

**b**) and (

**c**), classes are colourised with a graduated scale.

**Figure 7.**Alignment of four points defining a cross-ratio invariant in image and object space (

**top**). An example of height computation, deriving first the three vanishing points and then the unknown distance H

_{U}from the known H (

**bottom**).

**Figure 8.**3D view of the inferred building heights (orange) with respect to the ground truth data (white) for Trento. Despite metrics indicating acceptable accuracy (Table 5), a visual check highlights gross errors mainly on towers.

**Figure 9.**Data distribution before (

**left**) and after (

**right**) adding synthetic data for the under-represented classes in the Trento dataset.

**Figure 10.**3D view of the inferred building heights (orange) in Trento with respect to the ground truth data (white) after data augmentation. The visual inspection shows a relevant reduction of the errors on towers and churches.

**Figure 11.**Two examples of single-view-metrology applied to historical photos in Trento to determine heights of buildings not present anymore in the actual topographic database.

**Figure 12.**Overviews of the generated multi-temporal 3D buildings (LoD1) using machine learning and historical maps for the Trento case study.

**Figure 15.**Multi-temporal 3D reconstruction of Bologna with building heights inferred using machine learning and historical maps (1884 and 1945).

**Figure 16.**Predictors importance of the Random Forest method in the Trento dataset. The most relevant are function (e.g., civil or religious), class (e.g., churches, palaces, castle), and distance (from a polygon’s centroid from the nearest centroid). On the y-axis, the features importance score is given, i.e., the normalised average scores indicating the weighted variance decrease in a decision tree.

**Figure 18.**3D view of the inferred building heights (pink) in Trento with respect to the ground truth data (white), using the actual Bologna dataset as training. Training contains augmented data.

**Figure 19.**3D view of the inferred building heights (pink) in Trento with respect to the ground truth data (white), using the actual Bologna dataset as training. Data augmentation was removed from the training set.

**Table 1.**The number of polygons and their average area in the four digitised historical maps and in the actual topographic data (2016).

Dataset | Total n. of Polygons | Average Polygons Area (m^{2}) |
---|---|---|

1851 | 1274 | 197.18 |

1887 | 632 | 499.97 |

1908 | 1685 | 238.24 |

1936 | 3112 | 236.62 |

Actual | 4537 | 149.87 |

Dataset | Total n. of Polygons | Average Polygons Area (m^{2}) | Average Height (m) | Median Height (m) | St. Deviation (m) |
---|---|---|---|---|---|

Actual | 4537 | 149.87 | 12.26 | 12.41 | 5.36 |

**Table 3.**The number of polygons and their average area in the two digitised maps and in the actual topographic data (2017).

Dataset | Total n. of Polygons | Average Polygons Area (m^{2}) |
---|---|---|

1884 | 482 | 1750.35 |

1945 | 1174 | 738.62 |

Actual | 3241 | 215.04 |

Dataset | Total n. of Polygons | Average Polygons Area (m^{2}) | Average Height (m) | Median Height (m) | St. Deviation (m) |
---|---|---|---|---|---|

Actual | 3241 | 215.04 | 14.71 | 14.00 | 6.43 |

Regressor | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|

Linear | 5.56 | 4.38 | −0.08 | 0.68 | 6.13 |

Random Forest | 3.79 | 2.88 | 0.49 | −0.05 | 2.41 |

CatBoost | 4.03 | 3.05 | 0.43 | −0.02 | 3.05 |

Support Vector | 4.88 | 3.81 | 0.16 | 0.00 | 4.69 |

Multilayer Perceptron | 4.26 | 3.20 | 0.36 | −0.09 | 3.64 |

Regressor | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|

Linear | 7.64 | 6.06 | −0.36 | −0.87 | 7.57 |

Random Forest | 4.67 | 3.64 | 0.49 | −0.06 | 2.95 |

CatBoost | 4.76 | 3.69 | 0.47 | 0.00 | 2.70 |

Support Vector | 4.88 | 4.10 | 0.33 | 0.05 | 5.03 |

Multilayer Perceptron | 4.94 | 3.18 | 0.43 | 0.07 | 4.31 |

Dataset | Total n. of Polygons | Towers | Churches | Civil Buildings |
---|---|---|---|---|

Trento | 4537 | 30 (~1%) | 53 (~1%) | 4454 (~98%) |

Bologna | 3241 | 30 (~1%) | 21 (~1%) | 3190 (~98%) |

Dataset Data Augmentation | Total n. of Polygons | Towers | Churches | Civil Buildings |
---|---|---|---|---|

Trento | 5369 | 379 (~7%) | 531 (~10%) | 4454 (~83%) |

Bologna | 3553 | 176 (~5%) | 136 (~4%) | 3241 (~91%) |

**Table 9.**Accuracy evaluation of the regressor methods on the Trento dataset after data augmentation.

Regressor | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|

Linear | 5.90 | 4.69 | −0.15 | 1.15 | 6.66 |

Random Forest | 3.59 | 2.83 | 0.57 | 0.00 | 2.07 |

CatBoost | 3.76 | 2.95 | 0.53 | 0.00 | 2.61 |

Support Vector | 5.30 | 4.24 | 0.07 | −0.06 | 5.44 |

Multilayer Perceptron | 4.17 | 3.20 | 0.43 | −0.05 | 3.57 |

**Table 10.**Accuracy evaluation of the regressor methods on the Bologna dataset after data augmentation.

Regressor | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|

Linear | 8.52 | 6.81 | 0.04 | −0.75 | 8.51 |

Random Forest | 4.52 | 3.54 | 0.49 | 0.00 | 2.72 |

CatBoost | 4.64 | 3.66 | 0.52 | 0.00 | 2.49 |

Support Vector | 5.15 | 3.49 | 0.41 | 0.00 | 4.78 |

Multilayer Perceptron | 4.83 | 3.79 | 0.48 | −0.02 | 3.91 |

**Table 11.**Metrics evaluation of the Random Forest performance on the four historical datasets of Trento, considering twenty unaltered buildings digitised in all the maps as the test set.

Historical Map-Year | RMSE (m) | MAE (m) | R^{2} | Min Error (m) | Max Error (m) | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|---|

1851 | 0.96 | 1.13 | 0.97 | 2.95 | 1.75 | 1.09 | 0.74 |

1887 | 2.86 | 2.25 | 0.92 | 0.07 | 6.35 | 1.52 | 1.77 |

1908 | 1.80 | 1.50 | 0.96 | −2.78 | 3.91 | 1.18 | 0.99 |

1936 | 1.67 | 1.40 | 0.97 | −2.62 | 3.20 | 1.14 | 0.90 |

**Table 12.**Metrics evaluation of the Catboost performance on the four historical datasets of Trento, considering twenty unaltered buildings as the test set.

Historical Map-Year | RMSE (m) | MAE (m) | R^{2} | Min Error (m) | Max Error (m) | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|---|

1851 | 4.91 | 3.22 | 0.83 | −2.93 | 13.87 | 2.02 | 3.71 |

1887 | 6.09 | 3.93 | 0.67 | 0.24 | 17.15 | 2.31 | 4.65 |

1908 | 4.66 | 3.08 | 0.81 | −3.12 | 14.38 | 2.11 | 3.49 |

1936 | 5.80 | 3.31 | 0.70 | −2.72 | 19.02 | 1.38 | 4.76 |

**Table 13.**Evaluation on eight disappeared buildings visible in historical photos: heights predicted with Random Forest were compared with single-view-metrology heights and metrics derived.

Dataset | RMSE (m) | MAE (m) | Median (m) | St. Dev. (m) |
---|---|---|---|---|

Trento | 1.41 | 1.28 | −0.83 | 1.29 |

**Table 14.**Metric evaluation of the Random Forest prediction on the two historical datasets of Bologna, considering as a test set ten unaltered buildings digitised in both maps.

Historical Map-Year | RMSE (m) | MAE (m) | R^{2} | Min Error (m) | Max Error (m) | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|---|

1884 | 2.67 | 2.35 | 0.97 | 2.10 | 1.09 | 0.73 | 0.54 |

1945 | 1.71 | 1.63 | 0.99 | −1.66 | 6.35 | 1.52 | 1.89 |

**Table 15.**Metric evaluation of the Catboost prediction on the two historical datasets and ten unaltered buildings as the test set.

Historical Map-Year | RMSE (m) | MAE (m) | R^{2} | Min Error (m) | Max Error (m) | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|---|

1884 | 7.56 | 6.81 | 0.88 | −2.30 | 10.92 | 8.39 | 3.28 |

1945 | 7.98 | 7.05 | 0.83 | −1.98 | 12.38 | 7.76 | 3.70 |

**Table 16.**Evaluation metrics for the Trento dataset with and without a recursive feature elimination (RFE) approach.

Regressor | N. of Features | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | St. Dev. (m) |
---|---|---|---|---|---|

Random Forest—without RFE | 20 | 3.59 | 2.83 | 0.57 | 2.07 |

Random Forest—with RFE | 10 | 3.83 | 2.99 | 0.51 | 2.18 |

Regressor | N. of Features | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | St. Dev. (m) |
---|---|---|---|---|---|

Random Forest—without RFE | 20 | 4.52 | 3.54 | 0.49 | 2.72 |

Random Forest—with RFE | 10 | 4.59 | 3.54 | 0.53 | 2.76 |

**Table 18.**Accuracy evaluation with the Random Forest regressor, training on Bologna data, and predicting on Trento dataset, and vice-versa. In this test, positional attributes were removed, and augmented data were employed for both cases.

Dataset Prediction | Dataset Training | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|

Trento | Bologna | 4.60 | 3.57 | 0.53 | 0.00 | 2.77 |

Bologna | Trento | 3.73 | 2.95 | 0.54 | 0.00 | 2.15 |

**Table 19.**Accuracy evaluation with the Random Forest regressor and inverted training data. In this case, augmented data were removed from the training.

Dataset Prediction | Dataset Training | RMSE TEST (m) | MAE TEST (m) | R^{2} TEST | Median (m) | St. Dev. (m) |
---|---|---|---|---|---|---|

Trento | Bologna | 3.90 | 2.98 | 0.47 | −0.03 | 2.49 |

Bologna | Trento | 4.75 | 3.70 | 0.47 | −0.09 | 2.99 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Farella, E.M.; Özdemir, E.; Remondino, F.
4D Building Reconstruction with Machine Learning and Historical Maps. *Appl. Sci.* **2021**, *11*, 1445.
https://doi.org/10.3390/app11041445

**AMA Style**

Farella EM, Özdemir E, Remondino F.
4D Building Reconstruction with Machine Learning and Historical Maps. *Applied Sciences*. 2021; 11(4):1445.
https://doi.org/10.3390/app11041445

**Chicago/Turabian Style**

Farella, Elisa Mariarosaria, Emre Özdemir, and Fabio Remondino.
2021. "4D Building Reconstruction with Machine Learning and Historical Maps" *Applied Sciences* 11, no. 4: 1445.
https://doi.org/10.3390/app11041445