Structure-from-Motion (SFM) Photogrammetry as a Non-Invasive Methodology to Digitalize Historical Documents: A Highly Flexible and Low-Cost Approach?

Abstract

Historical documents represent a significant part of the world cultural heritage and need to be preserved from physical deformation due to ageing. The restoration of fragile documents requires economic resources that are often limited to only preserve the integrity of exceptional and highly valuable historical records. On the other hand, regeneration of ancient documents in digital form is a useful way to preserve them regardless of the material they are made of. In addition, the digitization of historical cartography allows creating a valuable dataset for a variety of GIS applications as well as spatial and landscape studies. Nonetheless, historical maps are usually deformed, and a contact-scanning process could damage them because this method requires planar positioning of the map. In this regard, photogrammetry has been used successfully as a non-invasive method to digitize historical documentation. The purpose of this research is to assess a low-cost and highly flexible strategy to digitize historical maps and documents through digital photogrammetry using low-cost commercial off-the-shelf sensors. This methodology allows training a wider audience of cultural heritage operators in digitizing historic records with a millimeter-level accuracy.

1. Introduction

Historical documents represent an important source of information about our past as well as an invaluable component of our cultural heritage. In particular, in recent years historical cartography has increasingly become a useful source for many research fields such as geomorphology [1,2], geoarchaeology [3,4] and geoheritage [5] to understand the human-induced modification on the landscape through time. Ageing is the main problem for the conservation of fragile material used to create this kind of geographic datasets. The regeneration of historic maps in digital form permits the preservation of these documents as cartographic heritage and to use it in GIS spatial analysis, for instance [6,7].

The conventional approach to digitize historical cartography consists of using the planar scanner, as still indicated for example, by the Central Institute for the Union Catalogue of Italian Libraries and Bibliographic Information [8]. Nevertheless, this method presents many issues: The archival maps may be fragile and difficult to handle, and sometimes the large size of historical maps may also require expensive digital scanners. Moreover, quite often, the digitization by flatbed scanners does not represent a suitable methodology because each map has its specific characteristics due mainly to:

• State of conservation of the map,
• map-sheet thickness [9],
• sensitivity to light [10],
• map format [11]: Through centuries, maps have been realized in a different format in length and width,
• map non-flat material [12,13]: The map could have suffered deformation of the support resulting corrugated. In other cases, the maps could be framed in a rigid wooden structure (as in the map presented in this case study),
• map accessibility [9,10]: It is common to find a map that has been painted on an entire wall, for example, in monasteries or royal palaces. That kind of historical cartography cannot be scanned (with the exception of laser scanners).

To avoid any physical deformations caused by temperature and humidity variations or by mechanical action caused using flatbed scanners, non-invasive contactless methodologies could represent a reliable solution. Nowadays, geomatics provides effective tools in this direction, and digital photogrammetry has been successfully used in order to digitize historical cartography [14]. In the framework of image-based photogrammetric techniques, structure-from-motion (SfM) is one of the most used [15,16] because of its flexibility, ease of use and cost-effectiveness [17]. Starting from 2D images, the epipolar geometry is estimated thanks to feature matching algorithms, for example, SIFT algorithm (scale-invariant feature transform) [18,19]. This approach allows to estimate the external orientation of the images and to reconstruct the 3D geometry of an object. In the last few years, a photogrammetric computer vision approach based on structure-from-motion algorithms has been applied for documentation of cultural heritage [18,20,21]. In the present-day, it is widely applied for many purposes in different fields and disciplines such as archaeology [22,23,24], architecture [25,26], geology [27], and geomorphology [28,29].

In the case of historical cartography digitization, photogrammetry has been applied in two different ways [30]: 2D [9,31] and 3D approaches [10,12,32]. The two-dimensional methodology for historical maps digitization generally consists of the generation of an ortho-projection of the original after a rectification process of the images acquired using a digital camera. If it is not possible to capture in a single shot the whole historical map, several rectified images can be merged in order to provide a single mosaic. These kinds of techniques can be applied only if the historical map has a flat surface or if it is possible to flatten it with no risk of damaging it. When it is not possible to fulfil this condition, it is necessary to use a 3D approach. Three-dimensional models of the maps and digital orthophotos can be obtained from the 3D point clouds generated by using recorded digital images [33]. During several past research experiences [10,12], a 3D photogrammetric workflow has been successfully applied in the framework of historical cartography digitalization, highlighting the reliability and potentiality of the method that enables to reach a metric accuracy below the millimeter [14]. Another advantage provided from this kind of approach is the possibility to consider the map as a 3D object and so taking into consideration the deformations of the support [7].

At state of the art, the use of SfM photogrammetric techniques is a well-known procedure that responds to the necessity of digitizing historical maps and documents without any direct contact. Nevertheless, one of the main issues about this methodology is represented by the necessity of developing a protocol for a “low cost with optimal response” (LCOR) [9] digitization of historical cartographic heritage. In fact, the possessors of old documents are not only wealthy and prosperous individuals but also many institutions and collectors with less available resources.

During the study presented in the current paper, a historic map from the National State Archive of Modena (Italy) was digitized with an SfM photogrammetry approach in order to achieve the following goals:

• Definition of a non-conventional workflow for historical cartography digitization,
• documentation of movable assets belonging to cultural heritage,
• preservation of the integrity of the original maps,
• creation of digital replicas to ease the dissemination to a wide stream audience.

This study aims to evaluate the reliability of an SfM photogrammetric approach to digitize historical cartography, focusing the method on two main principles:

• low-cost approach,
• high-flexibility of the method.

2. Materials and Methods

The whole approach consists in the digitization of historical cartography by a non-contact and harmless scanning method based on the LCOR basic requirements. Nowadays, low-costs/commercial off-the-shelf (COTS) hardware and software is widely available in the market and is accessible by most of the users operating in the framework of cultural heritage documentation. In this context, the solutions are many, and currently, it is possible to achieve acceptable results with minimal investment.

The presented methodology applied consists of:

• Acquisition of overlapped images of the historical map from different points of view by using a low-cost commercial sensor (mobile device camera),
• processing of the acquired images with a photogrammetric SfM-based software in order to generate a 3D model of the map,
• generation of an orthophoto of the digital model of the original map.

The quality of a photogrammetry-generated digital model can depend on many different factors including the state of conservation of the historical map, the lighting conditions during the data acquisition, the camera used, and the experience of the operator influencing the quality of the photographs needed to create the 3D model [34]. In the last few years, technologic improvements connected to mobile device cameras allow acquiring digital images characterized by an acceptable radiometric quality. However, nowadays, professional or semi-professional Digital Single-Lens Reflex (DLSR) cameras can be purchased at a relatively affordable price. As for the lighting conditions on the acquisition set, if the illumination is not sufficient, the use of auxiliary artificial lights is recommended (for example using photographic LED (Light Emitting Diode) panels).

In the case described in this paper, in order to emphasize the low-cost aspect of the methodology, the images were acquired using a common mobile device equipped with Snapdragon 821 Mobile Platform hardware (Qualcomm, San Diego, US) [35] and an optical sensor Sensor Sony IMX 298 (Sony, Tokyo, Japan). In

Table 1. Main specifications of the camera of the mobile device used.

Mobile Device	ONEPLUS A3003
Camera model	Sony IMX 298
Sensor size	1/2.8″
Megapixels	16
Pixel Size	1.12 µm
Image size	2610 × 4640 pixels

, it is possible to observe the main specifications of the camera used during the data acquisition phase.

The 3D model was generated using the photogrammetric software Agisoft PhotoScan 1.4 [36]. The data processing of the acquired images was performed on two different workstations, a high-performance one (A,

Table 2. Main specifications of the workstations used.

Workstation	Processor	RAM	GPU
A	Intel(R) Core(TM) i7-6800K, 3.40 GHz	128 GB	NVIDIA Quadro M2000
B	Intel(R) Core i7-4710HQ, 2.50 GHz	16 GB	NVIDIA GeForce GTX 850M

) (CPU Intel(R) Core(TM) i7-6800K, 3.40 GHz, RAM 128 GB, NVIDIA Quadro M2000), and a commercial off-the-shelf (COTS) laptop (B, Table 2) (CPU Intel(R) Core i7-4710HQ, 2.50 GHz, RAM 16 GB, NVIDIA GeForce GTX 850 M). In this case, the goal was to perform a comparison about the time required by both workstations during data processing in order to evaluate the suitability of the proposed methodology in the framework of relatively low-cost workstation solutions.

The proposed methodology was tested on a historical map preserved at the Modena National Historic Archive: “Pianta Tipografica dimostrativa il Territorio e Giurisdizione di Novellara con la rigorosa posizione de’ Cavi, Argini, e Strade, che la girano e con li manufatti pubblici, che vi sono contenuti; rilevata nell’anno 1774” (Grandi Mappe-4, Archivio di Stato di Modena. 130 × 63 cm). This map (Figure 1) is dated from 1774, and it represents the landscape assessment of the municipality of Novellara (RE) with the indication of the main channels and roads. This map was realized with colored ink on paper material. Two wood slats mounted at the top and at the bottom of the map surface limited the undulation deformation and kept the surface moderately flat.

The focus of this research is to propose a reliable workflow focusing on the suitability for a wide audience.

3. Work Phases

The application of the methodology proposed is presented in two subsections: Data acquisition and data processing.

3.1. Data Acquisition

The map used as a case study during the presented research was selected—in addition to its cultural and historical value—for the presence of a cartographical grid. The dimensions of the grid were estimated before the data acquisition through several manual measures. In particular, the segments of the regular grid were measured in the map extension (using a simple ruler) in order to verify if and where any deformation might have affected the map support. This operation took about 15 min to be performed. (Figure 2). The images were acquired at the Modena National Archive in a low natural-light condition with no auxiliary artificial lights. The mobile device was set up to the automatic high-quality mode (HQ) which, according to the producer manual, is the most reliable camera setting in low-light conditions.

This feature of the map allowed the identification of a high number of points—from which it was possible to estimate a relative local coordinate system—which could be used as ground control points (GCPs) and checkpoints (CPs) during the photogrammetric process, in order to evaluate metric accuracy. Two different operative workflows were applied. The first one can be applied to maps characterized (as in the considered case) by the presence of a cartographical grid. The other one can be applied to all kinds of maps as long as it is possible to approximate their shape to a flat surface. In order to test a low-cost approach, the acquisition of the images was performed with a commercial mobile device camera (as stated in the previous section, a Sensor Sony IMX 298 was employed). Sixty-one images were acquired following the photogrammetric principles [37,38] and allowing sufficient image overlap to facilitate the automatic matching of corresponding points (Figure 3). Image acquisition took only 10 min.

3.2. Data Processing

Map images were then processed with the photogrammetric SfM-based commercial software Agisoft Photoscan [36] by following a consolidated workflow [39,40]. The calibration of the mobile device camera was determined with the autocalibration implemented in the used software. As stated above, in the first proposed approach, the cartographic grid is used to estimate points with relative local coordinates. Since one of the main goals of this research is to test a rapid and low-cost digitization methodology and considering the good conservation conditions of the map used, the z-coordinates of these points were approximated to 0. Consequently, if the main goal is the generation of an orthophoto, both the tested workflows can be only applied to maps that do not present high deformations due to their conservation status or excessive deterioration of the support. Generally, a similar approximation needs to be evaluated on a case-by-case basis, depending on the ultimate aim of the end-user. Clearly, the final result will be more accurate as well in case of small discrepancies.

During the testing of the first methodology, 69 points of the cartographic grid were used as GCPs and 16 as CPs in order to evaluate metric accuracy (Figure 4). The points were randomly selected in order to cover the entire surface of the map uniformly.

In

Table 3. Mean errors on GCPs and CPs.

	RMSE [mm]
	X	Y	Z	XYZ
GCPs	0.6	1.0	1.9	2.2
CPs	0.9	1.2	1.9	2.4

, it is possible to observe the mean error after the bundle block adjustment. However, in the second case, the same dataset was processed without the use of the GCPs and the CPs; the obtained dense point cloud was scaled using direct measurements taken on the considered map. In total, the model was scaled with 4 manual measurements using as reference the four points of the grid placed near the map edges (alternatively, in the absence of a grid, metric scale bars previously positioned on the acquisition stage can also be used).

In both cases, the final outcome is a dense point cloud constituted by more than 5 million of points (

Table 4. Main details of the photogrammetric process.

N° of Images	Shooting Distance [cm]	Estimated GSD [mm/px]	N° of Tie Points	N° of Points of Dense Cloud
61	≈40	0.101	≈72,000	≈5,500,000

) from which it is possible to triangulate a three-dimensional triangulated irregular network (TIN).

4. Results

In order to perform a comparison between the obtained point clouds (generated by using the two different approaches), they were then registered together in the same coordinate system with the commercial software 3DReshaper [41]. A cloud-to-cloud registration procedure based on ICP (iterative closest point) algorithm was performed. In Figure 5, it is possible to observe the discrepancies between the two datasets, which can be considered almost insignificant for the purpose of this study.

In order to obtain an ortho-projection of the 3D model triangulated from the final point clouds, a 3D Cartesian coordinate system (X, Y, Z) is required. With the first observed process, a relative local coordinate system is provided by the coordinates estimated from cartographic grid. However, this is possible only in those cases where a cartographic grid or any other reliable reference to build a 3D Cartesian coordinate system is found on the surface of the historical map (for example the edges of the sheet). In many cases, historical maps do not have such references and, considering the state of conservation of many of these assets, the possibility that the edges of the sheet could be deteriorated has to be taken into consideration. In order to simulate this second extreme case, as for the second point cloud scaled by using the manual measurements, a best-fitting rectangular plane was interpolated (a least square interpolation was performed with 3DReshaper platform) in order to be used as projection plane (the x-axis and the y-axis were set on the sides of the interpolated rectangle, the z-axis was set on the normal direction of the flat surface) (Figure 6).

In Figure 7 it is possible to observe the discrepancies between the best-fitting plane and the dense point cloud.

In this case, observing that the discrepancies are millimeter-level, the approximation can be considered acceptable for both the approximation of the z-axis coordinates of the points of the cartographic grid and the interpolation of the projection plane. If the residual values were higher, different strategies would have been necessary in order to consider the eventual deformations of the map. As stated in the previous section, the acquired images were processed a first time with a high-performance workstation (Workstation A) and a second time with a COTS laptop (Workstation B). As it is possible to observe in Table 4, even though the required time for data processing using the Workstation 1 is significantly lower, it is possible to perform the described operations even with a less performing computer. In Table 4, the main details about the timing of data acquisition and data processing are summarized. The time required by these procedures depends on the number of the images and their resolution: The higher the number of images and their resolution, the longer the data processing time (

Table 5. Data acquisition and processing time. The image acquisition phase was rapid (it only required about 15 min to perform the manual measurements on the cartographic grid and about 10 min to acquire the digital images). For data processing time the following operations were considered (for both approaches): Image alignment and tie points generation; collimation of GCPs-CPs/setting of scale bars; dense cloud generation; three-dimensional mesh generation; orthophoto generation.

Data Acquisition	Data Processing (Workstation 1)	Data Processing (Workstation 2)
≈25 min	≈1 h	≈3 h

).

5. Discussion

In the previous sections, it was possible to observe the workflow of the proposed methodology, underlying its suitability in situations where it is not possible to use a standard planar scanner or other sensors that need direct contact to the fragile map support. Poor conditions and fragility of the materials are an obstacle, and a contactless approach for digitization is desirable in order to prevent damage to irreplaceable assets belonging to our heritage. In these cases, a digital photogrammetric approach represents a reliable and sustainable method for digitizing historical cartography (Figure 8).

Thanks to the technological progress observed in the last few years, image acquisition can be performed with both professional cameras and commercial mobile device sensors (our choice in using a commercial smartphone for data acquisition was taken in order to remark this point). The workflow tested during this research does not necessarily need high-performance workstations in order to be carried out: Of course, less performing workstation takes a longer time to complete data processing, as it is possible to observe in Table 5.

The main advantage of the methodology proposed in this paper consists of being a rapid, contactless, and harmless method that reduces almost completely the risk of damaging the map support. Other advantages include:

• High flexibility of the acquisition workflow, since the map format and size do not constitute a limit (on the contrary with a planar scanner),
• high-resolution of the digitized replica,
• low-cost of the solution.

Regarding the last point, no specific hardware equipment is required: Nowadays, mobile devices allow the acquisition of high-definition images, especially when professional cameras are not available. Of course, generally, a higher price range camera would produce better results in terms of image quality and resolution. However, as previously demonstrated, current commercial products achieve more than acceptable results.

The workstations do not require particular specifications, but it is supposed that they will fulfill at least the minimum software requirements. As for the most suitable employed platforms, in this research, licensed commercial solutions (Agisoft Photoscan and 3DReshaper), as well as open-source (CloudCompare), were used. License costs could vary between different versions, but low-cost academic or educational software versions are often provided. Also, free open-source photogrammetric SfM based solutions [42] are improving fast and represent a valid alternative to commercial platforms. A specification is necessary about the role of the operator in the framework of these kinds of digitization operations. Thanks to the improvements in acquisition techniques and implementations of automatic or semi-automatic procedures during data processing phases, the three-dimensional reconstruction of digital models on an image-based approach has become an extremely rapid procedure with high levels of reliability. However, it should be underlined that the management of these types of datasets by the operator remains fundamental for quality check (in terms of verification of metric accuracies) and data interpretation.

In this regard, the main weakness of the studied workflow concerns the metric control of the final product. In the absence of ground control points measured by topographical methods (like, for example, using a total station, which would allow greater control on the z-coordinates of the digitized map here approximated to 0) the accuracy check is based on manual measures taken directly on the map. This allows reaching a millimeter-level accuracy (which is coherent with the mean errors previously observed on GCPs and CPs after the bundle block adjustment) opposed to other methodologies and techniques which obtain submillimeter-accuracies [13]. In any case, if a millimeter-level accuracy is considered tolerable for the intended purposes, both the observed methods allow achieving acceptable 3D models.

6. Conclusions

The applications of the methodology proposed in this paper enables the preservation of the information associated with cartography and documents stored in historical archives through the creation of high-detailed digital replicas. The digitization of this kind of historical assets could improve the dissemination of the original data to future generations of users operating in the field of cultural heritage. Indeed, the application of 3D modelling techniques represents a reliable method to contrast the inevitable loss of historical documents due to ageing: only by documenting the originals it is possible to reduce the risk of losing irreplaceable data. Another useful application made possible by the digitization of historical cartography is the management of the obtained dataset in GIS environments in order to carry out, for instance, studies in interpretation and valorization of past landscapes [43] or environmental reconstructions [44].

Future perspectives might include the application of a laser scanner to construct a “gold standard” for assessing the reliability of the methodology proposed with high accuracy. Nonetheless, the potentiality of historical documents as datasets of information about the past is unquestionable, and the SfM digital photogrammetric digitization approach represents a rapid, highly flexible, low-cost, and contactless method to preserve and valorize valuable assets, rich of many kinds of information belonging to our cultural heritage.