A Multi-Level Grid Database for Protecting and Sharing Historical Geographic Urban Data: A Case Study of Shanghai

Abstract

Historical geographic data play an important supporting role in the study of long-term geographic studies, such as climate change, urban expansion and land-use and land-cover change. These data vary in source, format and accuracy and are widely found in historical documents, old maps, produced vector data, aerial photographs, old photographs, etc. The complex nature of data makes it difficult for researchers to organize, store and manage in a unified manner. Thus, GIS practitioners and social scientists will collectively face the challenge of integrating historical data into spatial databases. Herein, we introduced the concept of a multi-level spatial grid, selecting Shanghai as the study area, to construct the Shanghai historical geographic database and give the conceptual model and processing method. The experiment was performed using the China Historical Geographic Information System (CHGIS), which showed the historical evolution of Shanghai more conveniently. Meanwhile, we simulated one million rows of historical geographic data in Shanghai and compared the retrieval efficiency of the encoding method with the latitude–longitude and geometric object indexing methods, which demonstrated that our method was very effective. This research is important for the construction of a historical urban database, which can better preserve historical resources and promote urban culture with information science and technology.

1. Introduction

The integration of Geographic Information Systems (GIS) into the humanities and social sciences fields has increasingly gained attention, with many academic institutions in China and abroad continuing to develop methods for this integration [1,2]. In 2006, the Center for Geographic Analysis was established at Harvard University to support research and demonstrate the utilization of geospatial technology across all disciplines in the University [3]. The integration of spatial information science into the traditional humanities has brought new opportunities to the discipline and has also provided conditions for its expansion [4].

Historical geographic data play an important supporting role in the study of long-term geographic studies, such as climate change, urban expansion and land-use and land-cover change. Via these studies, we can learn about the historical geography and even reconstruct the past geography so as to further understand the relationship between man and land from a longer-term perspective. The study of historical geography and spatial information science facilitates innovations in historical geographic data collection, storage, retrieval and representation and overall improvements in effective data use [5,6]. Historical geographic data are widely presented in historical documents such as old maps, produced historical vector data, historical aerial images, old pictures and even old videos, as shown in Figure 1 [7,8,9]. With the continuous discovery of historical data and the improvement of digitizing methodology, the amount of spatial data available is increasing [10,11]. These data often have different sources and creators, diverse structures, scattered distributions, long time series, ambiguity and large differences in spatial resolution.

Therefore, accurately organizing and storing historical geographic data, particularly the continuous and complex historical changes in China, is necessary. In the China Historical Geographic Information System (CHGIS) project, Man Zhimin proposed the “lifetime” description method, which is different from the time section description [12]. By using the start and end time of the spatio-temporal entity to describe the existence time, the storage space can be effectively saved. Chen et al. developed a conversion method between the traditional Chinese chronology method and the standard calendar and proposed a spatiotemporal data model of historical place names for representative data, and taking “Sino-family-tree GIS” as an example, a spatiotemporal framework for Chinese history and culture was constructed [13]. From the perspectives of geography and history, Hu et al. combined the four elements of time (the time of the beginning and the end), place, person and event, integrated the idea of a “human–land relationship” in geography and finally designed a basic, general and historical GIS data model using the spatiotemporal object modeling method [14]. However, many historical GISs do not consider data formats that can be easily shared and integrated. Therefore, based on the precise and vast geographic data contained in geographic databases, the major challenge for humanistic scholars is to develop a historical geographic database that combines historic and contemporary data [15]. Nowadays, some research or applications based on ontologies, RDFs and Linked Open Data are gradually carried out. Sharing data in the FAIR principle is currently a popular trend in HGIS.

At the same time, we believe that there is a growing need to combine today’s geographic data with historical geographic data for long-term research in order to better understand land cover change and human–land relationships and to better protect digital cultural heritage. Historical geographic data are widely available in different formats, sources and precisions, as shown in Figure 1. Therefore, in combination with the China Historical GIS, we analyze its data characteristics, current data structure and existing problems and actively introduce the methods of GIS, hoping to explore a database construction plan that is more suitable for the characteristics of historical geographic data. In this study, we will not only propose a theoretical foundation for the method but also design corresponding experiments to finally verify the feasibility and effectiveness of our proposed method. We hope the proposal of our method will solve the data storage problems often encountered in the fields of urban cultural heritage, historical geography, etc. and directly improve data utilization and query efficiency. In the experimental part, Shanghai was selected as the study area owing to the availability of extensive historical data, especially surveyed maps and current aerial surveys, after the opening of Shanghai’s port. Additionally, Shanghai’s rapid development and the standardization of its modern database construction serve as an important reference for other cities (Figure 2).

This paper will be discussed in the following format. Section 2 describes the CHGIS project and lifetime structure. Section 3 provides a detailed description of the database using a multi-level grid. Section 4 details the experiment using CHGIS data and query efficiency in Shanghai, China. Finally, Section 5 concludes the study and explores the direction for future research.

2. CHGIS and Lifetime Structure

Historical geographic data have the characteristics of multi-source and multi-precision. Which data format does the existing platform or system use? In this section, the CHGIS project and lifetime structure are discussed in detail to analyze the development status of CHGIS and the problems of data structure display. China has a long history, and the geographical environment and human factors have been constantly evolving for thousands of years. The goal of the CHGIS project is to establish a basic geographic information system database of China’s historical period. Historical geographic information includes natural information, such as climate, landforms, disasters, animals and plants, etc.; it also includes human information, such as political regions, settlements, economy, population and culture. Considering the integration and sharing of more data, on the basis of CHGIS, the author’s HGIS research team further explored the integration platform of historical geographic data and possible new data models.

2.1. China Historical Geographic Information System

The China Historical Geographic Information System (abbreviated as CHGIS) provides a base GIS platform for researchers to use in spatial analysis or in visualizing the historical divisions of China as digital maps [16]. It was first launched in January 2001 to establish a database for densely populated areas and historical administrative units throughout China’s historical period (221 BC to 1911 AD) [12]. The benchmark of the CHGIS is the Historical Atlas of China (Chinese 中国历史地图集), which was compiled by Tan Qixiang at the Fudan University in China [17]. Using GIS technology, the historical geographic information data are standardized into a contemporary spatial framework (the base map is ArcChina), by which the temporal resolution is significantly improved (from one or several standard time slices per dynasty to one year). This project provides researchers with a GIS platform with which to undertake spatial analysis and temporal statistical modeling and represent selected historical units as digital maps. The CHGIS data contain continuous time series data and numerous time slide updates, particularly for counties and prefectures. Among them, the place name database of CHGIS is the Temporal Gazetteer.

Now, Fudan University has further developed the China historical geographic information platform [18]. They have inherited the CHGIS data and added modules such as old maps, spatiotemporal frameworks, databases and resource centers (shown in Figure 3). One of the biggest advantages of this platform is that CHGIS spatiotemporal data can be displayed intuitively and dynamically without professional GIS software, such as ArcGIS or QGIS.

2.2. Lifetime Data Structure

Geographical data are largely concerned with the spatial dimension of the data, whereas time is also an important dimension for historical data. In the CHGIS project, the concept of lifetime was proposed [19]. The lifetime concept refers to a condition under which locality names, identified spatial characteristics and administrative affiliations remain unchanged from the time of its establishment until that of its change. This is the most important data attribute in the CHGIS and can therefore be considered a spatiotemporal data model. The structure of the CHGIS database is shown in Figure 4a, and the lifetime data structure is shown in Figure 4b.

When opening the CHGIS data in the GIS software, there will be four columns (BEG_YR, BEG_RULE, END_YR, END_RULE), which represent the starting and ending year and the corresponding rules of the name of a geographic object (the administrative polygons). Any changes to a geographic object will generate a new code. A code is an ID that uniquely identifies a spatiotemporal specific object or event and has no spatiotemporal meaning. When the spatial extent of the administrative unit changes, the ID also changes correspondingly. For example, when identifying Fujian Province, ID#98014 and ID#98000 actually identify different ranges (seen in Figure 4c). In the year 1128, at the same time the administrative seat was moved, the jurisdictional area increased. The polygons (actually, a group of polygons regionalized as one object in GIS to account for coastal islands) that represent the two periods look almost the same, but there is a difference. In 1128, Fujian Lu expanded to include Penghu Island (circled in red).

Then, in the GIS software, we load the CHGIS data, select the Shanghai Bund area as the spatial query condition and select the time period of 600 to 1911 as the time query condition, which returns the attribute table (

Table 1. Query results for the bund area, Shanghai.

NAME_PY	NAME_FT	TYPE_PY	TYPE_CH	BEG_YR	BEG_RULE	END_YR	END_RULE	NOTE_ID
Su Zhou	蘇州	Zhou	州	602	4	605	4	90232
Su Zhou	蘇州	Zhou	州	606	4	606	4	90233
Wu Jun	吳郡	Jun	郡	607	4	615	3	90234
Wu Jun	吳郡	Jun	郡	616	4	620	4	90255
Su Zhou	蘇州	Zhou	州	621	4	623	4	90235
Su Zhou	蘇州	Zhou	州	624	4	741	4	90236
Wu Jun	吳郡	Jun	郡	742	4	757	4	90237
Su Zhou	蘇州	Zhou	州	758	4	938	4	90238
Xiu Zhou	秀州	Zhou	州	939	5	1194	4	90256
Jiaxing Fu	嘉興府	Fu	府	1195	4	1276	5	90257
Jiaxing Lu	嘉興路	Lu	路	1277	5	1291	4	90258
Songjiang Fu	松江府	Fu	府	1292	4	1325	4	90251
Jiaxing Lu	嘉興路	Lu	路	1326	4	1327		90262
Songjiang Fu	松江府	Fu	府	1328	4	1911	5	90261

) as the query result (only the main columns are displayed here; non-important parameters such as person and other columns are not listed).

3. Spatial Database Using a Multi-Level Grid

Uppercase is required here, which is uppercase in the original systemThe focus of this section is on HGIS research and the CHGIS project. Which data structure can be used to better organize and share data? Historical geographic data are often stored in different formats and created by various sources, which makes storage difficult. However, we can store the data by building a unified metadata template and indicate attributes such as the data type and person. This does not address the data inaccuracy that results from varying time periods and sources. For example, current spatial data may be extracted from Landsat 8 imagery that has a spatial resolution of 30 m; however, for historical spatial data, such as aerial imagery or printed maps, the spatial resolution may have high or unknown inaccuracies, which makes data integration difficult. Therefore, in the construction of long-term databases, an effective data model and database design are urgently needed [20,21,22]. In our study, multi-level spatial grids are used. Large-scale grids are selected for low-precision historical period data, and small-scale grids are selected for high-precision remote sensing information [23,24]. Different scales can be converted between each other. This multi-level spatial grid helps to store the historical geographic data.

3.1. GeoSOT Grid Model

Among the various kinds of spatial information grids, the GeoSOT grid was selected for this study (Figure 5). The geographic coordinate subdivision grid, with one-dimensional integer coding on the 2ⁿ-tree (GeoSOT) model, is a subdivision and coding method that subdivides the Earth’s surface into 2-D spatial grids.

In GIS, the spatial data structure includes a vector and raster. The encoding of the vector point can be directly converted from latitude and longitude to obtain its GeoSOT subdivision code. For the coding of the spatial range of the surface data and raster data, the appropriate hierarchical level is calculated according to its MBR (Minimum Bounding Rectangle), and then the subdivision patch is determined by the spatial position information; thus, the GeoSOT subdivision patch code corresponding to the spatial data is obtained. In particular, the grid coding calculation method based on the spatial grid model is an efficient position calculation system. The method for using grids for division and spatial conversion into codes is described in detail in the references [25,26,27].

3.2. Design of the Table

In general, historical geographic data are stored in the structure of “raw data + metadata”. Taking the most used old map in historical geographic study as an example, its metadata is an abstraction of the spatial information and attribute information. Through the retrieval and management of metadata, it is possible to avoid the direct manipulation of the original map with a large amount of data (especially high-resolution maps, which usually take up a lot of storage space), and this is the key to realizing data sharing.

Figure 6 shows the most simplified data storage template, where Codes is the GeoSOT code calculated through spatial position information and is used as the index of spatial query, supporting various operations of coding-binary; Level (the spatial granularity of the data) is reflected by the length of the code and is not specifically stored as a column (since GeoSOT uses quad-tree division, the longer the code, the finer the granularity); Name is the original data name of the data; Type is the original data type, including various types shown in Figure 1, such as documents, maps, vector data, images, video, etc.; and FilePath is the path of the data, which is usually a record in an existing database or a file in the file system (operating system). In particular, we preserve the lifetime data structure in CHGIS and preserve the concept of lifetime and four attribute columns (BEG_YR, BEG_RULE, END_YR, END_RULE). After the metadata are stored, we index the code of the Codes column and then associate the corresponding Data id. This index helps store data uniformly and speeds up its retrieval.

3.3. Optimization of the Grid-Based Database

In the process of storing, the multi-level grid system has the advantages of partitioning and sub-partitioning, along with the following additional advantages:

(1)

Load balancing: Combined with load balancing theory and application, the master node and slave node are set up (Figure 7). The master node stores coarse-grained data, and the slave nodes access fine-grained data. When data are acquired, they will go from the master node to the slave node and, finally, to the final node.

(2)

Data partition: The database is partitioned by the grid area, not necessarily on the same scale.

(3)

Read–write separation: The data are inverted according to the amount of data loaded by the grid, and the read and write functions are separated to optimize the database efficiency of the database.

For example, when a user searches, first, it requests the domain name resolution; then, the DNS server returns the corresponding IP address; then, it goes to the cluster to first analyze the accuracy (resolution) of the user’s search; finally, it locates the web server quickly according to the partition situation.

4. A Case Study of Shanghai

To verify the effectiveness and efficiency of our proposed method, we conducted the following experiments using Shanghai as our study area.

4.1. Experimental Environment and Design

The test software and hardware environment that were used are as follows: Python 3.6 is the development language; PyCharm 2020.2.3 is the compiler; MongoDB 4.2.0 is the database; the GIS software used is ArcGIS 10.7. The experiments were conducted on a server with an i7-10750H [email protected] GHz processor, 32 GB of memory and 1 TB of storage capacity.

The design of the experiments was as follows: first, in CHGIS, the Shanghai area was selected, a GeoSOT grid was established and a corresponding retrieval was performed and compared with the original method. Second, one million rows of different types of historical geographic data were generated (as shown in Figure 6) to test the feasibility of the storage and the efficiency of the retrieval (compared with the latitude and longitude indexing method). The data will be randomly distributed in the Shanghai area and randomly set to vector, raster and multi-media formatted data (e.g., old maps, old videos with geotagging, old aerial imagery).

4.2. Shanghai in CHGIS

In this study, Shanghai was selected for the GeoSOT preprocessing of the CHGIS data (Figure 8). In the CHGIS (raw data), a spatial location may correspond to multiple different administrative regions in the historical period, so multiple polygons are involved, and there is a one-to-many spatial mapping relationship overlapping the polygons. Here, we selected one point named Old Ports (Chinese “老码头”) from the data. Thus, any historical administrative region corresponding to this point will be displayed; the historical region is usually large and will have more than one. These regions overlap with each other, making the display of CHGIS poor.

Next, we performed the same spatial query on the CHGIS preprocessed data using the GeoSOT method. Only the grid required in the query was selected, and changes corresponding to different political districts in the historical period were only displayed in the attribute table, which was very user-friendly. We can directly see which administrative units the spatial location (corresponding grid) belongs to in different historical periods in the attribute table, and we can also sort them in chronological order.

To query the fine-grained grid more accurately, we further divided the specific parent grid through the quad-tree, retrieved the grid of the corresponding area and performed the corresponding spatial query. Given the fact that the child grid is completely contained in the parent grid in the spatial extent, the political area changes of the child grid can directly inherit the parent grid.

4.3. Query Efficiency Using Code

First, we generated one million historical geographic data in different regions of Shanghai randomly by the following three steps:

Step 1: Generate the spatial information of the data randomly. The structure of the vector data can be points, lines or polygons, and the spatial range of any point is within the region of Shanghai.

The generation of codes is complicated, and any nodes of the polygon need to be set as follows: the lat coordinate of the left node is less than the lon coordinate of the right node, and the lat coordinate of the upper node is greater than the lon coordinate of the lower node (this relationship is because Shanghai is in the eastern hemisphere and northern hemisphere of the Earth).

Step 2: Set the data to vector, raster and multimedia formats randomly, as shown in Figure 1.

Step 3: Generate the begin and end years of data randomly, and set beg_year <= end_year.

Then, according to the spatial resolution of the simulated data, the 15 layers of the GeoSOT grid were selected for the experimental verification of the query efficiency (since Shanghai, as an urban research area, is usually studied on a kilometer-scale grid, level 15 in the GeoSOT model is chosen, which corresponds to the kilometer-scale grid). We built a B-tree index for the latitude/longitude columns, a B-tree index for the code columns and a 2dsphere for the geometry object columns, respectively. Next, we arbitrarily selected polygonal regions of different regions (disjoint) within Shanghai for the query, returned all attribute columns, recorded the execution time of the returned results and, finally, took the average of multiple results for comparison. The statistical comparison of the query time is shown in Figure 9.

When comparing the different experiments in the MongoDB database, using the same query conditions, the query efficiency of the encoding was higher than that of the latitude/longitude and geometry object. The average time was improved by an order of magnitude more than that of using the latitude and longitude method, which means that our method is very efficient.

We analyzed the accuracy and integrity of query results. The accuracy of the results shown in

Table 2. Experiment results.

Retrieval Method	Lat/Lon	Geometry	Code
Time (ms)	1032.75	103.89	76.71
Accuracy (%)	90.3	96.8	88.9
Integrity (%)	100	100	100

tells us whether the results satisfy the query criteria, and the integrity tells the situation where all qualified data have been selected.

In terms of the accuracy of the search results, geometric search has the highest accuracy; more than 96% of the search results are consistent with the query conditions. When Code is used for retrieval, its precision is slightly lower than those of the other two. This is because an extra space region that does not belong to the search region will appear in the result of a grid-based query, and the range returned will be larger than the original query condition. The smaller the grid scale, the higher the query precision. The integrity of the three query results is 100%, that is, all the data that meet the conditions are detected.

4.4. Discussion

Through the experiments in Section 4.2 and Section 4.3, we have verified the feasibility and effectiveness of the method.

First of all, in Section 4.2, we saw that the polygons in the CHGIS project overlapped with each other, resulting in the poor visualization. Our method using the grid method makes the data more intuitive and clear and can conveniently sort the attributes in chronological order. The only thing of concern is that the scale of the grid needs to be constructed according to the amount of data in the query area itself. In our experiment, Shanghai, as the central and eastern coastal area of CHGIS, has more than 2000 years of data. Therefore, when Shanghai meets this query requirement, other regions will also meet it, thus demonstrating the feasibility of this method.

Then, in Section 4.3, we designed simulation data covering this area, and the amount of data reached 1 million. By comparing different storage indexing methods, we know that the efficiency of our method is significantly improved, approaching an order of magnitude. The only regret is that, comparing the accuracy and completeness of the query, the accuracy of our method is slightly lower than that of the other two, but considering the improvement of the query efficiency, we think it is acceptable. If the accuracy of the results needs to be further improved, this can be achieved by reducing the size of the grid so that less data exceed the query area.

Overall, the method in this paper is feasible and efficient. In terms of visualization, it makes presentations more intuitive and understandable. In terms of retrieval efficiency, there is also a significant improvement. When using our method, the spatial scope of the research area and the scale of the research data should be reasonably grasped, and an appropriate grid scale should be selected.

5. Conclusions

Historical geographic data have the characteristics of having many sources, different formats and significant differences in accuracy. Building a historical geographic database that combines history with the current situation is a major challenge for social scientists. In this study, the concept of a multi-level grid was introduced. Using Shanghai as the study area and the described conceptual model and method, the Shanghai historical geographic database was constructed. The experiments verified the effectiveness and efficiency of the method. Spatial queries were conducted using CHGIS data, and the results showed that gridded storage could more conveniently display the historical evolution of Shanghai. Additionally, the experiment also simulated one million rows of varying historical geographic data formats in Shanghai, and the retrieval efficiency of the coding method was compared with those of the latitude–longitude and geometric object indexing methods, which showed that the method was efficient.

This research is significant for the construction of the Shanghai historical geographic database. It serves as an important reference for the preservation of different forms of cultural resources. Furthermore, our future research can be extended to include other data types not mentioned above and facilitate the interdisciplinary cohesion between other disciplines.