The Construction Method of Jiangxi Geological Big Data Platform in China

Abstract

Aiming at the problems of low information management levels and low reuse rate of massive heterogeneous geological data of the Jiangxi Geological Bureau, a Jiangxi geological big data platform based on cloud service and big data technology was designed to realize the integration and sharing of Jiangxi geological big data. Firstly, the architecture of the Jiangxi geological big data platform is designed based on hierarchical thinking, including the infrastructure layer, data layer, platform service layer, application layer and user layer from the bottom up. Secondly, the key technologies for building a Jiangxi big data platform are described, including multi-layer service aggregation, geographic information service bus, geocoding service, Spark big data technology and elastic scaling technology. Finally, the main functions of the Jiangxi geological big data platform are introduced, including a platform portal website, a mobile portal system, a geological big data comprehensive analysis system and a geological 3D modeling system. The operation results of the platform show that the Jiangxi geological big data platform can effectively manage the massive heterogeneous geological data of the Jiangxi Geological Bureau and mine the value of the data.

1. Introduction

In geological work, the data obtained through drilling, geophysical exploration, geochemical exploration, remote sensing, field geological survey and other working methods are called geological data. Geological data has the characteristics of large quantity, variety and wide application, which conforms to the “5V” (Volume, Variety, Value, Velocity and Veracity) characteristics of big data. Therefore, it is typical big data [1]. At the same time, compared to other types of data, geological data has many characteristics, such as mixing, sampling, multi-source and spatio-temporal [2,3]. With the advent of the era of big data, the construction and innovative application of a geological big data platform has become the focus of geological research [4].

Geological survey agencies of the world’s major developed countries, such as United States Geological Survey (USGS), British Geological Survey (BGS) and Australian Geological Survey Organization (AGSO), attach importance to enhancing the role of geological surveys in promoting economic development and social change, solving major earth science problems [5] and actively promoting the innovation and development of geological surveys at a higher level [6]. These innovations and developments are mainly manifested in the following aspects:

(1)

Large scope of data management

The data management has 7–10 categories, and it is not limited to the traditional geological field, but extends to the cross-field [5].

(2)

Standardization of data management

Data management plans have been launched, such as the USGS’s published work “Science Strategy for Core Science Systems in the U.S. Geological Survey (2013–2023)” as the core scientific research program in the decade [7], the BGS’s Open Geoscience Program is committed to improving the sharing of geological information in a wider range [8] and the Natural Resources Canada (NRCan) development and implementation of the Geomapping for Energy and Minerals (GEM) Program [9]. The USGS’s Scientific Data Catalog defines standardized data and enables data parsing and analysis in multiple application areas.

(3)

Development of data management tools and databases

Data management tools and databases in the field of geoscience have been greatly developed in various countries. For example, the United States has built “The U.S. National Geologic Map Database (NGMDB)” [10] and Canada has built the “National Energy Use Database (NEUD)” [11]. At the same time, most databases also provide analysis and retrieval systems [12].

China has also made progress in the development of geological big data platforms. In November 2014, China’s first self-developed geological big data platform was put into trial operation in the Tibet Autonomous Region, realizing one-click storage and the management, organization, fast retrieval and intelligent mining of geological big data [13]. In 2015, the China Geological Survey (CGS) organized the implementation of the “Geological Big Data and Information Service Project”, and in 2017, China’s first national geological big data platform, “Geological Cloud 1.0”, was built. Subsequently, after further upgrades, “Geological Cloud 2.0” and “Geological Cloud 3.0” were released in 2018 and 2021, respectively [14,15]. “Geological Cloud” is a comprehensive geological information service platform which integrates computing resources, data resources and application systems, and plays an important role in prospecting, urban construction, disaster prevention and other aspects. Compared with the national level, there are great differences in geological surveys and geological investigation data among provinces. The Hubei, Shandong, Sichuan, Yunnan, Anhui, Henan and Shaanxi provinces have carried out geological big data construction in their own provinces [16,17].

The main department responsible for the production, application, storage and management of geological data in the Jiangxi Province is the Geological Bureau of Jiangxi Province, which was established in 2021. Its business involves geology and mineral resources, ecological protection, engineering construction, geographic information and other fields [18]. However, the level of geological informatization in the Jiangxi Province is relatively low. Data such as surveying and mapping, geological environment, and engineering investigation have not been centrally managed, and it is difficult to reuse historical data. Therefore, a modern software platform is urgently needed to improve the level of informatization.

In order to integrate and gather the geological results and professional data resources of the Jiangxi Geological Bureau, and eliminate the “data gap”, relying on the Internet, big data, cloud computing and other technologies, a Jiangxi geological big data center platform for geological business management and professional technology application was built [19]. This paper gives a general introduction to the Jiangxi big data platform, including platform architecture (Section 2), key technologies (Section 3), platform implementation (Section 4) and a discussion (Section 5). In Section 6, we make a summary of this paper.

2. Platform Architecture

With reference to a series of IT infrastructure and environment design specifications in China, and successful big data cloud platform implementation cases at home and abroad, the foundation support layer dedicated to the Jiangxi big data platform is built, including a computing resource pool, storage resource pool and network resource pool. Relying on the large geological database in the basic support layer, this platform realizes the storage, management, processing, analysis and mining of geological big data; supports various applications; and provides basic geological and mineral services for departments at all levels, enterprises and the public more efficiently [20]. The overall architecture design is shown in Figure 1.

2.1. Infrastructure Layer

Infrastructure-as-a-service (IaaS) [21] provides infrastructure services for the platform, mainly including network, basic hardware and software and other resources, which is the operating environment of the geological big data platform. KVM virtualization technology is used to integrate and manage IT resources in a unified manner, and automatic platform deployment and configuration modules are provided to simplify the installation and management of the basic cloud platform [22]. The IaaS layer provides a cloud-hosted working environment for businesses and users, and supports hybrid cloud management, multi-region, and multi-data center deployment.

2.2. Data Layer

Based on the big data platform framework, and through a professional data acquisition system, big geological data is collected and aggregated from different data sources, including basic geographic data, basic geological spatial data, geological text data, geological image data, geological business data, geological thematic data, geological archive data, etc.

Using Data as a Service (DaaS) [23] technology, the platform realizes unified storage framework management, ensuring that data can be stored and obtained quickly. A NoSQL solution provides high-performance data access using a distributed file system (HDFS) and various storage media such as Redis, HBase and MongoDB.

For large data files or computing files, it adopts multi-channel parallel computing and provides a parallel algorithm library API. The full-text search API is provided, which enhances the ability to retrieve data efficiently. In addition, for different computing scenarios, it also provides batch computing API based on Map Reduce, memory computing API based on Spark and Stream computing API based on Strom/Spark Stream.

2.3. Platform Service Layer

Platform as a Service (PaaS) [24] is a deep integration of business data resources and platform resources on the basis of IaaS and DaaS, which can directly provide various types of cloud services for users to support the development of business systems. For these cloud services, the PaaS layer provides users with a visual interactive interface and a unified platform development interface, so that users can complete various work related to business system construction on the platform, including resource service release, identity authentication management, access control, application construction, application deployment, etc., without maintaining and caring about the platform itself.

Unlike ordinary software or data services, “cloud services” are elastic and scalable, adjusting computing power as needed. The cloud services provided by the platform include general service, comprehensive integration platform, professional database management platform, cloud management platform, big data management platform and GIS support platform. The platform service layer provides application support for the SaaS layer through RESTful APIs, WMS, WFS, WCS, WPS and other protocols.

2.4. Application Layer

Software as a Service (SaaS) is the software supply model of the application layer, and the SaaS layer mainly provides application framework template services and universal services for end users [25]. Based on the PaaS layer, SaaS provides geological 3D modeling and display platforms, a geological big data comprehensive analysis system and other applications. Each application at the SaaS layer is designed and constructed based on the same basic platform and application framework, and deployed in the cloud partition mode to maintain the independence of service systems and facilitate the independent management, upgrade and maintenance of services [26].

3. Key Technology

3.1. Multi-Layer Service Aggregation

Multi-level service aggregation technology is used to aggregate address map services, data services and analysis services from different sources to provide rich GIS service capabilities.

Service aggregation is intended to integrate different types of services from different sources into a unified system through standardized processes and then release them in a unified way to provide end users, so as to break the single service model, provide a variety of new spatial information applications and realize the integrated application and online sharing of distributed, heterogeneous, multi-source and multi-temporal geospatial data.

Through service aggregation, you can directly use a geological online map or geological data services released by other departments, reducing the cost of system construction. Service aggregation is realized through configuration, which does not require complex development and reduces the difficulty of system development. Through service aggregation, it can effectively realize the integration of remote deployment services, and build new services quickly to meet the requirements of business agility by reusing existing services and extending the value of existing services.

3.2. Geographic Information Service Bus

An Enterprise Service Bus (ESB) is the intermediary to realize the intelligent integration and management of services and also the connection center of communication between application systems, which can eliminate the technical differences between different applications, so that different application servers can coordinate operation and realize the communication and integration between different services.

A Geographic Enterprise Service Bus (Geo-ESB) for geographic information applications is established. This system relies on standard technologies, such as SOAP (Simple Object Access Protocol) or JMS (Java Message Service), to enable interoperability between coarse-grained applications (such as services) and other components through simple standard adapters and interfaces. It is the connection center of communication between geographic information systems, which can eliminate the technical differences between different geographic information systems, provide a unified service interface in line with the geographic information industry standards, and support the communication between different systems, spatial data transmission and GIS service integration.

3.3. Geocoding Service

Using geographic coding technology to standardize the encoding of geographic spatial data, we can support the implementation of fuzzy matching, reverse matching, and batch matching services. Geocoding technology has two functions: geocoding and inverse geocoding. Geocoding is the process of obtaining map positioning information through known address data and marking it on the map. Inverse geocoding refers to the process of obtaining address information by coordinate position. Geocoding improves the readability of data, so that address information can be visualized on the map, and the inherent information contained in the address can be easily processed and applied by researchers, planners, developers or analysts, thus achieving the integration of information in a variety of spatial scopes. Through address matching, the data records in the address database and map database can be connected, and the address data in the address database can be endowed with map positioning information (e.g., spatial coordinates).

3.4. Geographic Data Distribution Services

The data distribution service of geographic information solves the concurrency bottleneck caused by the wide area complex network environment of the traditional single data center scheme of geographic information web service through the distributed multi-data center data distribution mode. The Jiangxi geological big data platform supports the rapid distribution of geographic information data and high concurrent access to massive data.

3.5. Spark Big Data Technology

Spark [27] is a memory-based open source computing framework, which is the next generation of the most popular open source general parallel computing framework project in the cloud computing field after Hadoop. As a platform for big data processing, Spark has the characteristics of fast running speeds, good ease of use, strong versatility and running anywhere. Geological big data processing technology based on Spark is developed to realize distributed storage/retrieval/management, high-performance analysis and calculation and the high-performance visualization of geological big data.

3.6. Elastic Scaling Service

The Elastic Scaling Service (ESS) is a data management procedure that automatically adjusts elastic computing resources based on users’ service requirements and policies. Elastic Scaling Service (ESS) is a service that automatically adjusts cloud computing resources (such as the number of server instances) based on business needs and preset policies. It is an important feature of cloud computing technology. However, the cloud computing platform itself often cannot be implemented in specific applications. In the GIS field, these features cannot be achieved by a GIS server alone.

In this project, we study elastic scaling technology, a fully combine cloud computing platform with a GIS server (iServer), which achieves VM-level elastic scaling and saves computing resources. When the cluster load increases, nodes can be dynamically added to ensure response time. When the load is reduced, nodes can be dynamically reduced, and computing resources can be released, thus realizing resource-intensive savings.

4. Platform Implementation

The Jiangxi geological big data platform is the window of the Jiangxi Geological Bureau’s external service. The platform runs on the Internet, displays information that can be socialized and disclosed, provides users with geological information metadata query and browsing and geological science popularization services and supports geological service collection. Users can submit resource requests and download resources on the portal. The portal provides unified identity authentication. Upon single sign-on (SSO), users can operate modules such as news bulletin, online map, resource center, application center, interactive platform, personal center and background management.

4.1. Platform Portal

The Jiangxi geological big data platform portal integrated data center, service center, development center, standard specifications, results data, three-dimensional map, operation and maintenance center and other subsystems provide a unified entrance for users to access the platform, to achieve SSO and to log into the interface, which is displayed in Figure 2.

After successfully logging into the platform, the home page is displayed, including the latest news, bidding information, platform announcements, hot data, latest results, platform resources, application center and other functional options.

The top of the home page is the function bar, which mainly has standard regulations, result data, a three-dimensional map, data centers, application centers, data exchanges, service centers, development centers, operation and maintenance centers and other modules.

The data center is an important module for the Jiangxi geological big data platform, which is the entrance for the platform’s achievement display and system application. All the geological resource data of the Jiangxi Provincial Geological Bureau will be displayed in different forms in the data center module, and users can query and make statistics on resources, realize the spatio-temporal comparison of resources and view three-dimensional scenes. The data center page is mainly composed of directories and resource display Windows, and the interface is shown in Figure 3.

The result data module mainly displays the geological result data of different departments according to different result categories, realizes the electronic management of the geological result data, and facilitates the search and use of data. The interface of the result data module is shown in Figure 4.

The service center provides a large number of geological service resources, including data services, functional services, etc. Users who successfully register and log into the portal can browse and view service resources in the resource center and can make online resource requests. The interface of the service center module is shown in Figure 5.

4.2. Mobile Client Portal

Based on the mobile client custom online map function, it provides “one map” of all kinds of geological resource data of the Jiangxi Geological Bureau, and can provide map browsing, query services, measurement tools and other functions. The interface of the mobile client portal is shown in Figure 6.

The achievement data module mainly displays the achievements of different departments and different categories, gathers the geological achievement data of various departments, realizes the electronic management of geological achievement data and facilitates the search and use of information.

4.3. Comprehensive Analysis System of Geological Big Data

A 3D geological “one map” can be used to overlay and manage all geological data and can also be used to manage and display geological information. The system supports unified data management, unified display, linkage query, professional analysis and other functions.

The comprehensive large-screen can display all kinds of statistical results and real-time monitoring information that users are most concerned about in the “cockpit” mode. The interface of the comprehensive large-screen is shown in Figure 7.

The function of classification statistics is to classify and analyze the data of basic geology, hydrology, environment, disaster, mineral dynamic monitoring and evaluate the results of different regions and different types, and display the statistical charts in the superimposed map window.

4.4. Geological 3D Modeling System

The Transparent Earth management platform is a cloud management system for 3D model data and other types of data. On the management platform, you can set user permissions as well as index information of the model and some information related to visualization. You can also upload the borehole histogram that needs to be identified. It supports the establishment of a 3D stratum model based on multi-source data, such as borehole, section, elevation, bedrock and so on. For complex areas, such as interbeds, pinches, lenses and other special geological phenomena, it can be automatically modeled by adding constraints. Using the idea of “top-down”, the top geological surface of each level is established layer by layer to complete the automatic modeling. The automatically constructed 3D geological body model is shown in Figure 8.

5. Discuss

GeoCloud 3.0 of the China Geological Survey, NGMDB/ScienceBase of the United States Geological Survey (USGS), Geoscience Australia platform, and OneGeology platform in Europe are all world-renowned geological data platforms. Compared to them, the Jiangxi geological big data platform has some innovations.

5.1. Compared to China’s GeoCloud 3.0

In terms of usage purposes, GeoCloud 3.0 is a portal for the aggregation and sharing of historical geological achievement data across China, while the Jiangxi geological big data platform serves the digitization of the entire process for provincial geological business, as well as data production, management and sharing.

In terms of hardware and software localization, GeoCloud 3.0 partially relies on foreign commercial software such as Oracle 11g and ArcGIS 10.8, while the full-stack localization of the Jiangxi geological big data platform can achieve independent and controllable data security.

In terms of 3D data, GeoCloud 3.0 provides basic 3D geological body display functions, while the Jiangxi geological big data platform realizes integrated surface-subsurface 3D modeling and visualization functions.

In terms of data governance, Geological Cloud 3.0 mainly relies on manual input and review, while the Jiangxi geological big data platform can connect with multiple business systems within the Jiangxi Provincial Geological Bureau to achieve the unified aggregation and integration of cross-system data.

5.2. Compared to Other National Geological Data Platforms

In terms of technical architecture, USGS’s ScienceBase adopts an AWS-based cloud data warehouse architecture, with scientific data storage and open sharing as its core, lacking service-oriented encapsulation capabilities for business. The Geoscience Australia platform adopts a hybrid cloud architecture and relies on commercial authorization for third-party system integration, which limits its flexibility. The Jiangxi geological big data platform adopts a fully autonomous and controllable cloud native microservice architecture, using Kubernetes technology for container orchestration, with fine service granularity and strong scalability, which has advantages in technological autonomy.

In terms of geological data modeling and localization adaptation, the geological classification systems of the USGS platform and Geoscience Australia platform are based on North American and Australian geological conditions (such as USGS’s NCGMP09 standard), which have significant standard incompatibility issues with the Chinese geological work. The OneGeology platform adopts the GeoSciML international standard and lacks a deep, localized data model for specific regions. The Jiangxi geological big data platform uses national standards such as the “China Stratigraphic Table” and the “Geological Mineral Terminology Standards” to establish professional data models that are in line with China’s geological reality, and designs special data structures for geological units such as rare earth mines and red bed basins unique to the Jiangxi Province, China.

In terms of 3D geological modeling capabilities, the USGS NGMDB platform does not provide an integrated 3D modeling environment and mainly relies on external tools such as Leapfrog and GOCAD for implementation. The Geoscience Australia platform has advantages in 3D crustal modeling, as its 3D modeling accuracy and refinement are suitable for the national scale, but not for the provincial scale. In the Jiangxi geological big data platform, the 3D GIS engine is deeply integrated with geological modeling technology, supporting fine 3D geological body modeling driven by multi-source data from drilling, profiles and contour lines. It also supports engineering-level professional calculations, such as reserve estimation. The entire process from modeling to analysis can be completed within the platform without relying on third-party professional modeling software.

In terms of data interoperability and service openness, OneGeology’s data services mainly rely on static publishing, lacking dynamic query and business linkage capabilities. The data acquisition method of USGS’s ScienceBase is mainly through file downloading, and its API interface capability is limited, making it difficult for the platform to meet complex business system integration requirements. The Jiangxi geological big data platform fully supports OGC standard services and RESTful APIs, and also has customized data exchange interfaces for government systems. It has more complete capabilities in business encapsulation and cross system interconnection of data services.

6. Summary

The Jiangxi geological big data platform is designed and developed by using cloud services and big data technology, which provides convenience for the storage and management of massive heterogeneous geological data for the Jiangxi Geological Bureau. At the same time, data sharing can reduce the cost of data usage and avoid the waste of data resources. The Jiangxi geological big data platform includes basic geographic data, basic geological data, mineral resource data, hydrogeological data, engineering geological data, environmental geological data, geophysical exploration data, geochemical exploration data and remote sensing geological data. The average response time of the platform is less than 3 s, and under peak concurrency conditions, the average query response time of the platform is less than 5 s. The response time of the statistical function can be appropriately extended, generally not exceeding 20 s. The number of requests and queries processed simultaneously by the server at the same time is less than 100 times per second. The proportion of successfully processed query requests to the total number of requests reaches 99.9%. The accuracy of querying and processing results reaches 100%.

Currently, the Jiangxi geological big data platform has become the most comprehensive, voluminous, efficient and standardized geological data platform in the Jiangxi Province, and has become the first choice for obtaining geological data in the Jiangxi Province. Starting from the height of the national macroeconomic and social development strategy, the construction of the geological big data platform is to meet the requirements of building data power, promoting resource integration, improving the modernization of national governance capacity, promoting industrial innovation and development and helping with economic transformation and upgrading. Based on the massive geographical, geological, exploration and other data accumulated by the Jiangxi Geological Bureau for more than 60 years, and relying on information technologies such as cloud services, Internet and big data, the Jiangxi geological big data platform is built to improve the utilization rate of existing data, improve the transformation and upgrading ability of geological work in the Jiangxi Province, and lay the foundation for prospecting the selection and preparation of exploration plans. This platform can comprehensively enhance the innovation and application capabilities of the geological and mineral industries.

However, there are still two major challenges in terms of platform applications. Firstly, due to the high economic value of geological data, data production institutions are unwilling to share the latest geological data on the platform. Secondly, due to the differences in software environments among end-users, the number of cloud services directly integrated into their software environment using this platform is still limited.