A Coordinated Bidirectional Data Fusion Processing System for Meteorological Applications

Abstract

This study proposes a coordinated bidirectional meteorological data fusion processing system for controlled data exchange across the intranet, demilitarized zone (DMZ), and Internet. The system adopts a three-layer isolation architecture and integrates Apache MiNiFi, Apache NiFi, and Apache Kafka to coordinate edge preprocessing, DMZ-based fusion processing, asynchronous message buffering, and Internet service publication. Kerberos authentication, access control, and operational monitoring support outbound data-product services and inbound user-request-driven workflows. Operational evaluation at the National Meteorological Science Data Center showed that edge preprocessing reduced cross-domain data volume by an average of 95%; representative bidirectional workflows were completed within minutes; and the average Kafka message-processing success rate over three consecutive months was 99.77%. The system has supported the automated generation and external publication of human comfort index products, while its core mechanisms have been generalized into a reusable software stack for cross-domain scientific data applications. The results indicate that the proposed architecture provides a practical and deployable approach to efficient and controlled bidirectional meteorological data fusion without changing existing network security boundaries.

1. Introduction

Meteorological data underpin weather forecasting, climate monitoring, disaster warning, and public weather services. As numerical prediction, smart-grid forecasting, remote sensing, and public-facing services have advanced, meteorological data volumes have continued to grow, while data sources, processing chains, and service users have become increasingly diverse. This trend has intensified the demand for data sharing and fusion across organizations, networks, and platforms [1,2,3,4,5,6,7,8,9]. At the same time, meteorological data must be managed according to their openness and security level: public data may be delivered over the Internet, whereas some operational, foundational, and restricted data must remain on the intranet and cannot be directly exposed to the public Internet. Consequently, moving meteorological data among the intranet, DMZ, and Internet is not merely a transfer problem; it also requires controlled exchange across security boundaries, low-latency and high-throughput processing of frequent data streams, and closed-loop responses to interactive, on-demand services. Conventional systems rely largely on one-way push, file exchange, or manual intervention, making it difficult to simultaneously provide security isolation, low latency, high throughput, bidirectional interaction, and operational visibility [3,4,5,6,7,8,9].

In recent years, Apache NiFi, Apache Kafka, edge computing, and message-oriented middleware have provided a strong technical basis for complex data-flow processing. NiFi supports data ingestion, flow orchestration, format conversion, backpressure control, and operational monitoring [10]; Kafka provides high-throughput messaging, persistent storage, asynchronous buffering, and producer-consumer decoupling [11,12,13]; and MiNiFi brings lightweight acquisition and preprocessing closer to data sources [14]. Previous studies have demonstrated the feasibility of NiFi-Kafka stream ingestion, scalable distributed stream processing, and edge-center collaboration in complex data-processing settings [15,16,17,18,19,20]. However, most existing applications focus on data aggregation within a single network domain, enterprise data integration, or general-purpose stream processing, typically assuming that producers, processors, and consumers operate in a relatively open or uniformly secured environment. In meteorological operations, where the intranet, DMZ, and Internet are separated into distinct security domains, traditional ETL, one-way file transfer, NiFi-only orchestration, or a standalone message queue cannot simultaneously ensure that restricted data remain on the intranet, that cross-domain flows are controlled, that high-frequency data are processed in near real time, that external requests trigger closed-loop services, and that operations are auditable.

To address these challenges, this study develops a coordinated bidirectional meteorological data fusion processing system. The system establishes a three-layer intranet-DMZ-Internet isolation architecture and combines MiNiFi-based edge acquisition and preprocessing, NiFi flow orchestration, Kafka-based asynchronous decoupling, Kerberos authentication, and operational logging. Following request-level authentication and authorization, least-privilege access, and controlled transmission principles [13,21,22,23], the system supports both outbound automated data-product services and inbound request-driven closed-loop responses. The resulting architecture offers a deployable and extensible technical approach to efficient, controlled, and bidirectional meteorological data fusion under strict network isolation.

The main contributions of this study are as follows:

(1)

A three-layer intranet-DMZ-Internet isolation architecture is proposed for meteorological data exchange across security domains. It provides controlled outbound and inbound data paths without directly exposing restricted data to the Internet.

(2)

A coordinated MiNiFi-NiFi-Kafka processing model is designed. Lightweight acquisition, format conversion, and preprocessing are placed at intranet edge nodes, whereas computationally intensive fusion, routing, and product-distribution tasks are assigned to DMZ-side central nodes.

(3)

Kafka-based asynchronous decoupling is integrated with cross-domain security controls. Message buffering, partition replication, consumption-rate regulation, authentication, access control, and operational monitoring improve stability, reliability, and observability under variable loads and bidirectional interaction.

(4)

The architecture is deployed and evaluated in the operational environment of the National Meteorological Science Data Center. A human comfort index service is used to demonstrate intranet data acquisition, DMZ-based fusion, Internet product distribution, and inbound request-response processing. The architecture and core mechanisms were further generalized into the FairStack scientific data software stack under a National Key Research and Development Program project.

2. Related Work and Positioning of This Study

2.1. National Meteorological Science Data Center

The National Meteorological Science Data Center, administered by the China Meteorological Administration and operated by the National Meteorological Information Center, is one of China’s national science data centers and an important component of the national science and technology infrastructure. The center has established mature operational capabilities for meteorological data aggregation, submission, service delivery, and platform-based management, thereby supporting both operational meteorology and scientific research. Research on national science data centers increasingly emphasizes data findability, accessibility, interoperability, and reusability, as well as provenance and verifiability throughout data processing [1,2,24,25]. However, existing work has focused primarily on data-resource management and service publication; systematic support for bidirectional data flows, controlled services for restricted data, and externally triggered closed-loop processing across the intranet, DMZ, and Internet remains limited.

2.2. Secure Cross-Domain Data Exchange

Network isolation, DMZ buffering, identity authentication, access control, encrypted transmission, and audit logging are key mechanisms for controlled cross-domain data movement. Both zero-trust architectures and the Kerberos protocol emphasize identity-, policy-, and ticket-based access control rather than implicit trust derived solely from network location [13,21,22,23]. Existing systems commonly use one-way gateways, file exchange, interface proxies, or dedicated security appliances. Although these approaches provide clear isolation boundaries, they can exhibit inadequate timeliness, limited orchestration flexibility, weak state tracking, and insufficient support for bidirectional service loops when processing high-frequency meteorological data, complex fusion tasks, and interactive Internet requests. A system architecture that combines controlled security boundaries with stream processing and cross-domain coordination is therefore required.

2.3. NiFi/Kafka-Based Data-Flow Processing

Apache NiFi provides visual flow orchestration, data routing, format conversion, backpressure control, data provenance, and reliable delivery, making it suitable for integrating heterogeneous data sources and managing complex flows [10]. Apache Kafka provides high throughput, persistent storage, partition replication, and message replay for asynchronous buffering and producer-consumer decoupling at scale [11,12,13]. MiNiFi performs lightweight acquisition and processing on resource-constrained nodes or near the data source [14]. Previous studies show that NiFi, Kafka, and related ingestion and stream-processing frameworks can effectively support data acquisition, message delivery, fault isolation, and scalability analysis [15,19,20,26].

Nevertheless, most deployments target enterprise integration, general-purpose big-data processing, or aggregation within a single security domain. There is still limited systematic work on the coordinated deployment of NiFi and Kafka across meteorological intranets, DMZs, and Internet-facing environments, particularly with respect to keeping restricted data unexposed, supporting bidirectional closed-loop services, and maintaining security visibility.

2.4. Edge-Center Collaborative Computing

Edge-center collaborative computing offers a practical means of reducing cross-domain traffic and central-node processing loads. Lightweight acquisition, format conversion, filtering, and preprocessing can be moved to edge nodes, while complex algorithms, workflow orchestration, and product distribution remain at central nodes. Existing edge and cloud-edge research focuses primarily on task offloading, resource allocation, scheduling, edge-cloud continua, and collaborative intelligence [16,17,18]. Their integration with meteorological data classification, secure cross-domain exchange, asynchronous messaging, and closed-loop services has received comparatively less attention.

2.5. Positioning and Distinctive Contributions of This Study

In summary, prior research has established important foundations in meteorological data platforms, secure exchange, stream processing, and edge collaboration. Most studies, however, focus on data aggregation within a single operational domain, one-way publication, general-purpose stream-processing frameworks, or isolated security controls. Integrated system designs and operational analyses for bidirectional meteorological data fusion across the intranet, DMZ, and Internet remain scarce.

Building on a National Key Research and Development Program project, this study addresses the operational need for meteorological data movement across security domains. It systematically develops and implements a three-layer intranet-DMZ-Internet architecture, a coordinated MiNiFi-NiFi-Kafka edge-center processing model, Kafka-based asynchronous decoupling with security controls, and both outbound automated product services and inbound request-driven closed-loop workflows. The system has been deployed in representative services at the National Meteorological Science Data Center, enabling an integrated analysis of its architecture, core mechanisms, operational workflows, and broader applicability.

3. System Architecture and Collaborative Model

3.1. Design Objectives and Overall Approach

The system is designed for meteorological data movement across the intranet, DMZ, and Internet under the principles of security isolation, controlled exchange, edge preprocessing, central fusion, asynchronous decoupling, and operational visibility. Without changing existing network security boundaries, the three-layer architecture coordinates data, computation, and control flows: the intranet performs restricted-data acquisition and lightweight preprocessing; the DMZ provides message buffering, workflow orchestration, and fusion processing; and the Internet-facing side supports external data access, user interaction, and product services. This design provides the architectural basis for both outbound automated product publication and inbound request-driven closed-loop processing.

3.2. Layered Architecture and Core Components

The system consists of three isolated domains—the intranet, DMZ, and Internet—and five vertical layers: application, service support, data service, software support, and hardware support (Figure 1). The intranet contains restricted meteorological data sources, internal operational systems, and edge nodes running Apache MiNiFi for acquisition, format conversion, element extraction, and lightweight preprocessing. The DMZ acts as a buffer and fusion-processing zone and hosts Apache NiFi and Kafka clusters, metadata services, state caches, and algorithm execution environments. The Internet-facing domain serves external users, sectoral systems, and third-party interfaces through HTTP/HTTPS, FTP/SFTP, object storage, and APIs [10,12,14].

Figure 1. Layered deployment architecture and key technical components. Dashed arrows indicate controlled communication paths across network-security boundaries.

At the application layer, the system provides acquisition, cleansing, distribution, monitoring, auditing, and flow orchestration. The service-support layer supplies intranet edge nodes, DMZ central nodes, and algorithm containers. The data-service layer provides protocol access, messaging, metadata management, state caching, and log management. The software-support layer hosts NiFi, MiNiFi, Kafka, Kerberos, Redis, MySQL, and related services. The hardware-support layer comprises servers, storage, network equipment, and security-boundary devices across the three domains. Component access follows identity authentication, authorization, and least-privilege principles [13,21,22,23]. Together, these layers support bidirectional flows, asynchronous decoupling, and security control.

3.3. Bidirectional Data-Flow Model

The system supports two high-level flow models. In the outbound data-product service flow, restricted source data are acquired and lightly preprocessed on the intranet. Exchangeable data entities or index records are then transferred through controlled channels to the DMZ for fusion, format conversion, and product distribution before being published to Internet-facing services. In the inbound request-driven flow, user requests or external data first enter the DMZ, where they are monitored, filtered, logged, and forwarded to trigger intranet workflows or algorithm pipelines. The resulting products are returned through the DMZ to Internet-facing systems. The two flows form a bidirectional service loop at the application level while preserving controlled initiation and DMZ relaying at the network level, thereby avoiding direct intranet-Internet connectivity.

3.4. Edge-Center Collaborative Computing Model

The model assigns acquisition, format conversion, and lightweight preprocessing to intranet edge nodes, while complex fusion, routing, product generation, and service distribution are performed at central nodes in the DMZ. MiNiFi parses GRIB files, extracts required variables, and performs format conversion at the edge, thereby reducing the amount of data transferred across domains. NiFi, Kafka, and algorithm containers in the DMZ then conduct fusion processing, queue buffering, task orchestration, and product output. This allocation follows the principle of moving low-complexity, data-proximate tasks to the edge and consolidating resource-intensive tasks centrally [14,16,17,18]. It reduces cross-domain traffic and the risk of transmitting restricted raw data beyond the intranet.

3.5. Kafka-Based Asynchronous Decoupling and State Management

Kafka decouples data production, fusion processing, and service consumption. After an intranet MiNiFi agent writes processed data or index information to Kafka, it can release its computing resources without waiting for downstream DMZ processing. DMZ-side NiFi consumers read messages according to current load, while Kafka partitions, replicas, persistence, and consumer-offset management improve delivery reliability [11,12,13]. Distributed runtime state, task progress, and operational logs are managed using Redis, MySQL, and NiFi/Kafka monitoring information, supporting fault diagnosis, retries, and performance statistics (Figure 2).

Figure 2. Distributed state storage and runtime-state management.

4. Key Implementation and Representative Operational Workflows

4.1. Intranet-Side Implementation

Apache MiNiFi is the core intranet component for real-time acquisition, lightweight preprocessing, and controlled delivery. It can be deployed close to data sources or operational processing nodes and uses scheduled tasks, directory listeners, scripts, and API calls to acquire meteorological data. Processor-based flows implement format conversion, element extraction, file naming, checksum generation, and retry handling [14]. For authenticated business data sources such as the Tianqing MUSIC (Meteorological Unified Service Interface Community) API, SDK scripts or MiNiFi processors generate access signatures and request parameters and record authentication status, execution logs, and exceptions. MUSIC is a unified and standardized meteorological data-access and application-programming interface on the intranet and serves as a business data-source interface in the proposed architecture. The edge deployment is shown in Figure 3.

Figure 3. MiNiFi-based edge-processing deployment. Solid arrows indicate data and processing flows; the dashed horizontal line separates the processor-based and script-based deployment modes.

During intranet-DMZ interaction, only preprocessed products, index records, or explicitly authorized data entities are transferred to the DMZ; restricted raw data remain within the intranet. MiNiFi communicates with DMZ components through Site-to-Site, SFTP, or the Kafka Producer/Consumer API, together with allowlists, port restrictions, identity authentication, and logging [10,12,13,14,21,22]. Automated intranet-to-DMZ and DMZ-to-intranet transfers are shown in Figure 4 and Figure 5.

Figure 4. Automated intranet-to-DMZ data transfer.

Figure 5. Automated DMZ-to-intranet data transfer.

To obtain data through the intranet Tianqing MUSIC API, MiNiFi first completes identity authentication and then retrieves the required station, gridded, or file-product data according to the request parameters. Two approaches are supported: MiNiFi can periodically execute the SDK script supplied by Tianqing, or native MiNiFi processors can be orchestrated to retrieve the data. The first approach is suitable when existing operational SDKs must be reused; the second provides greater flow visualization and centralized management. The MUSIC API acquisition procedure and its processor flow are shown in Figure 6 and Figure 7.

Figure 6. MUSIC API acquisition procedure implemented with MiNiFi.

Figure 7. MiNiFi processor flow for data acquisition.

For Kafka-based bidirectional exchange, intranet MiNiFi agents use PublishKafka processors to write index records, parameter messages, or preprocessed data entities to the Kafka cluster in the DMZ. MiNiFi agents or intranet workflows use ConsumeKafka processors to retrieve external-request parameters, processing indexes, or result notifications [11,12,14]. Access to the DMZ does not imply direct intranet-Internet connectivity; exchange is constrained by approved protocols, ports, allowlists, and authentication policies and is mediated through the DMZ message queue. The resulting bidirectional data flow is shown in Figure 8.

Figure 8. Bidirectional data flow between MiNiFi and Kafka. Solid arrows indicate message flow, and the dashed vertical line denotes the intranet-DMZ security boundary.

4.2. DMZ-Side Implementation

The DMZ provides buffering, routing, fusion processing, and service relaying. Kafka serves as the persistent messaging hub, managing partitions and decoupling producers from consumers; NiFi provides flow orchestration, routing, format conversion, task scheduling, queue management, and product distribution [10,11,12,14,15]. Computationally intensive fusion tasks, such as geographic-boundary overlay, gridded-data processing, and image generation, are executed in algorithm containers or on script execution nodes. The DMZ-side processing flow is shown in Figure 9.

Figure 9. DMZ-side data-processing workflow. Solid arrows indicate data, message, and processing flows, and the dashed vertical line denotes the network-security boundary.

The DMZ supports multiple exchange patterns, including publishing intranet products to the Internet, acquiring external Internet data, relaying user requests, and returning products. Kafka is not treated as a standalone security mechanism. Instead, its persistence, acknowledgments, replicas, and offset management are combined with Kerberos authentication, NiFi access policies, transmission controls, operational logging, and exception alerts [10,11,12,13,21,22,23] to provide reliable delivery and controlled processing.

4.3. Internet-Facing Services and Controlled Interaction

The Internet-facing domain serves external users, sectoral systems, and third-party interfaces through data distribution, object storage, APIs, and request submission. At the service level, the architecture supports both outbound and inbound closed-loop workflows; at the network and security levels, cross-domain requests are relayed through the DMZ and initiated only through controlled connections, avoiding direct intranet-Internet connectivity. Internet requests entering the DMZ are monitored, filtered, logged, and converted before being used as parameters or index records to trigger intranet workflows or DMZ-side fusion. The external-data acquisition and relay flow is shown in Figure 10.

Figure 10. Internet data acquisition and relay workflow. Solid arrows indicate data and message flows, and the dashed vertical line denotes the network-security boundary.

4.4. Operational Outbound Product Generation and Publication

At the National Meteorological Science Data Center, the system automates the publication of meteorological products generated from intranet data. An intranet MiNiFi agent executes a custom script to acquire the China Smart Grid Hourly Analysis Product (0.05° × 0.05°), parse the GRIB file, extract meteorological variables, and calculate the human comfort index, producing a NetCDF file. MiNiFi then transfers the result to NiFi and Kafka in the DMZ. In parallel, the system obtains provincial boundary data through the Amap API. A Python 3.9 algorithm generates a national human comfort index map in PNG format; the output is buffered in a NiFi queue and published to Alibaba Cloud Object Storage Service (OSS). The complete outbound workflow is shown in Figure 11.

Figure 11. Outbound automated data-product service workflow. Red arrows indicate the principal processing and transfer paths, while dotted lines delimit the network-security domains.

This workflow is operational at the National Meteorological Science Data Center and automatically generates and publishes a national map each day for public meteorological services. A sample human comfort index product and the corresponding file list in Alibaba Cloud OSS are shown in Figure 12 and Figure 13, respectively. Section 6 evaluates processing time, task success rate, and the publication chain using runtime logs, file timestamps, and object-storage records.

Figure 12. National human comfort index map.

Figure 13. Product files published to Alibaba Cloud OSS.

4.5. Inbound User Interaction and On-Demand Service Workflow

To support customized Internet requests, the system implements an inbound workflow. NiFi in the DMZ captures user interactions from an Internet-facing service through processors such as ListenHTTP and writes request parameters to Kafka. The intranet Tianqing processing pipeline or related algorithm workflows consume these parameters and perform fusion calculations. The resulting index information is written to Kafka in the DMZ, while result files are placed in a designated directory through SFTP. DMZ-side NiFi then retrieves the index records through ConsumeKafka and pushes the files to the Internet-facing business system, completing the request-response loop. This workflow is operational in the dynamic product-customization service of the National Meteorological Science Data Center and is shown in Figure 14.

Figure 14. Inbound user-request closed-loop service workflow.

5. Security Model and Reliability Mechanisms

5.1. Security Boundaries and Data-Control Principles

The primary security objective is to support cross-domain data services without changing existing network security boundaries. Restricted raw data and core operational systems remain on the intranet. The DMZ hosts exchangeable data, index records, fusion tasks, and service-relay flows, whereas Internet-facing services can access only processed, desensitized, or explicitly authorized products. Inbound requests are received, filtered, logged, and forwarded through the DMZ, preventing direct access from the Internet to intranet data sources. The design follows per-request verification, least privilege, and explicit authorization [13,21,22,23].

5.2. Identity Authentication and Access Control

Authentication and authorization are implemented through Kerberos, NiFi user/group policies, Kafka access control, host allowlists, and service-port restrictions. Kerberos authenticates users and services [21]; NiFi policies restrict the operations that roles may perform on flows, processors, queues, and data resources [10]; and Kafka ACLs or equivalent controls restrict topic read/write permissions [12]. Access to operational sources such as the Tianqing MUSIC API is governed by signatures, tokens, or SDK-based authentication, and access outcomes are logged. The Kerberos-based NiFi authentication mechanism is shown in Figure 15.

Figure 15. Kerberos-based authentication mechanism for NiFi. Solid arrows indicate authentication and authorization interactions, while dashed boxes delimit the relevant service components.

5.3. Transmission Protection and Integrity Assurance

Different links use Site-to-Site, SFTP, HTTPS, object-storage interfaces, or Kafka protocols as appropriate. Identity authentication, transmission encryption, file checksums, message acknowledgments, replicas, and retry mechanisms improve reliability and integrity [10,11,12,21]. For each cross-domain file or message, the system records its name, size, timestamp, topic, consumer offset, processing state, and exception information, supporting operational auditing and fault diagnosis.

5.4. Monitoring, Auditing, and Exception Handling

NiFi provenance records, processor logs, Kafka consumer lag, server monitoring, task-execution logs, and object-storage records are used to monitor flow state, queue accumulation, processing time, failures, and product publication [10,12,25]. Processing failures, network interruptions, message accumulation, and interface exceptions are addressed through retries, queue buffering, alerts, and manual verification when required. Section 6 analyzes message-processing success, queue waiting time, and task success within selected operational windows.

6. System Evaluation and Operational Analysis

The system was evaluated using operational records from representative services at the National Meteorological Science Data Center. The evaluation examines the effect of edge preprocessing on cross-domain traffic, end-to-end processing efficiency, Kafka-based asynchronous decoupling, operational stability, and the role of security controls in supporting bidirectional closed-loop services. Data sources include MiNiFi and NiFi logs, Kafka message-state records, file-generation and transfer timestamps, object-storage publication records, system-monitoring logs, and workflow execution records.

Evaluation scenarios include automated generation and publication of the human comfort index, outbound CODAS-SST, SWS, and GSSODS batches, and inbound on-demand requests for individual stations and geographic regions. These scenarios cover intranet acquisition, edge preprocessing, DMZ fusion, Internet data access, product generation and publication, request relaying, and result return, providing a representative view of multi-domain meteorological data processing.

6.1. Evaluation Objectives and Metrics

Five aspects were evaluated: (1) the effect of edge preprocessing on cross-domain traffic; (2) end-to-end efficiency for outbound and inbound workflows; (3) Kafka message processing and asynchronous decoupling; (4) stability during selected high-frequency task windows; and (5) operational evidence that security controls support controlled bidirectional services. The evaluation aspects, primary metrics, data sources, and analytical purposes are summarized in Table 1.

Table 1. Evaluation metrics and data sources.

Evaluation Aspect	Primary Metrics	Data Source	Purpose
Edge preprocessing	Original file size; post-processing cross-domain file size; data-volume reduction	MiNiFi logs and file records	Assess the effect of edge processing on cross-domain traffic
End-to-end efficiency	Acquisition, preprocessing, DMZ reception, fusion, publication/return time	File timestamps, NiFi provenance, and object-storage records	Assess automated and on-demand service response
Asynchronous decoupling	Message volume, consumption rate, maximum backlog, and queue waiting time	Kafka monitoring and NiFi queue state	Assess Kafka buffering and burst absorption
Operational stability	Total tasks, success rate, exceptions, recovery time, and manual interventions	Runtime logs and monitoring alerts	Assess reliability during selected operational windows
Security control	Authentication, access control, transport protocols, audit logs, and exception handling	Kerberos, NiFi, Kafka, and system logs	Document operational evidence of controlled cross-domain services

6.2. Effect of Edge Preprocessing on Cross-Domain Traffic

In the outbound service workflow, MiNiFi is deployed at an intranet edge node to acquire, parse, extract, and lightly preprocess source meteorological data. For the human comfort index service, the input is the China Smart Grid Hourly Analysis Product (0.05° × 0.05°) in GRIB format. The file contains multiple variables and complete spatial grids; transferring it unchanged to the DMZ would increase network traffic and downstream processing load.

The system parses the GRIB file, extracts required variables, and generates intermediate products on the intranet, transferring only the data needed for comfort-index calculation and map generation to the DMZ. The reduction therefore does not result from a generic compression algorithm; it is achieved by narrowing the transferred variable set and data granularity through edge preprocessing.

The reduction ratio was calculated as (1 − preprocessed cross-domain data volume/original data volume) × 100%. As shown in Table 2, the three batches achieved reductions of 94–96%, with an average of 95%. Moving GRIB parsing and variable extraction to intranet edge nodes and transferring only the data required for downstream fusion substantially reduced both cross-domain traffic and the DMZ-side receiving load. The reduction resulted from variable selection and format conversion rather than general-purpose file compression.

Table 2. Cross-domain data volume before and after edge preprocessing.

Batch	Original GRIB File Size (GB)	Cross-Domain File Size After Preprocessing (GB)	Data-Volume Reduction
07–09 (Batch 1)	0.186	0.0093	95%
07–09 (Batch 2)	0.172	0.0103	94%
07–09 (Batch 3)	0.194	0.0077	96%
Mean	0.184	0.0092	95%

6.3. End-to-End Processing Efficiency

End-to-end processing time is a key measure of the system’s operational utility. For the outbound workflow, it is defined as the interval from the start of intranet data acquisition to successful publication of the resulting product to object storage or an Internet-facing service. For the inbound workflow, it is defined as the interval from receipt of an Internet request in the DMZ to successful return of the resulting product to the Internet-facing business system. The stage-level and end-to-end timing results for the outbound and inbound workflows are presented in Table 3 and Table 4, respectively.

Table 3. End-to-end processing time for outbound data-product services.

Batch	Data Acquisition	Edge Preprocessing	DMZ Reception and Buffering	Fusion Processing	Product Publication	End-to-End Total
CODAS-SST	20 s	5 s	10 s	38 s	10 s	1 min 23 s
SWS	17 s	4 s	8 s	22 s	9 s	1 min
GSSODS	21 s	5 s	10 s	35 s	10 s	1 min 21 s
Mean	19.3 s	4.7 s	9.3 s	31.7 s	9.7 s	1 min 15 s

Table 4. End-to-end processing time for inbound user-request workflows.

Request	Request Reception	Kafka Write and Wait	Intranet Processing	Result Return	Closed-Loop Total
Single station, 7 days	0.09 s	3.5 s	8.2 s	0.38 s	28 s
Region, 3 days	0.12 s	7.5 s	14.6 s	0.45 s	35 s
Single station, 30 days	0.10 s	5.5 s	21.8 s	0.52 s	48 s
Mean	0.10 s	5.5 s	14.9 s	0.45 s	37 s

For the three outbound batches, end-to-end time ranged from 60 to 83 s, with an average of 75 s. Fusion processing was the largest listed component, averaging 31.7 s. The three inbound requests were completed in 28–48 s, with an average of 37 s; intranet processing time increased with the temporal and spatial scope of the request. The total closed-loop time in Table 4 was measured directly from entry and exit timestamps. It includes flow scheduling, protocol conversion, file I/O, and Internet-service processing that are not itemized in the table and therefore is not equal to the simple sum of the listed components. These results show that representative product publication and on-demand responses can be completed within minutes, while fusion and intranet algorithm execution remain the main targets for further optimization.

6.4. Asynchronous Decoupling and Operational Stability

Producer, processor, and service loads do not vary synchronously in cross-domain meteorological workflows. If acquisition must wait for fusion or Internet publication, upstream intranet processes can become blocked. Kafka provides an asynchronous buffer between MiNiFi, NiFi, and operational processing modules, allowing producers and consumers to operate at their own rates. When DMZ fusion or Internet publication experiences short-term fluctuations, Kafka absorbs the backlog and prevents downstream variation from directly blocking upstream acquisition. Monthly Kafka message-processing and asynchronous-decoupling metrics are reported in Table 5, and stability during the selected high-frequency task windows is summarized in Table 6.

Table 5. Kafka message-processing and asynchronous-decoupling metrics.

Period	Messages Produced	Mean Production Rate	Mean Consumption Rate	Maximum Backlog	Mean Queue Waiting Time	Processing Success Rate
April 2026	86,420	1.18 messages/s	1.22 messages/s	186	4.1 s	99.64%
May 2026	91,350	1.25 messages/s	1.28 messages/s	143	3.5 s	99.81%
June 2026	88,760	1.21 messages/s	1.24 messages/s	96	3.0 s	99.87%
Mean	88,843	1.21 messages/s	1.25 messages/s	142	3.5 s	99.77%

Table 6. Stability during selected high-frequency task windows.

Observation Window	Total Tasks	Successful Tasks	Failed Tasks	Success Rate	Mean Recovery Time	Manual Interventions
10 min	360	360	0	100%	-	0
3 min	30	30	0	100%	-	0
8 min	180	180	0	100%	-	0
Total	570	570	0	100%	-	0

Average production and consumption rates were calculated over actual business-operation windows rather than the full calendar month.

From April to June 2026, the system processed an average of 88,843 messages per month. Mean production and consumption rates were 1.21 and 1.25 messages/s, respectively. Maximum backlog decreased from 186 to 96 messages, mean queue waiting time decreased from 4.1 to 3.0 s, and the three-month mean message-processing success rate was 99.77%. The slightly higher consumption rate indicates that the system could absorb short-term backlogs during the measured business windows. Across the three selected high-frequency windows, all 570 tasks completed successfully without manual intervention. Recovery time is reported as “-” because no event requiring recovery occurred during these windows. The results demonstrate stable operation under the observed production load and effective handling of short-term backlogs. Longer observation periods and more diverse load profiles will be used in future work to characterize resource utilization and failure recovery.

6.5. Security Controls and Bidirectional Closed-Loop Operation

Because the system spans multiple security domains, security controls are embedded throughout acquisition, messaging, fusion, and publication rather than added as a separate function. The three-layer deployment limits direct exposure of intranet sources; Kerberos authentication, access control, protocol restrictions, and logging govern cross-domain access; and DMZ relaying with message buffering prevents direct intranet-Internet connectivity.

Bidirectional fusion does not imply a direct network connection between the intranet and Internet. Instead, DMZ relays, approved protocols, and message queues convert bidirectional service interaction into manageable and observable flows within established security boundaries. This design supports on-demand external services while preventing Internet-facing systems from directly accessing intranet data sources. The principal cross-domain control points, implementation mechanisms, functions, and corresponding operational evidence are summarized in Table 7.

Table 7. Cross-domain security controls and operational evidence.

Control Point	Implementation	Function	Operational Evidence
Identity authentication	Kerberos, service accounts, and access credentials	Prevent unauthorized access	Failed authentication attempts are rejected and logged
Access control	Allowlists, role permissions, topic permissions, and NiFi policies	Limit the scope of cross-domain access	Access policies and records are queryable
Transmission control	HTTPS, SFTP, Site-to-Site, and restricted Kafka protocols	Constrain data-transfer paths	Transfer configurations and connection logs are verifiable
Logging	NiFi logs, Kafka logs, and system-monitoring logs	Support auditing and fault diagnosis	Key tasks, exceptions, and state changes are recorded
Exception handling	Retries, queue buffering, and alerts	Reduce the effect of exceptions on service continuity	Failed tasks enter retry or buffering workflows

6.6. Comparison with Alternative Approaches

To clarify the applicability of the proposed design, the coordinated MiNiFi-NiFi-Kafka architecture was compared with traditional ETL, one-way file exchange, NiFi-only orchestration, RabbitMQ/ActiveMQ message queues, and general-purpose stream-processing frameworks such as Flink and Spark Streaming. The comparison considers platform functionality, message persistence and replay, flow-orchestration capabilities, published performance studies, and the security-boundary and closed-loop requirements of the target operational scenario [10,11,12,14,15,16,17,19,20,26]. The dimensions include adaptation to multiple security domains, bidirectional closed-loop support, streaming capability, asynchronous decoupling, deployment complexity, and target use cases. Table 8 compares the main characteristics, strengths, limitations, and relationships of these approaches to the proposed system.

Table 8. Comparison of data-integration approaches.

Approach	Main Characteristics	Strengths	Limitations	Comparison with the Proposed System
Traditional ETL	Batch-oriented centralized extraction, transformation, and loading	Simple implementation; suitable for periodic processing	Limited timeliness and workflow flexibility; weak support for cross-domain closed loops	The proposed system is better suited to streaming data, cross-domain services, and asynchronous processing
One-way file exchange	Transfer through directories, FTP/SFTP, or security gateways	Clear security boundary and mature engineering practice	Limited interaction, difficult state tracking, and restricted orchestration	The proposed system supports controlled outbound and inbound service loops
NiFi-only architecture	Flow orchestration and data routing	Rich visual orchestration and processor ecosystem	Burst buffering and long-term retention/replay often require an additional messaging component	Kafka adds persistent buffering, burst absorption, and replay
RabbitMQ/ActiveMQ	Traditional message queues	Effective task distribution, routing, and conventional messaging	High-throughput persistent stream retention and replay require workload-specific configuration	Kafka provides persistent stream storage and scalable consumption
Flink/Spark Streaming	Complex stream computation and large-scale analytics	Strong computation for sophisticated real-time analysis	Cross-domain access, security relaying, and visual orchestration require complementary components	The proposed system emphasizes controlled exchange, cross-domain orchestration, and closed-loop services
Proposed system	Coordinated MiNiFi-NiFi-Kafka stack with three isolated domains	Edge preprocessing, asynchronous decoupling, bidirectional workflows, and controlled data movement	Higher deployment and operational complexity; requires coordinated security and flow configuration	Designed for cross-domain meteorological data fusion and deployment in scientific data centers

Traditional ETL and one-way file exchange are appropriate for fixed batch-processing or one-way publication workflows but provide limited support for externally triggered closed-loop services. NiFi offers rich visual orchestration, while an independent messaging system strengthens buffering under burst traffic and supports longer message retention. RabbitMQ/ActiveMQ and Kafka emphasize different use cases: the former are well suited to task distribution and flexible routing, whereas Kafka is better aligned with the persistent stream, partition scaling, and replay requirements in this study. Flink and Spark Streaming focus on complex stream computation and require complementary components for cross-domain access, security relaying, and visual orchestration [10,11,12,14,15,16,17,19,20,26]. The proposed architecture combines MiNiFi-based edge acquisition and preprocessing, NiFi-based orchestration, Kafka-based asynchronous decoupling, and DMZ-based isolation and relaying; its principal advantage lies in bidirectional meteorological data fusion across multiple security domains.

6.7. Summary

Evaluation using operational records showed that edge preprocessing reduced cross-domain data volume by an average of 95%; outbound product services and inbound request-response workflows had mean end-to-end times of 75 and 37 s, respectively; the mean Kafka message-processing success rate from April to June 2026 was 99.77%; and all 570 tasks in the selected high-frequency windows completed successfully. These results demonstrate the contribution of the three-layer isolation architecture, edge-center collaboration, and asynchronous decoupling to representative meteorological services. Authentication policies, access control, transmission configuration, and operational logs further show that the system provides controlled access, state monitoring, and exception handling along the operational path.

7. Discussion

7.1. Technical Advantages and Interpretation of Operational Results

The central strength of the system is the integration of a three-layer isolation architecture, edge-center collaborative computing, Kafka-based asynchronous decoupling, and security controls within a single cross-domain meteorological workflow. The 95% average reduction in transferred data demonstrates the direct benefit of moving variable extraction and format conversion to the intranet edge. Mean end-to-end times of 75 s for outbound services and 37 s for inbound requests satisfy the current timeliness requirements for automated publication and on-demand response. The 99.77% mean message-processing success rate from April to June 2026, together with a consumption rate slightly higher than the production rate, indicates that Kafka buffering and consumer decoupling effectively handled the observed short-term backlog. These results demonstrate the operational effectiveness of the architecture at the current business scale and provide a basis for extending it to higher loads and additional services.

7.2. FairStack Deployment and Cross-Domain Applicability

The three-layer isolation architecture, edge-center coordination, asynchronous messaging, and secure exchange mechanisms developed in this system were generalized, modularized, and made configurable within the FairStack scientific data software stack. The system described in this paper is a representative meteorological implementation of these mechanisms, while FairStack provides reusable data access, flow orchestration, messaging, and secure exchange components for other scientific disciplines.

FairStack has been deployed at 66 scientific data centers, including 10 national-level centers, in fields such as ecology, space science, and agriculture. CoNet, a cross-disciplinary data-fusion service network built on FairStack, coordinates data and computing resources across six national scientific data centers and supports demonstration applications, including evaluation of the ecological benefits of check dams in the middle and upper reaches of the Yellow River. The figures refer to different deployment scopes: 66 centers use the FairStack software stack, whereas six centers participate in the CoNet cross-center coordination demonstration. These deployments indicate that the three-layer isolation, edge-center collaboration, and asynchronous decoupling mechanisms are applicable beyond meteorology to other scientific data-center scenarios.

7.3. Scope of Application, Evaluation Boundaries, and Future Work

The system is intended for meteorological and scientific-data services that require controlled exchange, fusion processing, and publication across multiple security domains. Its key strengths are explicit security boundaries, flexible flow orchestration, operational visibility, and component-level scalability. During deployment, NiFi, MiNiFi, Kafka, identity services, and boundary-security devices can be configured according to each organization’s network environment, security policies, and data-classification rules. Standardized ingestion templates and adapters can address domain-specific differences in data formats, quality control, and authorization policies.

The present evaluation uses real operational logs and representative workflows from the National Meteorological Science Data Center and focuses on data-volume reduction, end-to-end latency, message processing, and task execution at the current production scale. Because the production environment was not established as a controlled benchmarking platform, this study did not conduct systematic performance benchmarks under graded workload conditions or collect complete CPU and memory utilization profiles. The results therefore demonstrate operational feasibility and stability under the observed workloads rather than the maximum processing capacity of the architecture. Future work will extend the observation period and workload range and will evaluate resource utilization, concurrency, scalability, and failure recovery under controlled conditions.

It should also be noted that the security evaluation is based primarily on the system architecture, access-control configurations, and operational logs from the production environment and does not cover specialized security assessments such as penetration testing. The current results therefore indicate that the deployed controls support controlled cross-domain operation in the present business scenario. Future work will examine more complex security scenarios and further improve cross-center collaborative scheduling.

8. Conclusions

This study proposes and implements a coordinated bidirectional meteorological data fusion processing system for operational environments spanning the intranet, DMZ, and Internet. The system combines a three-layer isolation architecture with Apache MiNiFi, Apache NiFi, Apache Kafka, Kerberos, Redis/MySQL, and operational monitoring to provide edge acquisition and preprocessing, DMZ-based fusion, asynchronous messaging, security control, and bidirectional closed-loop services.

Operational records from the National Meteorological Science Data Center show that edge preprocessing reduced cross-domain data volume by an average of 95%; mean end-to-end times were 75 s for outbound product services and 37 s for inbound request-driven workflows; the mean Kafka message-processing success rate from April to June 2026 was 99.77%; and all 570 tasks in the selected high-frequency windows completed successfully. These results show that the system supports automated publication, on-demand processing, and controlled return of meteorological products at the current operational scale without changing existing network security boundaries, validating the three-layer isolation, edge-center collaboration, and asynchronous decoupling mechanisms in a real production environment.

FairStack, which generalizes the architecture and core mechanisms of the proposed system, has been deployed at 66 scientific data centers, including 10 national-level centers. CoNet, built on FairStack, coordinates data and computing resources across six national scientific data centers. These implementations indicate that the architecture has broader applicability to scientific data services beyond meteorology. Future work will extend long-term, multi-load, and cross-center evaluations and further develop cross-domain data standards and adaptive scheduling mechanisms.