An Architectural Optimization Framework for Scalable Spatial Clustering in High-Redundancy Environments

Abstract

Spatial Big Data mining is often hindered by high computational complexity and the intrinsic autocorrelation of georeferenced records. To address these challenges, this study proposes an architectural optimization framework for the CHSMST+ algorithm, designated as CHSMST+MR. Rather than introducing a brand-new clustering paradigm, the framework focuses on a Distributed Spatial Cardinality Reduction (DSCR) layer that aggregates redundant spatial records before the core iterative mining logic begins. By transforming raw records into a weighted key-value representation within the Apache Spark environment, the proposed approach significantly mitigates the shuffling bottleneck common in distributed systems. Experimental validation using high-density biological datasets demonstrates an average execution-time reduction of 51.36%, with performance gains reaching up to 79.96% in specific high-redundancy scenarios. The results, obtained through controlled local emulation, confirm that this architectural optimization provides a scalable, deterministic, and lossless solution for accelerating spatial clustering. This work contributes a methodological path for enhancing the performance of iterative spatial mining algorithms in environments characterized by massive data density and coordinate redundancy.

1. Introduction

The dissemination of high-resolution sensor networks and global positioning systems has moved spatial data from a specialized niche to the very center of modern Big Data analytics. Today, the challenge is no longer just how to store georeferenced records, but whether it is computationally feasible to extract meaningful patterns from them in real-time [1]. While Spatial Data Mining (SDM) draws heavily from traditional mining techniques, it faces the challenge of dealing with the complex geographic dependencies and non-stationary behaviors that define modern datasets. Standard architectures often fail to meet the iterative demands of spatial clustering, especially when forced to process the high-dimensional attributes required for current urban and environmental modeling [2].

Due to the inadequacy of traditional processing methods, there is a need to develop distinct approaches for all stages of large-scale spatial data management, from storage to analysis. The software ecosystem developed for Big Data management includes widely used open-source systems, such as Hadoop and Spark, which allow storage, processing, and analysis tasks to be performed more efficiently [3]. Recent advancements in distributed computing have positioned Apache Spark 3.5.0 as the leading framework for large-scale spatial analysis, surpassing legacy disk-based models like Hadoop in iterative performance. Despite these framework-level improvements, a significant research area remains focused on identifying the most computationally efficient clustering algorithms to handle massive workloads [4].

Spark MLlib acts as a library for distributed machine learning, leveraging broadcast variables and Resilient Distributed Datasets (RDDs) to parallelize iterative processes, such as K-means, across multiple data dimensions [1]. Expanding these capabilities to the geographic domain, Apache Sedona provides a specialized infrastructure by introducing Spatial RDDs (SRDDs) and spatial SQL tools designed to load and analyze large-scale georeferenced datasets [5]. However, while Sedona excels in spatial partitioning and visualization tasks [3], it typically treats each record as an independent object. This approach often leads to redundant distance calculations in high-density scenarios where numerous records share identical spatial coordinates [6]. Consequently, there remains a need for architectural optimizations capable of adding spatial redundancy at the data collection phase to improve scalability in Big Data environments.

In addition to the physical redundancy of coordinates, the inclusion of non-spatial attributes (qualitative or quantitative characteristics associated with each point) introduces an additional layer of complexity. As the dimensionality of the data increases, the effectiveness of traditional spatial indexing methods, such as R-trees or Quadtrees, tends to decrease. Data skewness can lead to bottlenecks where specific cluster nodes become overloaded while others remain underutilized. Therefore, a pre-aggregation strategy must not only consider geographic positioning but also ensure that the fusion of records preserves the integrity of non-spatial information, ensuring that the subsequent mining process remains statistically valid [7].

To address these challenges, this study proposes an architectural optimization of the CHSMST+ algorithm, referred to as CHSMST+MR. Rather than directly parallelizing the clustering logic, the proposed approach introduces a MapReduce-based pre-aggregation layer within the Apache Spark environment. By leveraging a (Key, Value) structure (where the key is defined by spatial coordinates and non-spatial attributes, and the value represents the occurrence count) the dataset size was reduced. This strategy not only preserves the deterministic nature of the original clustering logic but also provides a scalable framework for handling high-density spatial datasets.

The MapReduce layer implemented in this work via Apache Spark 3.5.0 performs cardinality reduction without loss of information. Mathematically, grouping N records located at exactly the same coordinate is equivalent to processing a single record weighted by N. As a consequence, the reported performance gains do not result in distortions in the final clusters, maintaining the same hypersurface topology and Minimum Spanning Tree (MST) refinement defined in the original algorithm, but with a fraction of the computational cost in terms of I/O and memory [8].

In this study, the validation of the CHSMST+MR framework is conducted using a biological occurrence dataset from the Academy of Natural Sciences [9], where coordinate redundancy is frequent due to multiple sightings at fixed monitoring locations. The primary objective is to demonstrate the reduction in execution time and analyze how these savings scale as attribute complexity varies.

The rest of this paper is organized as follows. Section 2 presents a review of related works. Section 3 describes the theoretical background, including the foundations of the original CHSMST+ algorithm. Section 4 details the proposed CHSMST+MR architectural optimization, its formalization through the DSCR layer, and the specific implementation within the Apache Spark environment. Section 5 provides the experimental setup and the obtained results. Section 6 presents a qualitative analysis and a critical discussion of the framework’s performance, scalability, and limitations. Finally, Section 7 concludes the paper and outlines future research directions.

Objectives and Methods

Given the critical role of spatial data in modern analytics, this work focuses on leveraging parallelization to develop an architectural optimization framework. The primary objective is to bridge the performance gap found in Big Data environments by specifically addressing the computational bottlenecks caused by spatial redundancy in high-density regions. By shifting the focus toward a density-optimized data model, the study aims to provide a solution that scales effectively without compromising the mathematical precision required for complex spatial analysis.

The core scientific contribution of this paper lies in the integration of parallelization logic during the initial data collection stage. Since high-dimensional georeferenced data requires substantial processing power, the proposed architecture focuses on improving execution efficiency while strictly maintaining the statistical and deterministic integrity of the mining process. This design choice ensures that the reduction in computational cost does not result in any information loss or distortion of the final clusters.

The development process was informed by a technical survey of Big Data architectures and the MapReduce model. Building upon these foundations, an architectural enhancement of the CHSMST+ algorithm [10] was engineered. Apache Spark was utilized as the distributed engine to implement a pre-aggregation layer where records sharing identical spatial and non-spatial identities are collapsed into single weighted representative points. This transformation effectively reduces dataset cardinality by eliminating duplicate coordinate processing before the core clustering kernels are executed.

To validate the efficiency of this framework, a comparative analysis was performed against established methodologies, focusing on input parameter requirements and noise handling. The evaluation concluded with rigorous performance and scalability tests conducted under controlled local emulation, using massive high-density datasets to analyze the framework’s behavior in scenarios defined by significant coordinate redundancy.

2. Related Work

A set of papers on the subject in question was researched and studied within the main means of scientific dissemination, these papers have in common the implementation of parallelization resources, such as the MapReduce method for performance optimization in spatial data mining algorithms.

Yue et al. present a new clustering algorithm which uses K-Medoids instead of K-Means [11], which makes the algorithm less sensitive to noise. The K-Medoids clustering algorithm has a high running time when processing massive two-dimensional spatial points, although it is robust in relation to outliers. For this reason, it became necessary to use a strategy to improve the performance of the algorithm in large amounts of data. The experimental results confirm that the use of MapReduce has effectively contributed to improving the performance of the algorithm.

Lang and Chai investigate the efficiency of clustering algorithms for mining massive GPS data from taxi trajectories [8]. The authors proposed an optimization of the K-means algorithm by using the Canopy algorithm for pre-clustering, which effectively solves the problem of initial centroid selection and reduces the number of iterations. By parallelizing the implementation on the Apache Spark framework and utilizing Broadcast variables to minimize data transmission overhead, the study achieved significant improvements in computational speed and efficiency for urban traffic management systems.

Manjula and Narsimha propose an effective process for soil classification which uses input data from a soil database called Harmonized World Soil Database [12]. Pre-processing helps to generate an improved representation and uses the least amount of time possible. Then the MapReduce framework divides the input dataset into a complementary part. In the map task, the main component analysis is used to reduce the data, and the outputs contribute to reducing the tasks. Finally, the proposed process is employed to categorize soil type using an optimal neural network classifier. The conventional network uses the optimization process for its customization. Derived from the classifier, the soil type is categorized. The performance of the procedure is evaluated using various metrics, such as sensitivity, specificity, accuracy, precision, recall and F-measure. The results of the analysis illustrate that the process has a high accuracy, which validates the purpose of meeting the objective of soil classification.

Awaad and Hefny propose a parallel implementation of a statistical DBSCAN algorithm (SCbs-DBSCAN) designed for Spark-based clustering on the Google Cloud platform [7]. To overcome the challenges of uneven data distribution in large spatial datasets, the authors used R-tree partitioning to create balanced workloads across distributed executors. Furthermore, they integrated statistical analyses to automatically determine the ideal clustering parameters. The results showed that the proposed approach is up to 3 times faster than other parallel DBSCAN methods, highlighting the crucial role of efficient data partitioning and resource management in Big Data environments.

Sheena et al. proposed a new solution for classification of spatial data is proposed here, which uses an assorted linear method based on Pareto Exponentiated [13]. The algorithm applies the Pareto exponentiated technique to find recurrence relations between properties of spatial data, so that spatial association rules are generated, and the data are passed to the linear sorted method. After applying the methods, efficient classification of spatial data is generated. Such classification was analyzed through experimental works, which are based on metrics of precision, classification, time complexity and space complexity. Experimental results show that the proposed technique is able to increase classification accuracy and reduce time complexity.

Li et al. addresses the high computational complexity of area skyline queries in map-based Big Data [6]. They developed a distributed algorithm on the Apache Spark framework that employs a local partial skyline extraction technique. This method minimizes the volume of data transmitted from parallel executors to the central driver by filtering irrelevant spatial points in the early stages of processing. Their experimental results demonstrate that leveraging Spark’s distributed capabilities significantly enhances scalability and reduces the computation time required for complex spatial decision-making queries.

3. Background

Upon taking into consideration the restrictions of the CHSMST algorithm, such as the need for input parameters and not considering non-spatial attributes in the formation of clusters, CHSMST+ was proposed. A performance improvement strategy with multithreading was also applied. Such strategy was possible because the construction of hypersurfaces by the CHS algorithm facilitates the incorporation of such a technique, since dividing the space into units, all processing in each unit is performed independently. Furthermore, as the units can also be subdivided into smaller regions, the subdivision itself could be carried out in parallel [10].

In order to eliminate the input parameters, the calculation of local and global distances was introduced, in which the average distance of the objects in each unit is calculated, and then the average of these distances is obtained for all units. Thus, the algorithm will subdivide a unit if the local distance is greater than the global distance, which allows to reduce the spatiality of the data within the units [10].

With the changes made, the algorithm now requires only the data set, and no parameter defined by the user to run the algorithm. The CHS algorithm performs both the stages of distributing the points in the initial rectangle and subdividing it into smaller units in CHSMST+. The division of the units into smaller regions is performed asynchronously through threads, while the number of subdivisions is defined automatically by calculating the local and global distance [10].

By the end of the thread’s execution, the initial units are obtained, and then the process of merging the adjacent units is carried out in order to contemplate the respective samples in a single unit. After this step, any samples belonging to adjacent units are grouped and a new unit is generated considering the borders of the previous units [10].

4. Proposed CHSMST+MR Algorithm

CHSMST+MR was developed as an architectural optimization of the spatial data clustering process, designed to handle High-Density Spatial Big Data by integrating a MapReduce-based pre-aggregation layer within the Apache Spark framework. While traditional distributed approaches attempt to parallelize iterative clustering logic directly on raw data, this framework focuses on reducing data cardinality before the core mining stage begins. This study evolves the original CHSMST+ algorithm by introducing a Distributed Spatial Cardinality Reduction (DSCR) phase, which solves the common “identical coordinate bottleneck” in massive georeferenced datasets.

As illustrated in Figure 1, the CHSMST+MR workflow is structured into three technical zones: (i) Ingestion and Spark Setup, where the JavaSparkContext is initialized and data is partitioned; (ii) Cardinality Reduction, where the MapReduce engine collapses redundant records; and (iii) Core Clustering, where the Parallel Clustering based on Hyper Surface (CHS) and Minimum Spanning Tree (MST) refinement are executed. This layered approach ensures that the subsequent Minimum Spanning Tree (MST) refinement operates on a reduced set of unique spatial keys, maintaining statistical integrity while decreasing the iterative overhead.

Figure 1. Detailed architectural workflow of the CHSMST+MR framework. The process illustrates the transition from raw record ingestion (N) to a weighted representation (U) via Spark MapReduce transformations, followed by the parallel execution of the CHS and MST clustering kernels.

4.1. Implementation with Apache Spark

The implementation of the CHSMST+MR logic is operationalized through a distributed pipeline that follows the technical zones defined in Figure 1. The pre-mining process is designated as Distributed Spatial Cardinality Reduction (DSCR), as formalized in Algorithm 1. The core of this architecture is an identity-preserving mapping function, $\varphi (s_i,a_i)$, which ensures that the statistical integrity of both spatial (s) and non-spatial (a) data is maintained while reducing the cardinality of the input.

Algorithm 1 Distributed Spatial Cardinality Reduction (DSCR)

1: Input: Raw dataset $D=\{r_1,r_2,\dots ,r_n\}$, where $r_i=(s_i,a_i)$ (s: spatial coordinates, a: non-spatial attributes)   2: Output: Reduced weighted dataset $D^{\prime }=\{(k_1,w_1),(k_2,w_2),\dots ,(k_m,w_m)\}$   3: {Phase 1: Distributed Partitioning}   4: $RDD\leftarrow \mathrm{Distribute}D\mathrm{across}P\mathrm{computing}\mathrm{partitions}$   5: {Phase 2: Mapping and Key Synthesis}   6: for all record $r_i\in RDD$ in parallel do   7:     $k_i\leftarrow \varphi (s_i,a_i)$ {Identity mapping function for spatial and attribute data}   8:     Emit pair $(k_i,1)$   9: end for 10: {Phase 3: Distributed Reduction} 11: $D^{\prime }\leftarrow \mathrm{Aggregate}\mathrm{pairs}\mathrm{by}\mathrm{key}k\mathrm{such}\mathrm{that}{w}_k=\sum \mathrm{occurrences}$ 12: {Phase 4: Output} 13: return $D^{\prime }$

As shown in Zone 1 of Figure 1, the process begins with the ingestion of the raw spatial dataset (Step 1) and the initialization of the Spark Context (Step 2). In Step 3, the RDD is partitioned into logical units to maximize the capacity of the testing hardware.

Operationalizing Algorithm 1, the Map function (Step 4) reads the points and creates pairs containing their coordinates and characteristics. In this stage, the collected data is divided into small datasets, where it is transformed into a key-value pair in parallel. The key k is uniquely defined by the X and Y coordinates combined with the non-spatial attributes, while the value is initialized as a unit count.

Then, the Reduce function (Step 5) reads the pairs and calculates occurrences of points with the same coordinates and characteristics and associates these values to each pair. These pairs generated are then merged (Step 6), where a tuple is created with the pairs that have the same values and their total number of occurrences during the data collection. The output file has the tuples generated with the coordinates, characteristics and the number of occurrences of these points.

This deterministic reduction ensures that the total number of physical points is preserved via the weight attribute, but the actual number of records to be processed by the core clustering logic is minimized. This ensures that the subsequent Parallel Clustering based on Hyper Surface (CHS) (Step 7) and MST refinement (Step 8) operate on a lower-cardinality input, providing an improvement in execution time.

Table 1 shows what the data looks like before applying MapReduce. It is possible to observe that records 1, 2 and 3 have the same coordinates; however, only records 1 and 3 have the same non-spatial characteristics. In the same way, the records 5 and 6 also have the same coordinates, but different non-spatial characteristics.

Table 1. Example of the data collected. The geographic X and Y coordinates of the record are collected, as well as non-spatial characteristics that are important for creating the clusters.

X	Y	Age Group	Order
−51.3427279	−20.4259774	Young	Carnivora
−51.3427279	−20.4259774	Adult	Rodentia
−51.3427279	−20.4259774	Young	Carnivora
−50.9399757	−20.2197357	Young	Rodentia
−50.9399757	−20.2197357	Young	Passeriformes
−50.9399757	−20.2104868	Young	Passeriformes

In this context, when applying Apache Spark to this example, it is possible to obtain the following data shown in Table 2. In reference to this figure, it is possible to observe the decrease in the records to be treated and that the number of occurrences of them still remains.

Table 2. Data after applying Apache Spark. After application, data that have the same geographic coordinates and the same spatial characteristics are combined into one record, in which quantity information is added.

X	Y	Age Group	Order	Occurrences
−51.3427279	−20.4259774	Young	Carnivora	2
−51.3427279	−20.4259774	Adult	Rodentia	1
−50.9399757	−20.2197357	Young	Rodentia	1
−50.9399757	−20.2197357	Young	Passeriformes	1
−50.9399757	−20.2104868	Young	Passeriformes	1

4.2. Computational Complexity and Scalability Analysis

The scientific impact of the DSCR process (Algorithm 1) was primarily evaluated through its effect on asymptotic complexity. By distinguishing between the total number of raw spatial records N and the count of unique spatial attribute keys U it is possible to formally justify the performance gains. In high-density scenarios, such as the biological datasets used in this study, it was observed that U is significantly smaller than N ($U\ll N$), representing a large optimization opportunity.

While the original CHSMST+ algorithm scales as a function of the raw input size, $O\left(f\right(N\left)\right)$, this architectural change moves the workload from the most costly iterative stages to U. Consequently, the complexity of the main clustering is reduced to $O\left(f\right(U\left)\right)$. Although the pre-aggregation phase introduces a linearly distributed cost of $O\left(N\right)$, this is an “embarrassingly parallel” operation that Spark’s shuffling mechanism handles efficiently. By offloading redundancy at the origin, the subsequent non-linear mining logic operates on a much lower cardinality, which explains the drastic reduction in execution time.

4.3. Spark Configuration and Reproducibility

To ensure that the results are reproducible and to provide a benchmark for future research, the specific configuration of the Apache Spark environment redis detailed below. The system was configured to utilize the full computational capacity of the testing hardware (Intel i5, 12 threads). The JavaSparkContext was initialized using the following parameters:

• Master Mode: local[*], enabling parallel execution across all available CPU cores.
• Executor Memory: 2048 MB, ensuring sufficient heap space for high-density spatial aggregations.
• Data Partitioning: The RDDs were partitioned following the standard parallelism hint of at least twice the number of available threads ($2\times 12=24$ partitions). This ensures optimal CPU utilization, minimizes idle time between tasks, and manages the data shuffling process efficiently during the MapReduce stage.
• Persistence Strategy: RDDs were cached in memory to avoid redundant I/O during the iterative stages.

4.4. Technologies

For the development of the application, due to its use in the base work and its connection capabilities with the DBMS used and other Web services, the Java language was used. The development environment was NetBeans 13.

The database management system used was PostgreSQL 12.0, a powerful open-source object relational database system that uses the SQL language. For the treatment of spatial data, the PostGIS extension of PostgreSQL was used, where it offers many resources for the treatment of georeferenced data, rarely found in other competing spatial databases.

5. Results

This section presents the experimental setup, the dataset characteristics, and the performance results obtained after implementing the Distributed Spatial Cardinality Reduction (DSCR) layer within the CHSMST+ algorithm.

5.1. Setup and Evaluation Methodology

The experimental evaluation was conducted on a system equipped with an Intel Core i5-10400F CPU (2.90 GHz, 12 logical threads) sourced from Intel Corporation, Santa Clara, CA, USA, 16 GB of RAM, and the Windows 10 operating system. The software environment utilized the Apache Spark framework configured in local mode, a configuration that allowed to force the engine to utilize all available processor cores, effectively simulating the behavior of a multi-executor distributed cluster within a controlled environment.

For validation, georeferenced biological occurrence records from the National Academy of Natural Sciences [9], as previously mentioned, were used. This dataset consists of 50,000 georeferenced records. While this volume is smaller than some global benchmarks, it was intentionally selected as a High-Density Proof of Concept (PoC). In spatial data mining, a critical computational bottleneck is spatial redundancy—where multiple records share identical or near-identical coordinates [7]. This specific dataset provides a high frequency of coordinate overlap ideal for evaluating how the MapReduce-based pre-aggregation layer minimizes iterative overhead without losing statistical integrity.

The performance tests were structured into three distinct scenarios, where it systematically varied the number of non-spatial features (1, 2, and 3) included in the records. This variation was essential to observe how the uniqueness of the aggregation key k influences the cardinality reduction rate. To provide a clear comparison, the proposed CHSMST+MR framework was compared with the original CHSMST+ algorithm, which uses multithreading but does not have the distributed MapReduce aggregation stage.

To address the inherent runtime variability of distributed systems, we calculated the Standard Deviation (SD) for each scenario. Given the deterministic nature of the new lossless aggregation logic, the observed variance is minimal, ensuring that the performance gains are statistically significant and reproducible.

5.2. Obtained Results

In the first test stage, the characteristic order was considered, in a dataset with 50,000 records. Table 3 shows the execution times for each test and Figure 2 shows the chart with the values.

Table 3. Execution times with characteristic order.

Experiment	with MapReduce (in s)	Without MapReduce (in s)	SD (with MapReduce)
1	1118	4692	-
2	866	4826	-
3	888	4819	-
Average	957.33	4779	139.75

Figure 2. Chart with execution times with one non-spatial characteristic. Executions using MapReduce had an average time reduction of around 79.96% compared to executions without the method.

It can be seen from the chart shown in Figure 2, that the application of MapReduce together with Multithreading presented a lower time than the other applications, with an average reduction of approximately 79.96% in relation to the execution without MapReduce. For the second stage of testing, the same structure as the first was maintained, but the gender attribute was added. Table 4 shows the execution times for each case and Figure 3 shows the respective chart.

Table 4. Execution time with order and genre characteristics.

Experiment	with MapReduce (in s)	Without MapReduce (in s)	SD (with MapReduce)
1	2749	5852	-
2	3078	5835	-
3	2743	6213	-
Average	2858.33	5966.66	191.75

Figure 3. Chart with execution times with two characteristics. Executions using MapReduce had an average time reduction of around 52.09% compared to executions without the method.

From Figure 3, it can be seen that, for the second test stage, the proposed algorithm also presented a lower time compared to the other tests, with an average reduction of approximately 52.09%. For the third and final testing stage, the previous structures were maintained, and the location attribute was added. The execution times for each case are presented in Table 5 and in Figure 4 the respective graph.

Table 5. Execution time with order, genre, and location characteristics.

Experiment	with MapReduce (in s)	Without MapReduce (in s)	SD (with MapReduce)
1	4093	6423	-
2	4545	7315	-
3	4151	6513	-
Average	4263	6750.33	245.86

Figure 4. Execution time with three characteristics. Executions using MapReduce had an average time reduction of around 22.03%. The greater the number of non-spatial features, the smaller the time reduction.

Upon following the same model, it can be seen on Figure 4 that the execution of the algorithm with MapReduce and Multithreading with 3 non-spatial characteristics, maintained a lower execution time than the other applications, with an average reduction of approximately 22.03% in comparison to the algorithm without MapReduce.

6. Discussion

From the results obtained, it was possible to observe a decrease of about 79.96% of the processing time when using one non-spatial attribute for the detection of patterns and formation of clusters, but when using two attributes, the reduction decreased to 52.09% and, consequently, adding a third attribute decreased the reduction a bit more, down to 22.03%.

As a consequence, it was possible to detect that as the number of non-spatial features increases, the time reduction through parallelization is reduced. This is due to the fact that the more attributes selected, the smaller the chance of them being equal becomes, so the number of points to be treated is not reduced as it would be with fewer non-spatial features. The algorithm’s time reduction curve can be seen in Figure 5, the average execution times are used for each test.

Figure 5. Graph with average execution times. As the number of non-spatial attributes increases, execution time increases and the time reduction with MapReduce also decreases.

6.1. Qualitative and Architectural Comparison with Distributed Frameworks

To understand the performance gains reported in this study, it is essential to compare its architectural principles with standard distributed libraries, specifically Spark MLlib and Apache Sedona. While these frameworks are widely regarded as the state-of-the-art for Big Data analytics, they operate under different paradigms regarding spatial data representation and processing flow.

6.1.1. Comparison with Spark MLlib

Spark MLlib is recognized for its ability to scale iterative tasks, often performing up to 100 times faster than legacy disk-based systems like Hadoop [14]. According to Hasan et al. (2022) [14], MLlib provides extraordinary capabilities for grouping and rule extraction, making it a robust stage for powerful Big Data examination. However, recent findings by Theodorakopoulos et al. (2025) [15] reveal that the resource consumption of standard MLlib operators, such as K-Means, remains highly unpredictable on massive datasets. Even with predictive hyperparameter tuning, the framework often struggles to maintain real-time scalability.

By treating every georeferenced record as an independent object in a standard RDD, MLlib incurs massive computational waste in high-density scenarios. The CHSMST+MR addresses this through a structural shift. By collapsing identical coordinates into weighted records before clustering begins, it bypasses the iterative overhead that makes standard MLlib implementations to exhibit performance fluctuations [15].

6.1.2. Comparison with Apache Sedona

Apache Sedona is specifically engineered for high-performance spatial operations, introducing Spatial RDDs (SRDDs) and sophisticated indexing structures like R-trees and Quadtrees [5]. Sedona’s strength lies in its ability to optimize spatial joins and range queries through intelligent space partitioning [3].

However, the analysis identifies a qualitative gap between partitioning and cardinality reduction. Sedona excels at moving data to the right nodes, but it does not natively aggregate spatial redundancy within those partitions. For instance, if a dataset contains 100 identical observations at a single monitoring station, Sedona will efficiently route them, but a standard clustering algorithm on top of it might still evaluate all 100 points individually. The proposed framework complements Sedona’s partitioning strengths by ensuring the input is “clean” of redundancy at the collection stage. This architectural choice is the primary driver of our reported speedup, as it mitigates the shuffling bottleneck that typically degrades performance in multi-node environments [6].

6.2. Comparative Summary of Framework Features

The architectural distinctions summarized in Table 6 highlight that while general-purpose frameworks focus on the process of mining, CHSMST+MR focuses on the input model.

Table 6. Architectural and functional comparison between Spark MLlib, Apache Sedona, and the proposed CHSMST+MR framework.

Feature/Criteria	Spark MLlib	Apache Sedona	CHSMST+MR (Proposed)
Primary Purpose	General Distributed ML	High-performance Spatial SQL	Optimized Spatial Data Mining
Data Representation	Standard RDD/Dataframe	Spatial RDD (SRDD)	Weighted (Key-Value) RDD
Spatial Awareness	Low (General Purpose)	High (Indexing-based)	High (Pre-aggregation-based)
Redundancy Handling	None (Processes all points)	Low (Spatial Indexing only)	Maximum (MapReduce Collapse)
Bottleneck addressed	Iterative CPU load	Spatial Join/Partitioning	Spatial Shuffling and Cardinality
Clustering Logic	Standard (e.g., K-Means)	Standard (e.g., DBSCAN)	Hybrid (HyperSurface + MST)
Scaling Metric	Total number of records	Total number of records	Number of unique coordinates

6.3. Limitations and Constraints

Despite the significant efficiency gains, it is recognized that the CHSMST+MR framework operates within specific boundary conditions. Its performance is intrinsically tied to the spatial density of the data; in datasets where georeferenced points are highly dispersed, the MapReduce pre-aggregation layer may introduce a slight computational overhead without a proportional reduction in cardinality.

Furthermore, while the current results were obtained through a controlled local emulation on a single-node cluster, we acknowledge that the number of experimental runs was limited to three. Although the deterministic nature of the DSCR logic inherently minimizes algorithmic variability, three repetitions may not fully capture the complex runtime fluctuations of massive distributed systems. Consequently, future work should expand the statistical scope to 10–20 runs within multi-node cloud environments (e.g., Amazon EMR or Azure HDInsight). Such a scale would allow for a more robust variance analysis, specifically addressing the impacts of network latency and inter-node communication costs on the cardinality reduction rate. Recognizing these constraints is essential for the practical deployment of this architectural optimization in diverse Big Data scenarios.

7. Conclusions

This research presented the implementation of a MapReduce-based pre-aggregation stage into the CHSMST+ algorithm using the Apache Spark framework. Designated as the Distributed Spatial Cardinality Reduction (DSCR) layer, this architectural optimization effectively reduced the number of records to be processed during the mining phase, leading to a significant decrease in execution time compared to the base implementation. The experimental results, obtained through controlled local emulation, confirm that integrating a distributed data reduction layer before the clustering logic provides a scalable and efficient solution for handling High-Density Spatial Big Data.

The analysis indicated that while the execution time increases as more non-spatial attributes are selected—due to the higher uniqueness of the aggregation keys—the framework maintains its performance advantage. The synergy between distributed pre-aggregation and multithreading was found to be a robust strategy for mitigating the “identical coordinate bottleneck.” Consequently, it is established that the proposed CHSMST+MR framework is a viable methodological path for spatial data mining, offering a predictable path for scaling iterative algorithms in high-redundancy environments.

By addressing the intersection of georeferenced databases and parallel computing, this study optimized the data collection phase for knowledge extraction, thereby minimizing both computational overhead and execution latency. The primary scientific contribution of this work is the demonstration that performance gains in Spatial Big Data can be achieved through architectural shifts in data representation. By collapsing redundant spatial records into weighted entities, the framework ensures that iterative mining logic operates on a streamlined cardinality without compromising the deterministic integrity or the statistical topology of the final clusters.

Since the MapReduce optimizations were focused on the data collection stage, the implementation of similar distributed strategies within the cluster formation and refinement stages is proposed as future work. Such a development would allow for further end-to-end optimization of the mining process. Additionally, future research should explore the integration of more complex attribute characteristics and evaluate the framework’s behavior in massive multi-node cloud clusters to analyze the impact of network latency on the cardinality reduction rate.