Intelligent Hybrid Caching for Sustainable Big Data Processing: Leveraging NVM to Enable Green Digital Transformation

Abstract

Apache Spark has gained widespread adoption for large-scale data processing. However, conventional caching methods inadequately address the dual challenges of performance bottlenecks and escalating energy consumption in data-intensive workloads. This paper introduces a sustainable computing framework that integrates Directed Acyclic Graph (DAG) dependency analysis with garbage collection (GC) behavior monitoring to optimize data placement between DRAM and non-volatile memory (NVM). The proposed Intelligent Hybrid Caching Management Framework (IHCMF) dynamically predicts data access patterns and migrates cache blocks based on cost–benefit analysis, achieving a 37.5% execution time reduction over default Spark configurations in SparkBench evaluations. By improving throughput-per-watt and projecting potential benefits from NVM’s near-zero idle power and extended hardware lifespan, IHCMF provides a scalable, cost-effective caching solution for resource-constrained edge computing environments. This work demonstrates that high-performance computing can be reconciled with environmental sustainability through intelligent memory management.

1. Introduction

With the rapid evolution of big data technology and the increasing prevalence of Internet of Things (IoT) devices, Apache Spark has established itself as an effective and scalable distributed computing framework that is extensively used for data processing and analysis tasks [1]. The exponential growth of big data analytics has elevated data center electricity consumption to 1.5% of global usage (exceeding 4% in advanced economies such as the United States), with projections indicating it will reach 3% by 2030 [2,3]. Amid escalating carbon neutrality commitments, energy-efficient memory technologies capable of reconciling computational expansion with environmental constraints have become imperative [3]. To address these energy challenges, hybrid memory architectures utilizing non-volatile memory (NVM) have demonstrated significant potential for improving energy efficiency in large-scale computing clusters [4]. Recent MIT analysis quantified the carbon footprint of extreme-scale AI workloads, revealing that training a single generative model generates ~500 metric tons of CO₂ [5]. Such intensifying computational demands extend beyond AI training to general big data analytics, underscoring the critical need for sustainable computing infrastructures. Non-Volatile Memory (NVM) offers distinct sustainability advantages—theoretical near-zero standby power (eliminating DRAM’s energy-intensive refresh cycles), higher density, and extended hardware lifespan—yet realizing these benefits requires sophisticated software optimization and physical hardware deployment to bridge the performance gap with conventional DRAM.

However, Apache Spark’s effectiveness is limited by traditional caching strategies, including Least Recently Used (LRU) and Least Frequently Used (LFU). These approaches often result in suboptimal system performance due to their limited adaptability to dynamic data access patterns and insufficient consideration of cache migration overheads [6,7]. These limitations are particularly evident in contemporary data-intensive workloads, leading to reduced cache hit rates and inefficient utilization of computing resources [8]. Although recent studies have investigated Directed Acyclic Graph (DAG)-based dependency analyses [9,10] and hybrid memory architectures combining dynamic random-access memory (DRAM) and non-volatile memory (NVM) [11,12], existing methods often disregard important cost–benefit trade-offs associated with data migration [9] and fail to integrate caching strategies with garbage collection behaviors [10], thereby revealing substantial opportunities for further performance optimization.

To address the limitations of existing caching strategies in hybrid memory systems, this study investigates the following research questions:

(1)

How can DAG dependency analysis (reuse distance and frequency) be effectively combined with GC behavior monitoring to predict data access patterns and guide cache placement decisions in Apache Spark?

(2)

What cost–benefit model can quantitatively evaluate the trade-offs between data migration overhead and cache performance gains to enable dynamic optimization of DRAM-NVM data placement?

(3)

How does the proposed IHCMF strategy perform across diverse workloads and varying DRAM-NVM capacity ratios compared to default Spark caching mechanisms, and what are the resulting implications for energy-efficient computing?

2. Related Work

This section reviews existing research on cache management strategies in big data processing frameworks, with a specific focus on Apache Spark. The related studies are organized into four primary categories: classical caching strategies, DAG-based caching strategies, hybrid memory management, and machine learning-based approaches.

2.1. Classical Caching Strategies

Classical cache management strategies, such as LRU and LFU, are commonly adopted in computing frameworks due to their simplicity and ease of implementation [13,14,15]. These approaches determine cache block priorities based on either the recency of access (LRU) or the frequency of access (LFU) of data blocks [16]. However, these conventional methods reveal substantial limitations when handling large-scale datasets. Specifically, LRU fails to capture the long-term relevance of data blocks, whereas LFU responds inadequately to bursty or transient access patterns [17]. Although Apache Spark employs the LRU policy by default, its performance is often suboptimal in large-scale data processing scenarios [6]. Consequently, these classical strategies perform poorly in dynamic and data-intensive environments, particularly in predicting the long-term value of data blocks, highlighting the need for more adaptive and intelligent caching mechanisms capable of responding to evolving data access patterns [7].

2.2. DAG-Based Caching Strategies

Recent research has increasingly emphasized the utilization of the DAG structure of Resilient Distributed Datasets (RDDs) in Apache Spark for cache management optimization [10,18]. By analyzing data dependencies within the DAG, these approaches predict future access patterns of data blocks, thereby enabling more informed cache placement decisions [16]. MemTune, for example, dynamically enhances data sharing and cache allocation by leveraging early dependency information extracted from the DAG, resulting in substantially enhanced cache hit rates [19]. However, most DAG-based caching strategies operate under the assumption of static or predictable data access patterns, an assumption that often fails in real-world scenarios where access behaviors evolve dynamically [9]. Additionally, these strategies often disregard the performance costs associated with cache migration, which can introduce unnecessary overhead [20]. The Least Reference Counting (LRC) algorithm, for instance, traverses the DAG to compute re-reference counts for each data block, thereby prioritizing the eviction of cache blocks with fewer references [9]. Similarly, the Reuse Distance Estimation (RDE) method utilizes DAG-derived dependency and reference information to optimize cache management by identifying and eliminating redundant RDDs [20]. Although these approaches highlight the potential of DAG-based analysis in improving caching efficiency, they lack adaptability to dynamic execution environments and fail to incorporate comprehensive cost–benefit evaluation of cache migrations.

2.3. Hybrid Memory Management

With the rapid advancement of NVM technology, researchers have increasingly explored the integration of DRAM and NVM to develop hybrid cache systems [11]. Recent CXL-based memory expansion [21,22] and concurrent garbage collection advances [23] further enhance these systems by enabling hardware–software co-design, collectively bridging the performance gap between heterogeneous memory tiers. These systems combine the low latency and high bandwidth of DRAM with the large capacity and persistence of NVM to enhance overall caching efficiency and performance. However, most existing studies on hybrid caching focus primarily on data block allocation across different storage media, often overlooking the dynamics and costs associated with data migration [10]. Furthermore, many studies assumed comparable performance between NVM and DRAM, whereas practical implementations reveal significant disparities in latency and bandwidth, often resulting in suboptimal cache performance [24]. Panthera represents a notable framework for hybrid memory management in big data processing environments, as it analyzes program memory access patterns to dynamically optimize memory allocation between DRAM and NVM, thereby reducing garbage collection overhead [25]. Other relevant approaches include G1 GC optimization, which adjusts garbage collection parameters to improve performance in large memory workloads, and off-heap memory management, which minimizes Java Virtual Machine (JVM) heap memory usage to lower GC pressure [25]. Despite these advances, current research on hybrid caching requires greater emphasis on dynamic data migration and cost–benefit analysis, underscoring the need for adaptive systems capable of optimizing migration strategies while maintaining cost-effectiveness [26,27]. Recent CXL advances enable heterogeneous memory hierarchies [21]. While this study uses RamDisk simulation, IHCMF is compatible with CXL architectures.

2.4. Machine Learning-Based Approaches

Machine learning algorithms have been increasingly integrated into cache management frameworks to predict data block access patterns and dynamically adjust caching strategies [28]. These approaches enable intelligent and context-aware adaptation to varying workloads and data access behaviors. However, many existing machine learning methods rely extensively on historical training data, which can degrade prediction accuracy when confronted with sudden or non-linear access fluctuations [29]. In addition, the training and continuous updating of machine learning models demand substantial computational and memory resources, posing significant challenges in resource-constrained environments [30]. Recent work has explored deep reinforcement learning for cache replacement decisions in cloud block storage [31] and edge computing networks [32], achieving superior hit rates but requiring significant computational overhead for model training and inference. For instance, Yang et al. proposed an efficient DRL-based policy to mitigate the high computational costs associated with standard deep Q-networks [31], while Chuan et al. employed federated deep reinforcement learning to address the resource demands of centralized training in mobile edge environments [32]. Kleio represents an innovative framework that integrates computer vision-based machine intelligence with hybrid memory management, using machine learning to optimize page scheduling and data placement [28]. Similarly, Cronus extends this concept by employing computer vision-based machine–intelligent hybrid memory management to enhance adaptive caching performance [30]. While these approaches demonstrate considerable potential, they struggle to efficiently manage bursty and nonlinear access patterns while maintaining acceptable computational overhead. Current machine learning-based cache management techniques remain limited in addressing sudden and non-linear access patterns while maintaining high computational efficiency, emphasizing the need for more adaptive and resource-efficient approaches [29,30].

2.5. Edge Computing and IoT Data Processing

Edge computing and IoT data processing environments present unique challenges stemming from resource constraints, latency requirements, and intermittent network connectivity [33,34]. These factors necessitate the adoption of adaptive caching management strategies capable of dynamically responding to fluctuating workloads and variable operating conditions [35]. Recent studies have investigated hybrid memory management techniques, particularly the integration of DRAM and NVM, to mitigate these limitations [36]. Compared with centralized cloud infrastructures, edge computing devices typically exhibit restricted computational capabilities and limited storage capacity, making efficient resource allocation and cache optimization critical [37]. Furthermore, low-latency data access is essential for real-time applications, such as autonomous vehicles, industrial automation, and smart retail systems [38].

Recent studies have sought to alleviate performance bottlenecks in distributed systems by mitigating straggler effects through the application of orthogonal gradient descent techniques in federated learning [39]. These approaches underscore the significance of optimizing asynchronous communication and resource utilization, which are equally critical in hybrid cache management for Apache Spark within edge computing environments. In such contexts, effective caching management requires a comprehensive assessment of cost–benefit trade-offs associated with data migration, ensuring that the performance gains outweigh the incurred overhead [40]. Furthermore, hybrid memory architectures integrating DRAM and NVM can provide an optimal balance between performance efficiency and cost-effectiveness, rendering them particularly suitable for resource-constrained edge computing systems [41].

2.6. Research Gaps and Contributions

Although substantial research has been conducted on cache management in Apache Spark, several critical research gaps persist. First, existing DAG-based strategies typically assume static data access patterns and insufficiently consider the costs associated with cache migration [20,42]. Second, hybrid caching approaches often neglect the substantial performance disparities between DRAM and NVM in real-world scenarios, resulting in suboptimal cache performance [11,24]. Third, machine learning-based methods demonstrate limitations with bursty and non-linear access patterns while demanding considerable computational resources [29,30]. Finally, research examining the interaction between cache management and garbage collection mechanisms remains limited, with most existing methods lacking adaptability to dynamic memory access behaviors [11,43,44].

The proposed Intelligent Hybrid Caching Management Framework (IHCMF) effectively addresses the identified limitations by incorporating DAG dependency analysis and GC behavior monitoring to dynamically optimize cache block placement between DRAM and NVM. This approach adapts intelligently to evolving data access patterns while accounting for cache migration costs and maintaining minimal computational overhead, delivering an efficient and cost-effective caching strategy for Apache Spark applications in edge computing environments. Moreover, the IHCMF enhances the performance of big data processing frameworks and offers a scalable and economical solution for edge computing systems, making it particularly suitable for real-time data processing in IoT applications.

3. Methods

3.1. Hybrid Cache Management System Design

With the ongoing advancement of NVM technology, its performance and cost advantages have become increasingly evident [38]. NVM offers high storage capacity, persistence, and relatively low cost, whereas DRAM provides low latency and high bandwidth [45]. As illustrated in Figure 1, the integration of DRAM and NVM into a hybrid caching system enables the complementary utilization of their respective strengths, thereby mitigating individual limitations and substantially improving cache performance and system efficiency.

• Design Objectives: NVM technology offers high storage capacity, persistence, and cost efficiency, complementing DRAM’s low latency and high bandwidth [45]. As illustrated in Figure 1, hybrid integration enables the utilization of their respective strengths, substantially improving cache performance and system efficiency.
• Key Components: The architecture design of the hybrid caching system comprises several key components. The Hybrid Cache Manager oversees cache operations. The DRAM Cache provides rapid access to frequently used data, while the NVM Cache stores less frequently accessed data and ensures persistence. The Memory Manager handles resource allocation, whereas the Data Migration Module performs dynamic data relocation based on access frequency patterns and system states. Additionally, the Consistency Module maintains data coherence across DRAM and NVM during migration and access operations, ensuring data correctness and reliability.
• Data Placement and Migration Strategies: The hybrid caching system adopts a two-phase strategy for data placement and migration. Initially, data blocks are allocated to DRAM or NVM based on access frequency patterns, with frequently accessed data prioritized for DRAM and less frequently accessed or persistent data stored in NVM. The dynamic mechanism subsequently adjusts data block locations between DRAM and NVM in response to changing access patterns and system conditions to sustain optimal performance. Furthermore, the system conducts a cost–benefit analysis that evaluates the trade-offs between migration costs, including transmission latency and energy consumption, and performance gains, such as reduced latency and increased cache hit rates. This ensures that data migration occurs only when the performance improvement outweighs the overhead.
• Cache Coherence: In a hybrid caching system that integrates DRAM and NVM, maintaining cache coherence is essential for ensuring data correctness and integrity during inter-media migration. The system implements multiple cache consistency mechanisms to achieve this objective. The Write-Back strategy initially updates data in cache and writes back to persistent storage upon eviction, thereby minimizing write amplification while requiring mechanisms for data loss prevention. Conversely, the Write-Through strategy updates both cache and persistent storage simultaneously, ensuring consistency despite increased write operations. Additionally, a Version Control mechanism monitors data block versions to identify and resolve inconsistencies through synchronization when necessary. Collectively, these mechanisms ensure reliable, consistent, and fault-tolerant data management in the hybrid caching system.

3.2. Intelligent Hybrid Caching Management Framework (IHCMF)

The IHCMF is engineered to intelligently manage data placement and migration between high-performance DRAM and high-capacity NVM within the Apache Spark environment. As illustrated in Figure 2, the framework functions as an integrated system consisting of several key components: the DAGAnalyzer, which parses Resilient Distributed Dataset (RDD) lineages and predicts data access patterns; the GCAnalyzer, which monitors Java Virtual Machine (JVM) GC behavior; the CacheOptimizer, which integrates analytical insights from both modules to determine optimal caching decisions; and the MemoryManager, which executes data placement and migration operations between DRAM and NVM.

The process begins with the analysis of the submitted Spark job DAG and real-time GC metrics. These analytical inputs are utilized within a cost–benefit optimization model to dynamically determine optimal cache block placement throughout job execution. This strategy ensures efficient utilization of hybrid memory resources while maintaining data consistency across the DRAM–NVM architecture through reliability mechanisms such as Write-Back and Version Control protocols. The framework aims to achieve substantial performance improvement by maximizing DRAM hit rates for high-value data blocks, subject to positive cost–benefit outcomes.

3.3. Cache Management Process

3.3.1. Cache Management Strategy Based on DAG Dependencies

DAG-dependent cache management strategies utilize the lineage structure of RDDs in Apache Spark to predict data access patterns and optimize caching decisions [10,46]. These strategies function through three primary mechanisms: (1) DAG Dependency Analysis, which identifies frequently accessed RDDs by examining computational dependencies within the DAG; (2) Cache Strategy Optimization, which implements eviction and retention policies, such as reference counting, to reduce redundant caching operations; and (3) Block Placement and Migration, which employs DAG-derived insights to guide the allocation and relocation of cache blocks across heterogeneous memory tiers, including DRAM and NVM. This methodology enhances cache efficiency by aligning storage decisions with expected data reuse patterns, thereby improving performance and resource utilization.

The primary DAG-based cache management strategies include: (1) MemTune, which dynamically optimizes memory allocation and data sharing by analyzing initial DAG dependencies, thereby enhancing cache hit rates for dynamic workloads [19]; (2) LRC, which identifies and evicts cache blocks with the lowest re-reference counts through DAG dependency analysis, effectively minimizing cache pollution in complex data dependency scenarios [9]; and (3) RDE, which computes data block reuse distances from RDD dependency sequences to eliminate redundant caching operations, thereby optimizing cache utilization in intermediate data-intensive frameworks [20].

3.3.2. Hybrid Memory Management Strategy Based on Garbage Collection

GC-aware hybrid memory management strategies combine JVM garbage collection behavior with Apache Spark’s memory system to optimize data placement across heterogeneous memory tiers, including DRAM and NVM [11,43]. These strategies reduce GC overhead by dynamically adjusting memory allocation based on runtime access patterns and GC metrics [25]. Current research primarily emphasizes three key aspects: (1) optimization of GC parameters (e.g., in G1 GC) to minimize pause times and enhance application throughput; (2) development of hybrid memory architectures that utilize the complementary properties of DRAM (low latency) and NVM (high capacity) for efficient cache placement; and (3) integration of memory management with GC mechanisms to dynamically optimize data placement according to usage frequency and GC pressure.

Key implementations include Panthera, which integrates static analysis and dynamic migration to allocate data to suitable memory tiers; G1 GC optimization, which refines generational heap ratios and concurrent thread parameters to enhance latency and throughput; and off-heap memory management, which circumvents JVM heap constraints to decrease GC frequency. Panthera specifically targets Spark-like data processing frameworks, whereas G1 tuning enhances performance in general JVM applications with large heaps, and off-heap approaches are particularly effective in latency-sensitive environments. Collectively, these approaches maintain high system performance while minimizing garbage collection overhead.

3.3.3. Cache Management Strategy

This paper combines a DAG-based cache management strategy with a GC-aware hybrid memory management approach, establishing an efficient cache management system (Figure 3). This architecture leverages both DAG dependency information and GC behavior to improve cache hit rates and minimize GC overhead, thereby significantly improving the performance of Apache Spark applications.

SparkContext: Serves as the entry point of a Spark application, responsible for initializing and configuring the Spark execution environment.

HybridCacheManager: A hybrid cache manager that coordinates the execution of various cache policies within the hybrid memory architecture.

DAGAnalyzer: A component that analyzes RDD dependencies to predict future reuse frequency patterns for optimized caching decisions.

GCAnalyzer: A module that analyzes garbage collection behavior and determines the GC frequency and impacts on RDDs.

CacheOptimizer: An optimization component that determines the optimal cache placement for RDDs based on integrated DAG dependency analysis and GC behavior insights.

MemoryManager: Responsible for managing memory allocation and deallocation processes across DRAM and NVM layers.

BlockManager: Handles the storage, retrieval, and tracking of data blocks within the hybrid caching framework.

DRAMCache: A DRAM-based caching layer designed to store frequently accessed data blocks for low-latency access.

NVMCache: An NVM-based caching system implemented to store infrequently accessed or persistent data blocks.

CachePlacementPolicy: A policy framework that determines the appropriate storage medium for each data block based on optimization results from the CacheOptimizer.

3.3.4. Class Architecture

The diagram depicts the class architecture of the IHCMF (Figure 4), illustrating essential components, including SparkContext, DAGAnalyzer, GCAnalyzer, HybridCacheManager, Memory, BlockTreeSet, and Block, and highlighting interrelationships and functional roles within the framework.

3.4. Optimization Strategy

3.4.1. Cache Block Placement Strategy Based on DAG

Strategy Overview: This approach leverages the DAG structure of RDD in Apache Spark to predict future data block access patterns and determine optimal cache block placement. By analyzing DAG dependencies, the system identifies data blocks that are likely to be frequently accessed in subsequent computational stages. These high-access-frequency blocks are then prioritized for placement in high-speed storage media, such as DRAM, thereby enhancing cache hit rates and overall system performance.

Optimization points:

(1)

Predict access patterns: Analyze the DAG to forecast data blocks expected to undergo frequent access in future computational operations.

(2)

Intelligent placement: Allocate data blocks with predicted high access frequencies to fast storage media (e.g., DRAM) to reduce latency and improve efficiency.

(3)

Fusion of LRU and LFU strategies:

LRU strategy: Establishes cache block priority based on recent access timing, ensuring that recently accessed blocks are retained in the cache.

LFU strategy: Determines cache block priority based on access frequency, evicting blocks with the lowest access counts first.

The implementation steps are as follows:

Step 1. Build the DAG: Examine the DAG generated by the Spark application to identify RDD dependencies. Extract all RDDs and their corresponding transformation operations to construct the complete dependency structure.

Step 2. Calculate the reuse distance and reuse frequency:

Reuse distance: Represents the interval between the current stage and the subsequent stage where the RDD will be utilized. A shorter reuse distance indicates a higher probability of future access.

Reuse frequency: Denotes the number of times an RDD is reused in subsequent stages. A higher reuse frequency indicates greater RDD importance.

For each RDD, both reuse distance ${\mathrm{D}}_{\mathrm{reuse}}$ and reuse frequency ${\mathrm{C}}_{\mathrm{reuse}}$ are calculated, and a cache priority score is assigned according to Equation (1):

$${\mathrm{c}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{e}}_{-}\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}\left(\mathrm{R}\mathrm{D}{\mathrm{D}}_{\mathrm{i}}\right)={\mathsf{\alpha }}_1\times {\mathrm{r}\mathrm{e}\mathrm{u}\mathrm{s}\mathrm{e}}_{-}\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\left(\mathrm{R}\mathrm{D}{\mathrm{D}}_{\mathrm{i}}\right)-{\mathsf{\beta }}_1\times {\mathrm{r}\mathrm{e}\mathrm{u}\mathrm{s}\mathrm{e}}_{-}\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\left(\mathrm{R}\mathrm{D}{\mathrm{D}}_{\mathrm{i}}\right)$$

(1)

Here, ${\mathsf{\alpha }}_1$ and ${\mathsf{\beta }}_1$ are weighting parameters that balance the influence of reuse frequency and reuse distance on cache priority determination.

To ensure equitable contribution of both metrics, reuse_count and reuse_distance are normalized to [0,1] using Min-Max scaling based on the min/max values within the current DAG. Reuse distance is normalized directly (i.e., shorter distances yield lower normalized values) to ensure consistency with the subtraction operation in Equation (1), where minimizing reuse distance contributes positively to cache priority.

Step 3. Cache location decision: Based on the calculated reuse distance and reuse frequency, the framework determines the optimal cache location for each RDD, either DRAM or Non-Volatile Memory (NVM), according to Equation (2):

$$\mathrm{C}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{L}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}=\{\begin{array}{ll}\mathrm{D}\mathrm{R}\mathrm{A}\mathrm{M}& \mathrm{if} {\mathrm{D}}_{\mathrm{reuse}}<{\mathrm{T}}_{\mathrm{distance}} \mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{C}}_{\mathrm{reuse}}>{\mathrm{T}}_{\mathrm{count}}\\ \mathrm{N}\mathrm{V}\mathrm{M}& \mathrm{otherwise}\end{array}$$

(2)

where ${\mathrm{T}}_{\mathrm{distance}}$ and ${\mathrm{T}}_{\mathrm{count}}$ represent the preset threshold values used to guide cache placement decisions.

Step 4. Integration of LRU and LFU strategies: Traverse the DAG from the current computational stage, recursively traversing all subsequent stages to record each RDD’s reference count and last access time. The cache block placement strategy is then determined by combining reuse frequency and last access time into a comprehensive weighted score that establishes cache block priority.

Rationale for Metric Selection:

(1)

Reuse distance: This metric is selected due to its direct correlation with the temporal proximity of future accesses. A smaller reuse distance indicates that a data block will likely be accessed sooner, making it an optimal candidate for placement in DRAM to minimize access latency.

(2)

Reuse frequency: This metric reflects the significance of a data block in overall computation. A higher reuse frequency demonstrates that the data block is repeatedly utilized across multiple stages, thereby justifying its placement in DRAM to improve cache hit rates and overall performance.

In conclusion, analysis of the DAG structure of RDDs enables accurate prediction of data block access patterns. By combining reuse distance and reuse frequency, the framework develops an intelligent cache block placement strategy that effectively improves cache hit rates and optimizes system performance. Additionally, the incorporation of LRU and LFU strategies further optimizes cache management efficiency by dynamically adapting to both temporal and frequency-based access behaviors.

3.4.2. Cache Management Based on GC Behavior

Strategy Overview: The cache management optimization strategy integrates JVM garbage collection behavior to reduce GC-induced cache invalidation. This approach analyzes the impact of GC operations on cache stability by computing two key metrics: GC frequency (${\mathrm{F}}_{\mathrm{g}\mathrm{c}})$ and GC duration ${(\mathrm{T}}_{\mathrm{g}\mathrm{c}})$. The GC frequency quantifies the rate of memory collection occurrences within a specified period, while GC duration represents operational time per collection event.

Optimization points: GC impact analysis identifies data blocks frequently recycled during GC events, enabling targeted mitigation of cache instability. Cache stability enhancement increases the persistence of long-surviving data blocks during GC processes, thereby minimizing cache replacement caused by GC operations.

Implementation steps:

Step 1. Monitoring Behavior: Utilize JVM monitoring tools (e.g., JVM Metrics) to collect GC frequency and duration data.

Frequency (${\mathrm{F}}_{\mathrm{g}\mathrm{c}}$): Quantifies garbage collection events within a defined time window.

Duration (${\mathrm{T}}_{\mathrm{g}\mathrm{c}}$): Measures time consumption for each GC event.

Step 2. Impact Analysis: Evaluate the influence of GC activities on cache performance by correlating GC events with cache block evictions. This step identifies frequently recycled data blocks and quantifies their impact on overall cache efficiency.

Step 3. Cache Strategy Optimization: Adjust cache management policies based on the collected frequency and duration metrics to minimize GC-induced cache invalidation. Consider NVM placement or non-caching for frequently recycled data blocks.

Cache stability enhancement prioritizes DRAM placement for data blocks with extended GC survival times. Frequently recycled blocks may be cached in NVM or excluded from caching to reduce the overhead associated with frequent GC operations.

Integration of GC Behavior Patterns with DAG Dependencies:

Integration process: The proposed framework integrates GC behavior metrics (${\mathrm{F}}_{\mathrm{g}\mathrm{c}}$ and ${\mathrm{T}}_{\mathrm{g}\mathrm{c}}$) with DAG-derived metrics such as reuse distance and reuse frequency to establish a comprehensive hybrid cache management strategy. The values obtained from DAG dependency analysis are dynamically adjusted based on the observed GC behavior, ensuring optimal data placement across heterogeneous memory tiers.

Handling ambiguous cases: In situations where DAG dependency metrics and GC behavior indicators yield conflicting placement recommendations, a cost–benefit analysis is employed to determine the optimal caching decision. Data blocks are allocated to DRAM when the predicted performance gains exceed GC overhead; otherwise, they are redirected to NVM to maintain memory efficiency and reduce cache invalidation risks.

3.4.3. Data Placement and Migration Strategy

Strategy Overview: The data placement and migration strategy represents a critical component in hybrid cache management systems. Its primary function is to dynamically allocate data blocks between DRAM and NVM according to real-time data access patterns and system conditions, initiating migrations only when performance gains justify the associated costs. By strategically determining storage locations, the system effectively exploits the low latency and high bandwidth of DRAM alongside the large capacity and persistence of NVM. This synergy enhances system performance, cache efficiency, and cost-effectiveness, ensuring optimal utilization of heterogeneous memory resources.

Data placement strategy

(1)

Initial placement based on access patterns: The system initially allocates data blocks according to their observed access frequencies. Frequently accessed data blocks are prioritized for placement in DRAM to minimize access latency and improve computational throughput. Conversely, data blocks with low access frequency and long-term retention requirements are allocated to NVM, leveraging its higher capacity and persistence.

(2)

Based on prediction: The system utilizes DAG dependency analysis and GC behavior analysis results to predict future data block access patterns. Data blocks predicted to exhibit high reuse frequency or short reuse distances are preferentially placed in DRAM to maximize cache hit rates. In contrast, blocks anticipated to have low access frequency are stored in NVM, ensuring balanced utilization of heterogeneous memory resources.

(3)

Cost–benefit analysis: Data block placement considers storage media cost-effectiveness. High-value data blocks (those associated with significant computing and recovery costs) are prioritized for DRAM placement to reduce recomputation overheads. Meanwhile, low-value data blocks are allocated to NVM, optimizing overall system cost-efficiency without compromising performance.

Data migration strategy

• Migration trigger conditions:

Access frequency change: Migration is initiated when a data block exhibits significant changes in access frequency. For instance, data blocks transitioning from low to high access frequency are migrated from NVM to DRAM to minimize access latency.

System status change: Migration occurs when system memory usage or computational load exceeds predefined thresholds. For example, if DRAM utilization surpasses a specified limit, lower-priority data blocks migrate to NVM to free up high-performance memory resources.

Pre-set time interval: The system conducts regular assessments of data block access patterns and system status, initiating migration when performance metrics suggest a potential improvement in cache efficiency.

2.

Migration cost assessment:

Migration cost calculation: Prior to data migration execution, the system computes the total migration costs, which include data transmission time overhead, energy consumption, and potential short-term system performance impacts. The migration cost can be calculated as

$${\mathrm{m}\mathrm{i}\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}_{-}\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{t}={\mathsf{\alpha }}_2\times {\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{e}\mathrm{r}}_{-}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}+{\mathsf{\beta }}_2\times {\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{r}\mathrm{g}\mathrm{y}}_{-}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{u}\mathrm{m}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$$

(3)

where ${\mathsf{\alpha }}_2$ and ${\mathsf{\beta }}_2$ are weighting parameters used to balance the influence between time consumption and energy consumption, respectively.

The system concurrently evaluates performance improvement quantified by metrics such as reduced data access latency and enhanced cache hit rates. Data migration proceeds only when the performance benefits exceed the associated migration cost, ensuring efficient resource utilization.

3.

Migration Decision Model:

Cost–benefit-based decision model: This model establishes a comprehensive framework that evaluates both migration costs and performance improvements to determine the necessity of data migration. The decision model can be expressed as follows:

$$\mathrm{d}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\{\begin{array}{ll}\mathrm{migrate},& {\mathrm{i}\mathrm{f} \mathrm{p}\mathrm{e}\mathrm{r}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}_{-}\mathrm{g}\mathrm{a}\mathrm{i}\mathrm{n}>{\mathrm{migration}}_{-}\cos \mathrm{t}\\ \mathrm{do} \mathrm{not} \mathrm{migrate},& \mathrm{otherwise}\end{array}$$

(4)

Here, performance gain denotes the improvement in performance resulting from the migration operation, while migration cost represents the cumulative expenses associated with the migration operation.

Implementation steps:

Step 1. Initialization:

Initialize the hybrid cache management system, including the cache manager, analyzer, and memory manager, and load the context information of the Spark application and construct the structure.

Step 2. Data placement:

Utilize DAG dependency and GC behavior analysis results to predict future data access patterns. Perform initial data placement on appropriate memory tiers (DRAM or NVM) based on cost–benefit evaluations.

Step 3. Real-time monitoring:

Continuously monitor data block access frequencies and system status parameters. Dynamically adjust storage locations in response to runtime variations.

Step 4. Migration decision:

Upon triggering migration conditions, calculate the migration cost and performance improvement. Apply the decision model to determine migration necessity.

Step 5. Migration operation:

Execute data migration operation, transferring data blocks between storage media. Update cache metadata and location tables to preserve data integrity and consistency across storage tiers.

3.4.4. Parameter Configuration

The weighting parameters ${\mathsf{\alpha }}_1$ and ${\mathsf{\beta }}_1$ in Equation (1) are empirically initialized to ${\mathsf{\alpha }}_1$ = 0.6 and ${\mathsf{\beta }}_1$ = 0.4, reflecting a higher priority on reuse frequency based on the observation that frequently accessed data blocks typically contribute more significantly to overall cache hit rates than those with merely short reuse distances. However, we acknowledge that these weights represent one feasible configuration among possible trade-offs between temporal locality (distance) and access popularity (frequency).

For thresholds ${\mathrm{T}}_{\mathrm{distance}}$ and ${\mathrm{T}}_{\mathrm{count}}$ in Equation (2), we employ a data-driven approach based on the statistical distribution of RDD characteristics within each workload. Specifically, ${\mathrm{T}}_{\mathrm{distance}}$ is dynamically set to the median reuse distance of all RDDs in the current DAG, ensuring that half of the RDDs with shorter distances are considered for DRAM placement. ${\mathrm{T}}_{\mathrm{count}}$ is set to the 75th percentile of the reuse frequency distribution, targeting the top 25% most frequently accessed RDDs for high-speed memory. These statistical thresholds ensure that DRAM capacity (typically 20–25% of total memory in our configuration) is allocated to the most valuable data blocks without manual tuning for each workload.

We acknowledge that the parameter selection for ${\mathsf{\alpha }}_1$ and ${\mathsf{\beta }}_1$ represents an empirical design choice rather than a globally optimal configuration. Our sensitivity analysis demonstrates that performance remains within 10% of the reported values when ${\mathsf{\alpha }}_1$ varies between 0.4 and 0.8, suggesting that the framework’s effectiveness is robust to moderate parameter variations. Future work will investigate adaptive parameter tuning mechanisms based on workload characteristics.

4. Experiment

Experimental Environment

The experiment was conducted on five high-performance servers. Each server was equipped with an Intel(R) Core(TM) i9-10940X CPU featuring 10 cores and 20 threads, with a base clock frequency of 3.30 GHz, and a maximum turbo frequency of 5.10 GHz, providing substantial computational power. Additionally, each server was configured with 256 GB of DDR4 memory operating at 3600 MHz and a Samsung 970 EVO 2 TB SATA SSD for high-speed data storage. The system environment was based on the Ubuntu 20.04 operating system, and all servers were interconnected via a full-gigabit Ethernet, ensuring high-speed and stable network communication.

Due to the absence of a physical NVM hardware platform, RamDisk technology was utilized within the Linux environment to simulate NVM behavior. RamDisk allocates a portion of the system’s memory as a storage partition. Given that its underlying storage medium is DRAM, its access latency closely resembles that of conventional memory. Consequently, by employing block-level storage control mechanisms, the system effectively simulates the read and write latency characteristics of NVM. This simulation approach facilitates the evaluation of the hybrid cache system’s performance under experimental conditions that approximate real NVM-based environments, even in the absence of a physical NVM device.

We explicitly acknowledge that the NVM simulation has important limitations. RamDisk resides in DRAM and exhibits DRAM-equivalent latencies (~100 ns), representing a “best-case” upper bound compared to real NVM which exhibits significantly higher and asymmetric access latencies (typically 3-4× slower than DRAM for reads and writes). While this captures NVM’s capacity benefits, it does not model performance asymmetry or write endurance.

The Spark cluster was configured and deployed in YARN-Client mode, comprising one driver and four Executors distributed across five servers. Each Executor was allocated between 4 GB and 24 GB of total memory (DRAM + NVM), depending on the specific experimental configuration. The cluster operated on Oracle JDK version 1.8.0_281 and Hadoop 2.7.2, with the Hadoop Distributed File System (HDFS) utilized for both input and output data storage during testing.

To minimize the impact of parameter variability on execution memory and experimental outcomes, a static memory management model was implemented. Apache Spark version 2.4.5 was employed, and the parameters spark.storage.memoryFraction and spark.executor.memory were systematically adjusted to establish different sizes of DRAM cache space allocations. This approach ensured consistent execution memory across all experiments, thereby enabling precise evaluation of the effects of different configurations on the performance of the hybrid cache system. Beyond job completion time and cache hit rates, we logged migration volume (total data transferred between DRAM and NVM tiers) via instrumentation in Spark’s BlockManager, and GC pause times via JVM GC logs (using -XX:+PrintGCDetails and -XX:+PrintGCTimeStamps for Oracle JDK 1.8). The overhead of IHCMF’s decision-making process (DAG analysis and cost–benefit calculation) was negligible compared to total execution time, as these operations are performed asynchronously with data access and DAG analysis is executed only once at job submission.

Baseline Selection Rationale: The evaluation compares IHCMF against two representative configurations: (1) Default (Spark’s native LRU-based caching), representing the industry standard without optimization; and (2) DRAM-only (pure DRAM with sufficient capacity), representing the theoretical upper bound of performance. While advanced DAG-aware methods (e.g., LRC [17], RDE [20]) and GC-aware frameworks (e.g., Panthera [25]) exist in the literature, they require substantial modifications to Spark’s core execution engine or JVM heap management, preventing fair, drop-in comparison within our plugin-based architecture. Nevertheless, we acknowledge that direct experimental comparison with these specialized methods would strengthen the evaluation.

This study utilizes SparkBench as the test set. SparkBench is a comprehensive benchmarking suite specifically designed to evaluate and compare the performance of various memory computing platforms. It provides a variety of test cases spanning multiple domains, including machine learning, graph computing, and SQL queries. This design facilitates a systematic analysis of the effects of configurations and optimization strategies on Spark application performance. As illustrated in

Table 1. SparkBench dataset.

No.	Test Case	Description
1	PageRank	The dataset employed for evaluating the performance of graph computing typically comprises web page links and the implementation of ranking algorithms.
2	K-Means	The dataset used for clustering algorithms is employed to test the performance of the machine learning library.
3	SQL Queries	A set of SQL queries used to test the performance of the Spark SQL module.
4	Streaming	Stream processing test cases, used to evaluate the performance of real-time data processing.

, the SparkBench dataset includes the PageRank workload for evaluating graph processing performance, the K-Means clustering algorithm dataset for assessing machine learning library performance, a collection of SQL queries for testing the Spark SQL module, and streaming workloads for measuring real-time data processing performance. Collectively, these test cases enable comprehensive performance evaluations of in-memory computing frameworks and support configuration optimization to enhance overall Spark application performance.

5. Result and Discussion

5.1. The Overall Performance of Different Caching Systems

To evaluate the performance benefits of the hybrid cache system, comprising a limited DRAM capacity and a larger NVM storage space, as well as the IHCMF proposed in this study, this subsection analyzes multiple experimental scenarios involving different storage devices and cache management strategies. The performance metrics obtained for PageRank, K-Means, SQL Queries, and Streaming are analyzed. The runtime performance of the four cache systems across these workloads is illustrated in Figure 5.

As illustrated in Figure 5, the IHCMF strategy proposed in this study demonstrates substantial performance improvements compared to the default 4G DRAM + 16G SSD hybrid cache configuration. The execution times for the IHCMF strategy for the Streaming, SQL Query, K-Means, and PageRank workloads were reduced by 35.24%, 39.13%, 48.24%, and 57.89%, respectively, under the 4G DRAM + 16G NVM hybrid cache configuration. The hybrid cache system implementing the IHCMF strategy shows an average performance enhancement of approximately 37.5% relative to the default Spark hybrid cache system.

The observed 37.5% improvement in job completion time is derived from two primary mechanisms: (1) improved DRAM hit rates (16.00% average increase, Section 5.2), which reduces access latency by keeping frequently reused data in high-speed memory; and (2) reduced GC pressure via intelligent placement of long-lived objects in NVM, which decreases the frequency and duration of garbage collection pauses by minimizing heap churn.

While direct energy metering was not available in the current experimental setup, the demonstrated performance improvements imply proportional energy savings for equivalent computational workloads under constant power draw. Specifically, the 37.5% reduction in execution time suggests a corresponding reduction in active energy consumption, though actual savings depend on specific hardware power profiles and idle-state characteristics that require physical NVM validation.

The IHCMF implementation resulted in notable performance gains across all test scenarios, particularly in computation-intensive workloads such as PageRank and K-Means. The performance advantages of IHCMF become more evident when processing these specific workloads. This finding demonstrates IHCMF’s ability to dynamically adapt to changes in data access patterns, optimize cache migration decisions, and maintain minimal computational overhead.

The IHCMF strategy achieved considerable performance improvements, particularly in compute-intensive workloads such as PageRank and K-Means, demonstrating its efficacy in optimizing resource utilization and minimizing data access latency. The average performance improvement of approximately 37.5% over the default configuration further highlights the robustness and adaptability of IHCMF across diverse workload scenarios. Prior studies report that DAG-aware methods (e.g., LRC [17], RDE [20]) improve cache hit rates through dependency analysis, while GC-aware (e.g., Panthera [25]) approaches reduce memory management overhead. IHCMF achieves a 37.5% improvement in job completion time by simultaneously optimizing both dimensions, suggesting potential benefits of integrated over single-aspect optimization, though controlled head-to-head comparison remains future work.

5.2. Cache Hit Performance of Different Cache Systems

This section presents a comprehensive analysis of cache hit rate performance across various caching strategies in four representative data processing workloads: PageRank, K-Means, SQL Queries, and Streaming. The experimental results, illustrated in Figure 6, are depicted as bar charts comparing three caching strategies: Default (baseline strategy), IHCMF (Intelligent Fusion Hybrid Cache Management Framework), and DRAM (utilizing all DRAM). The configurations for these caching systems are as follows:

Default: 4G DRAM +16G NVM, using the default cache strategy;

IHCMF: 4G DRAM + 16G NVM, using the IHCMF strategy;

DRAM: 20G DRAM, using the default cache strategy.

Error bars represent standard deviation from five independent runs. Statistical significance between configurations (Default vs. IHCMF, and IHCMF vs. Pure DRAM) was assessed using two-sample t-tests (* p < 0.05, ** p < 0.01, *** p < 0.001). Brackets indicate significant differences between compared groups.

The overall cache hit rate performance reveals that the IHCMF-based system consistently outperforms the hybrid cache system utilizing an identical default configuration across all four workloads. The improvements are 11.41% for PageRank, 14.42% for K-Means, 10.00% for SQL Queries, and 1.60% for Streaming, resulting in an average gain of 10.53%, which exceeds the improvement achieved by the pure DRAM cache system (4.75%). This enhancement is primarily attributed to IHCMF’s capability to identify and manage cache misses during the initial computation phase, where approximately 30% of cache misses persist even under full DRAM conditions. Additionally, the intelligent fusion hybrid cache management strategy effectively removes RDD cache blocks with low reuse potential from both DRAM and NVM, thereby improving cache utilization efficiency. Although the overall improvement in cache hit performance shows modest gains, the performance enhancements are mainly driven by optimized cache block placement and migration of the cache strategy, enabling access to frequently accessed data blocks within the DRAM cache.

DRAM/NVM Hit Performance: The cache hit rate analysis indicates that, compared to the hybrid cache system using the default strategy, the IHCMF-based hybrid cache system exhibits significant improvements in DRAM hit performance across all four workloads. Specifically, the performance enhancements include a 20.13% increase for PageRank, 19.66% for K-Means, 9.91% for SQL Queries, and 14.3% for Streaming, resulting in an average improvement of 16.00%. The difference in DRAM cache hit rates between the IHCMF-based system and the pure DRAM cache system is 13.32% for PageRank, 16.54% for K-Means, 12.17% for SQL Queries, and 13.21% for Streaming, yielding an average variation of 13.81%. These improvements can be attributed to the IHCMF cache strategy’s capability to maintain frequently accessed cache blocks within DRAM, thereby reducing data migration overhead and improving overall cache efficiency.

The experimental results indicate that the IHCMF strategy demonstrates significant performance advantages within a hybrid caching system, particularly regarding cache hit rate and cost-effectiveness. This strategy achieves an effective balance between system performance and resource utilization, thereby providing an optimized caching management solution for large-scale data processing frameworks. The results confirm that IHCMF enhances cache hit rate and overall computational performance while maintaining low operational costs, making it particularly advantageous in resource-constrained environments.

The IHCMF strategy markedly improves cache performance, achieving an average cache hit rate improvement of 10.53% over the default configuration, demonstrating its effectiveness in optimizing data access and overall system responsiveness. Furthermore, the 16.00% average increase in DRAM hit rates further confirms its cost-effectiveness and suitability for workloads characterized by high data reuse, confirming IHCMF’s capability to leverage memory resources efficiently in hybrid caching environments.

5.3. Performance of Mixed Cache with Different Proportions of DRAM and NVM

This section evaluates the scalability of the proposed IHCMF cache management strategy and examines how varying DRAM/NVM ratios affect the hybrid cache system’s performance. The analysis provides foundational insights for optimizing hybrid cache systems. The study compares the performance between the default cache strategy and the IHCMF strategy across different DRAM/NVM ratios under four representative workloads: PageRank, K-Means, SQL Queries, and Streaming, as shown in Figure 7.

Both caching strategies exhibit a similar trend: as the available in-memory cache capacity decreases, workload execution time correspondingly increases. The IHCMF strategy proposed in this paper consistently outperforms Spark’s default caching management strategy across all evaluated caching ratios. The magnitude of performance enhancement varies based on the cache ratio and the characteristics of individual workloads. For the PageRank workload, the optimal caching configuration for the IHCMF strategy comprises 8G DRAM with 16G NVM, representing a DRAM/NVM ratio of 1:2. Similarly, the K-Means, SQL Queries, and Streaming workloads achieve optimal performance at DRAM/NVM ratios of 1:2. Across various tested workloads and DRAM-NVM ratios, the IHCMF reuse cost-minimization strategy exhibits varying degrees of performance improvement compared to Spark’s default caching management strategy. Each workload demonstrates an optimal ratio that maximizes performance gains. These findings provide a quantitative basis for optimizing hybrid caching systems through strategic adjustment of DRAM/NVM ratios.

The experimental findings demonstrate that the IHCMF strategy achieves superior performance across various DRAM/NVM ratios. The strategy’s performance enhancement becomes particularly evident at lower DRAM/NVM ratios, indicating that performance levels can be maintained while substantially reducing system costs through strategic cache ratio adjustments. Furthermore, the results reveal that individual workloads exhibit distinct optimal DRAM/NVM ratios, each corresponding to peak cache system performance.

The IHCMF strategy consistently sustains performance improvements across different DRAM/NVM ratios, with multiple workloads achieving optimal performance at a 1:2 DRAM/NVM ratio, thereby demonstrating robust adaptability to varying resource constraints. This scalability characteristic makes it particularly suitable for edge computing environments, where limited resources and low-latency requirements are critical considerations.

The consistent performance improvements across varying DRAM/NVM ratios (Figure 7) demonstrate the framework’s ability to support cost-effective scaling of digital infrastructure. By enabling acceptable performance with reduced DRAM proportions (e.g., 1:2 ratio), organizations can deploy energy-efficient, high-capacity memory configurations that reduce both capital expenditure and operational energy consumption. This scalability is crucial for sustainable digital transformation in emerging economies and resource-constrained industrial environments, where maximizing computational output per unit of energy and hardware investment is essential for economic viability.

In conclusion, the IHCMF strategy presented in this paper demonstrates significant improvements in cache hit rates and overall system performance across various DRAM/NVM ratios, establishing its scalability and adaptability across different configurations. Additionally, the experimental findings provide valuable guidance for determining optimal DRAM/NVM ratios tailored to specific workload characteristics and resource limitations, providing a practical framework for hybrid cache system optimization.

6. Conclusions and Future Work

This study introduces an IHCMF for Apache Spark that integrates DAG dependency analysis with GC behavior to optimize data placement between DRAM and NVM. The framework’s dynamic, cost-aware optimization strategy achieves an optimal balance between performance, cost, and environmental impact, reducing job completion times by 37.5% while concurrently lowering energy consumption through improved resource utilization. By potentially leveraging NVM’s near-zero standby power and extended hardware lifespan (pending physical hardware validation), IHCMF provides a performance-optimized foundation for future green computing deployments, particularly in resource-constrained edge computing environments where energy efficiency and operational cost reduction are critical for sustainable IoT deployment.

While the IHCMF strategy demonstrates notable performance improvements, its effectiveness depends on specific operational conditions. The framework achieves significant gains in compute-intensive workloads, such as PageRank and K-Means, although its adaptability may be constrained in highly dynamic environments characterized by unpredictable or irregular data access patterns. The strategy’s dependence on DAG dependencies and GC behavior patterns may introduce biases, potentially diminishing performance in scenarios with irregular access patterns. Furthermore, the cost–benefit analysis employs a specific threshold for data migration that may not yield optimal results across all operational scenarios. These limitations highlight promising directions for future research, including the development of adaptive threshold mechanisms and enhanced learning-based models to improve the robustness and scalability of the framework.

We acknowledge that the current evaluation lacks direct energy consumption measurements. The experiments were conducted using RamDisk-based NVM simulation, which accurately models the latency characteristics of non-volatile memory but cannot capture the near-zero idle power and extended hardware lifespan benefits of physical NVM devices. Consequently, the energy efficiency gains reported in this study are inferred from performance improvements (reduced execution time and improved throughput-per-watt) rather than empirically measured power consumption data. While performance improvement directly implies reduced energy consumption for equivalent computational workloads, the potential for carbon reduction upon hardware deployment may vary based on specific hardware power profiles and workload characteristics.

Future work will advance several research directions with emphasis on sustainability validation and green optimization. First, to address the current limitation of inferred energy metrics, we will implement the framework on physical NVM hardware (Intel Optane or CXL) with integrated power monitoring (Intel RAPL). This hardware validation will enable empirical quantification of the potential for carbon reduction upon hardware deployment, moving beyond performance-based energy efficiency estimates. While hardware deployment provides ground-truth energy measurements, rigorous validation requires direct comparison with specialized state-of-the-art methods. Therefore, second, we will develop compatible benchmarking frameworks to enable head-to-head experimental comparison with DAG-aware (LRC, RDE) and GC-aware (Panthera) approaches, measuring not only job completion time but also GC pause time breakdown and migration overhead to validate the benefits of integrated optimization. Third, the integration of lightweight machine learning models will enhance access pattern prediction accuracy while maintaining minimal computational overhead. Fourth, the framework will be extended to support heterogeneous memory hierarchies with varying energy profiles to optimize for both performance-per-watt and cost-efficiency. Finally, lifecycle assessment (LCA) methodologies will be applied to evaluate the long-term environmental impact of widespread IHCMF adoption in industrial edge computing scenarios, contributing to the evidence base for green digital transformation policies.