M-DGNN: Accelerating Large-Scale Dynamic Graph Neural Network Training via PCIe-Interconnected Multiple Computational Storage Devices

Abstract

The explosive growth of temporal graph data has led to significant training overheads for Dynamic Graph Neural Networks (DGNNs), a bottleneck primarily driven by massive data movement between host processors and storage arrays across conventional PCIe I/O buses. While near-data processing with Computational Storage Devices (CSDs) can alleviate this bottleneck, a single CSD is inherently incapable of meeting the terabyte-scale capacity requirements and complex sequence modeling demands of modern large-scale DGNNs. Horizontal scaling with multi-CSD clusters over standard PCIe topologies presents a viable, cost-effective solution, yet our in-depth profiling identifies two critical architectural bottlenecks in naive multi-CSD architectures: host-bounced memory copies significantly compromise inter-device communication efficiency, and sparse graph sampling frequently exceeds the capacity of the tightly constrained local DRAM of CSDs, resulting in excessive flash I/O and performance degradation. To address these interconnected bottlenecks, we propose M-DGNN, a hardware–software co-designed architecture optimized for standard PCIe interconnects. First, M-DGNN orchestrates direct peer-to-peer (P2P) DMA dataflows for inter-CSD hidden state exchange, completely bypassing host operating system intervention and reducing the context-switching overhead. Second, we design a host-assisted caching strategy with a Host-Pinned Memory Extension (HPME) mechanism, which leverages host-pinned memory as an asynchronous DMA extension pool to shield resource-constrained CSDs from high-latency flash I/O during structural subgraph sampling. Extensive experimental evaluations across seven large-scale dynamic graph datasets demonstrate that M-DGNN delivers up to a 6.2× end-to-end speedup over the state-of-the-art DGNN systems. This work establishes an efficient, scalable near-data computing paradigm for large-scale DGNN training.

1. Introduction

The proliferation of continuous relational interactions across digital platforms ranging from Meta’s evolving social graphs [1] to dynamic user-content engagements on YouTube [2] has catalyzed a surge in temporal data mining [3]. To uncover evolving patterns within these structural datasets [4], Dynamic Graph Neural Networks (DGNNs) have emerged as the premier analytical paradigm. By coupling Graph Neural Networks (GNNs) [5] for spatial extraction with Recurrent Neural Networks (RNNs) [6] for temporal tracking, DGNNs serve as the backbone for critical applications such as pandemic forecasting and real-time recommendation engines [7,8].

Figure 1a illustrates the intrinsic execution logic of modern DGNNs. To capture deep temporal dependencies, frameworks universally adopt a sliding window strategy [9,10], where multiple consecutive snapshots (termed a frame) are processed simultaneously. This algorithmic design introduces a high degree of temporal locality, as overlapping snapshots are reused across consecutive windows. However, as dynamic graphs expand to the terabyte (TB) scale [11,12], they massively exceed the DRAM capacity of any single machine. Consequently, conventional systems inevitably fall victim to the bottleneck (Figure 1b). The strict separation between computation clusters and bulk storage creates a severe data movement crisis, congesting the PCIe bus and suffocating end-to-end performance [13].

Figure 1. Large-scale DGNN training suffers from severe data movement.

Exploiting near-data processing (NDP) via Computational Storage Devices (CSDs) [14,15] represents a practical approach to alleviate this I/O overhead. By executing operations at the storage edge, CSDs unlock internal flash bandwidths that easily exceed the throughput of external PCIe lanes [16,17]. While recent explorations [18,19] have successfully ported static GNN logic to individual CSDs, mapping the multi-snapshot temporal demands of DGNNs exposes critical architectural gaps. A single embedded drive lacks the requisite compute density to handle continuous sequence tracking. While horizontal scaling via a cluster of PCIe-interconnected CSDs appears to be an intuitive remedy, our comprehensive profiling reveals two major architectural bottlenecks that block efficient distributed execution:

• Host-Bounced Barrier in Communication: Distributed DGNN parallelization requires continuous horizontal exchange of sequential contexts (e.g., hidden states) between devices. Since these states are highly fragmented (typically small payloads), routing them through the conventional host-centric stack—$\mathrm{Source}\mathrm{CSD}\to \mathrm{Host}\mathrm{DRAM}\to \mathrm{Destination}\mathrm{CSD}$—imposes a disproportionate software-layer overhead. The compounding delays of context switching and constant interrupt servicing far exceed the actual data movement time, creating a communication barrier that negates scaling gains.
• Temporal Locality Waste in Storage: The sliding window mechanism of DGNNs relies on the intense reuse of snapshot data across windows. However, the rigidly constrained local DRAM of standard CSDs (typically ≤4 GB) [20] cannot retain these massive intermediate footprints. This leads to severe waste: unable to buffer reused features locally, the device is forced into a cycle of redundant evictions and re-fetches from the sluggish NAND flash for every sliding step, reconstructing the very I/O bottleneck that near-data computing was intended to bypass.

These observations motivate M-DGNN at a high level. We defer the detailed challenge analysis to Section 3, where the concrete communication and storage behaviors of distributed DGNN execution on multi-CSD systems are examined to derive the design rationale of the proposed architecture and mechanisms.

To overcome these interconnected bottlenecks without custom hardware silicon, we propose M-DGNN, a practical hardware–software co-designed architecture. M-DGNN targets system-level communication and storage workflows by leveraging existing PCIe P2P semantics and host-side memory resources. Accordingly, the contributions of this work are organized around the overall M-DGNN architecture and the two key mechanisms that enable its efficient execution, as follows:

(1)

A Scalable Multi-CSD Architecture for Large-Scale DGNN Training: As the overall execution foundation of M-DGNN, we propose a system-level hardware–software co-designed framework that natively maps the spatial–temporal execution workflow of DGNNs onto a scalable PCIe-interconnected multi-CSD cluster. This architecture relegates the host to a lightweight global orchestration role, while fully offloading the core dense tensor computations of DGNN training to the storage edge, laying a scalable foundation for near-data DGNN acceleration.

(2)

A Host-Bypassed Peer-to-Peer (P2P) Data Routing Strategy: As a key communication mechanism within M-DGNN, we design a direct P2P DMA dataflow for inter-CSD hidden state exchange to address the inter-device communication bottleneck. By completely bypassing host operating system intervention and redundant memory copies, this strategy collapses the conventional two-hop host-bounced data detour into a single-hop, zero-copy hardware transmission path.

(3)

A Host-Pinned Memory Extension (HPME) Mechanism: As a key storage optimization mechanism within M-DGNN, we develop the HPME mechanism to fully leverage the temporal locality inherent in sliding-window execution, which allocates page-locked memory in host DRAM to act as an asynchronous buffer pool. This mechanism retains the footprints of frequently reused graph features across consecutive sliding windows, effectively shielding CSDs from redundant flash I/O during GNN phase.

The rest of this paper is organized as follows. Section 2 introduces the background of Dynamic Graph Neural Networks and in-storage computing for large-scale DGNNs. Section 3 analyzes the core challenges and design motivations of distributed DGNN training on multi-CSD systems. Section 4 presents the detailed system design of M-DGNN, including its hardware architecture, software framework, inter-drive P2P payload routing, and HPME mechanism. Section 5 reports the extensive experimental results related to performance, the effectiveness of key mechanisms, scalability, and energy efficiency. Section 6 discusses the related work on GNN acceleration, in-storage computing, and DGNN systems. Finally, Section 7 concludes this paper and outlines future research directions.

2. Background

2.1. Dynamic Graph Neural Networks

Historically, Graph Neural Networks (GNNs) were primarily engineered to extract latent relational features from static topologies [7,21,22]. However, virtually all modern large-scale applications, ranging from financial tracking to social interactions, generate continuously evolving data streams [23,24,25,26]. To capture both structural connectivities and chronological trends, Dynamic Graph Neural Networks (DGNNs) have been developed to traverse time-varying networks [3,8,27]. To ingest this continuous data, DGNNs typically rely on discrete-time topology discretization [27,28], partitioning the dynamic timeline into sequential static cross-sections (snapshots).

From a systemic perspective, the end-to-end execution lifecycle of a DGNN pipeline spans multiple phases, among which the continuous batch training iterations phase is notoriously the most resource intensive. Because initialization and model checkpointing are relatively infrequent, the architectural design of M-DGNN focuses specifically on accelerating this core training loop.

From an algorithmic standpoint, analyzing sequential snapshots demands a dual-engine architecture at every algorithmic depth: a spatial extractor (typically a GNN) mapped to a temporal tracker (typically an RNN) [29,30]. Furthermore, to bound the computational complexity while maintaining a sufficiently wide temporal receptive field, DGNN frameworks universally apply a sliding window technique [9]. Instead of observing the entire history, a configured sequence of consecutive snapshots, defined as a frame, is batched into the model simultaneously. In industrial deployments, the frame length (window size) typically ranges from 4 to 16, striking an optimal balance between prediction accuracy and memory consumption.

The underlying dataflow of a standard DGNN is illustrated in Figure 1a. Assuming a strictly configured model layer s (equivalent to the specified frame scope), the advancing sliding window ingests snapshots spanning from interval $i+1$ to $i+s$. Within this active frame, the computation unfolds identically: initially, the GNN spatial module aggregates topological neighborhood features from the specific snapshot. Immediately after, the generated spatial embeddings are funneled into the corresponding RNN unit. Crucially, strictly ordered temporal dependencies emerge here—the intermediate hidden states generated by the RNN must be transmitted to the next adjacent GNN-RNN paired layer. Once the full sequence is iteratively digested, the final loss is computed to trigger the backward propagation phase. This tight coupling of spatial sampling and sequential dependency ensures robust temporal pattern recognition, but imposes significant synchronization and communication challenges for distributed hardware architectures.

2.2. In-Storage Computing for Large-Scale DGNNs

The explosion of dimensional attributes and node topologies has pushed traditional host-centric computing architectures to their breaking points. When terabyte-scale dynamic networks easily exceed the physical VRAM restrictions of modern GPUs, the system is fundamentally hamstrung by the memory wall. While several high-profile software-level frameworks have attempted to mitigate this—such as ESDG’s [29] workload-balanced partitioning, BLAD’s [31] snapshot group distribution algorithms, and DGC’s [32] chunk-based scheduling—they all share a critical physical flaw. Ultimately, they cannot evade the massive data thrashing occurring between main host memory and remote SSD storage across the narrow PCIe bus [17,33].

To decisively bypass the conventional I/O toll, In-Storage Computing (ISC) has gained immense traction. By integrating application-specific accelerators (FPGAs or dedicated ASICs) alongside NAND flash controllers, Computational Storage Devices (CSDs) seamlessly execute tasks near the data source [34,35,36]. This paradigm effectively filters unnecessary data movement [37], absorbing the I/O intensity within the SSD chassis. However, while ISC has proven highly effective for parallelizable database queries, the systematic exploration of complex, temporally coupled DGNN workloads over multi-CSD environments remains drastically underexplored. For the foundational static graph acceleration step, the existing literature provides a robust baseline.

BeaconGNN [17] currently represents the state-of-the-art methodology for porting static graph traversals into functional CSD logic. As depicted in Figure 2, BeaconGNN aggressively optimizes the heavily bottlenecked mini-batch generation lifecycle through hardware–software hybridization. It restructures the graph mapping format to unlock out-of-order neighbor sampling, heavily parallelizes near-data execution elements across multiple flash hierarchies (controller, channel, and die levels), and integrates a dedicated spatial accelerator strictly for executing MAC operations. Consequently, because individual DGNN snapshots fundamentally require static spatial parsing, M-DGNN inherits this mature near-data design profile as the elemental building block. The core challenge we address lies profoundly in how to orchestrate these isolated static processing engines to untangle the cross-device, highly dependent temporal demands of DGNNs without triggering severe PCIe communication breakdowns.

Figure 2. A mini-batch process for BeaconGNN.

3. Challenge Analysis and Design Motivation

Section 1 has outlined the high-level motivation and the core challenges addressed in this work. In this section, we further analyze these challenges under the target multi-CSD DGNN execution model, since the communication and storage bottlenecks must be examined in detail before introducing the architecture and mechanisms of M-DGNN.

Scaling dynamic graph learning to accommodate terabyte-sized datasets fundamentally exceeds the hardware limits of any standalone CSD. Bound by restricted thermal envelopes, minimal local DRAM, and modest embedded processing units, a single CSD is inherently underequipped to handle the continuous sequence tracking demands of large-scale DGNNs. Consequently, constructing a horizontally scaled, multi-CSD cluster interconnected via standard PCIe backplanes becomes a practical necessity.

Defining the Naive Multi-CSD Architecture (M-Base). To understand the intrinsic bottlenecks of distributed near-data DGNN processing and establish a representative evaluation baseline, we formalize a foundational multi-CSD execution paradigm termed M-Base, as illustrated in Figure 3. This architecture is constructed by combining two state-of-the-art technical routes: it adopts the mainstream snapshot-level distributed partitioning scheme from ESDG [29] to evenly map temporal snapshots across peer CSDs, and inherits the optimized in-storage GNN processing logic from BeaconGNN [17] for single-snapshot spatial feature extraction. This represents the most straightforward and universally logical multi-CSD implementation for near-data DGNN acceleration.

Figure 3. The execution dataflow of the naive multi-CSD architecture (M-Base). While the spatial feature extraction (GNN) achieves high parallelism across devices, the sequential dependency tracking (RNN) dictates a strict serial execution, heavily bottlenecked by host-bounced memory copies of hidden states.

In this architecture, consecutive temporal snapshots (e.g., $S_1,S_2,\dots ,S_n$) are sequentially partitioned and distributed across an array of N peer CSDs. The execution paradigm is strictly decoupled into two distinct phases for both forward and backward propagation:

• Parallel Spatial Extraction:The GNN phase is highly parallelized. Each CSD autonomously samples local structural topologies and generates spatial features from its assigned snapshots, efficiently keeping the physical NAND flash I/O localized.
• Serialized Sequential Tracking: Conversely, capturing sequential dependencies via RNN dictates strict serial execution. To accurately maintain temporal continuity, the generated hidden states (forward propagation) and gradient tensors (backward propagation) must be fed forward between adjacent snapshot blocks across consecutive layers.

However, transitioning from an isolated near-data offloading paradigm to this synchronized, multi-drive collaborative ecosystem introduces substantial system-level inefficiencies. As illustrated by the “Memory copy” path in Figure 3, the standard M-Base architecture suffers from two tightly coupled bottlenecks across the communication and storage planes, which inherently stem from its two-phase execution design:

(1)

Host-Bounced Routing Overhead in the Communication: This bottleneck is rooted in the strict temporal dependencies of RNNs. While the GNN spatial phase is highly parallelizable across N devices, capturing temporal evolutions requires the continuous “forward passing” of hidden states and gradient tensors horizontally across CSDs. Under the conventional host-centric routing paradigm of M-Base, these spatio-temporal tensors are forced into an expensive two-hop detour: $\mathrm{Source}\mathrm{CSD}\to \mathrm{Host}\mathrm{DRAM}\to \mathrm{Destination}\mathrm{CSD}$. Because hidden states are typically small (a few hundred bytes per node) but extremely frequent, the cumulative overhead of context switching far exceeds the actual data movement time. This communication effectively negates the throughput gains from horizontal scaling, leaving high-speed storage-edge accelerators frequently idling while waiting for the next temporal state. This fundamental bottleneck motivates our first core design: a P2P data routing strategy that collapses the two-hop detour into a single-hop, zero-copy hardware path.

(2)

CSD Memory Constraints and the Flash I/O Wall in the Storage: This bottleneck is intrinsically derived from the Parallel Spatial Extraction phase. The GNN spatial aggregation phase generates massive, highly sparse randomized read requests. While buffering intermediate topological representations is a standard practice to shield the high-latency NAND flash circuitry [29,33], this strategy faces a severe physical mismatch on standard CSDs. Bounded by thermal and economic constraints, commercial CSDs typically feature minimal local DRAM capacities (frequently ≤4 GB) [20]. The sliding window execution of DGNNs is characterized by high temporal reuse, where a single snapshot is repeatedly sampled and processed across consecutive overlapping frames. In an ideal scenario, intermediate features should be cached to exploit this temporal locality. However, since the footprint of large-scale graph features easily expands to tens of gigabytes, the constrained local DRAM of CSDs cannot retain snapshots across window shifts. This results in Temporal Locality Waste; unable to buffer reused features locally, the device is forced into a cycle of redundant evictions and re-fetches from the sluggish NAND flash array for every sliding step. Consequently, the random access latency of NAND flash (orders of magnitude higher than DRAM) creates a severe I/O bottleneck, reconstructing the bottleneck that near-data computing was meant to bypass.

Crucially, from a systems perspective, these two bottlenecks form a compounding negative impact that further degrades end-to-end performance. The host-bounced overhead in the RNN phase elongates the overall pipeline latency, extending the residence time of intermediate features inside the limited CSD memory. In turn, the frequent flash I/O stalls in the GNN phase leave CSD compute units underutilized, amplifying the relative performance penalty of the communication overhead. Mitigating these intertwined bottlenecks requires practical hardware–software co-design. Rather than proposing modifications to localized device hardware, our fundamental motivation is to formally explore how standard CSD arrays and host PCIe infrastructure can be orchestrated to break this cycle. By explicitly utilizing native PCIe P2P semantics and repurposing host DRAM structures, we aim to demonstrate that an efficient, highly utilized distributed DGNN execution pipeline can be cultivated entirely over widely deployed standard server interconnections. This methodology lays the groundwork for the M-DGNN architecture detailed in the following section.

4. System Design of M-DGNN

4.1. Overview of M-DGNN

M-DGNN is designed to address the communication and storage bottlenecks in distributed DGNN training on multi-CSD systems. At a high level, M-DGNN provides an overall multi-CSD execution architecture, together with two key mechanisms that improve its efficiency: (1) an inter-drive P2P payload routing mechanism that enables direct hidden-state transfer without host-bounced communication, and (2) a Host-Pinned Memory Extension (HPME) mechanism that improves the reuse of temporally local data during sliding-window execution. In this design, the host is responsible for lightweight orchestration and metadata management, while the CSDs execute graph processing tasks and exchange intermediate data through PCIe P2P communication. Following this overview, we present the detailed design of the hardware architecture, software framework, inter-drive P2P payload routing, and HPME mechanism.

4.2. Hardware Architecture

Figure 4 delineates the overarching hardware abstraction of M-DGNN. Emphasizing immediate deployment viability, the proposed architecture avoids fabricating specialized interconnects; instead, it natively maps onto a standard, commercially available server PCIe topology. The hardware ecosystem is functionally decoupled into three cohesive tiers:

Figure 4. Hardware architecture.

• The Host Orchestration Tier: Comprising the host CPU and host DRAM, the host operates as the global control plane. It interfaces with the downstream interconnect via the PCIe Root Complex (RC). Crucially, to mitigate the localized storage limits of peripheral devices (the flash I/O bottleneck identified in Section 3), the host DRAM is partially repurposed as an expanded memory pool to buffer intermediate graph features that spill over from the devices.
• The PCIe Interconnect Fabric: Serving as the structural backbone, this tier is modeled around a standard multi-port PCIe Switch. Beyond facilitating conventional tree-based host-to-device communications, the PCIe Switch serves a more critical function in M-DGNN: it physically enables native P2P transactions. This allows data payloads to be routed directly between multiple PCIe Endpoints (EPs) entirely independent of the RC, thereby fundamentally eliminating host memory bandwidth saturation.
• Compute Nodes (CSDs): Functioning as the primary processing engines, an array of CSDs is attached as independent PCIe EPs. Zooming into the micro-architecture of a single CSD (Figure 4, right), each device integrates a specialized P2P DMA Engine and an NVMe Controller to manage asynchronous I/O and cross-device routing. For computation, an embedded Accelerator is coupled with local ARM cores and cache hierarchies to execute General Matrix Multiplication (GEMM) workloads. The Local DRAM serves as both a localized execution workspace and a staging buffer (exposing mapped I/O regions via PCIe Base Address Registers). Finally, the backend flash interface utilizes multi-channel parallelism to achieve the high-throughput retrieval of dynamic graph structures.

4.3. Software Framework

To fully actualize the theoretical throughput of the distributed hardware topology, M-DGNN introduces a stratified software stack, as illustrated in Figure 5. Adhering to a strict Control Data Decoupling philosophy, this framework balances centralized host orchestration with autonomous peripheral execution, mitigating software stack overhead:

Figure 5. Software framework.

• The Global Orchestration Plane (Host-side): This tier assumes a purely supervisory role, exclusively handling lightweight control signals. The Mini-batch Pipeline Controller (MPC) orchestrates the global workload by dispatching non-blocking NVMe commands (categorized into GNN spatial parsing and RNN temporal tracking instructions) to the target CSDs. Simultaneously, the Sliding Window Tuner dynamically parses the model layer metadata (e.g., matching the coupled frame ID with the target model layer address) to resolve the temporal mapping of incoming frames. To facilitate horizontal scaling, the Address Management module registers the memory Base Address Registers (BARs) of peer CSDs, establishing transparent cross-device spatial visibility for subsequent routing.
• The Data Communication Plane: Operating asynchronously beneath the application layer, this plane exclusively governs heavy payload transfers via DMA, completely bypassing the CPU intervention. For inter-CSD sequence continuity, the P2P Comm. Engine leverages native PCIe P2P semantics to directly stream intermediate RNN parameters (e.g., hidden states) into the peer’s P2P I/O Buffer. Furthermore, this plane maintains the Host Feature Cache in the main system memory. Through standard Host-CSD DMA, devices can asynchronously pre-fetch vertex features or offload aggregated subgraph embeddings, effectively utilizing the host memory as a capacity extension.
• The Execution Plane (CSD-side): Embedded deeply within the localized device logic, this tier is responsible for the heavy lifting. Upon receiving standard NVMe directives from the host, the localized controller autonomously schedules intensive neighbor sampling from the graph dataset (e.g., historical snapshots $S_1,\dots ,S_{N+1}$). The sampled topologies are staged in the Local DRAM. The embedded hardware accelerator then seamlessly executes the tightly coupled GNN-RNN tensor computations. The generated outputs (such as extracted vertex features or updated hidden states) are subsequently pushed into the designated CSD I/O Region, ready to be harvested by the Data Communication Plane for the next temporal frame.

4.4. Inter-Drive Payload Routing

Eliminating the inter-CSD communication bottleneck (identified as Challenge 1) demands a fundamental restructuring of the RNN sequence dataflow. As depicted in Figure 6a, conventional topologies force sequential hidden states through a bidirectional, host-centric trajectory (MemCpy). This indirect routing incurs severe latency penalties stemming from host context-switching and redundant DMA setup cycles, which collectively choke the overall pipeline.

Figure 6. The P2P routing fabric and lightweight synchronization mechanism bypassing host intervention.

To optimize this over standard, non-coherent PCIe topologies, M-DGNN engineers a pragmatic Data-Control Decoupled P2P Fabric (Figure 6b). The core architectural philosophy is to delegate bulk tensor payloads exclusively to the internal PCIe switching matrix, while ensuring that synchronization tokens and control logic remain strictly centralized at the Host. This mechanism operates across two distinct planes:

• Data Plane via Zero-Copy P2P DMA: During the initialization phase, the Address Management module maps a contiguous I/O Region within each CSD’s Local Memory using PCIe Base Address Registers (BARs). During execution, when a source CSD (e.g., CSD 3) generates hidden states required by an adjacent destination CSD (e.g., CSD 4), it bypasses the host memory stack entirely. The local DMA engine of the source CSD issues a native P2P write transaction. The intermediate PCIe switch acts purely as a physical router, streaming the tensor payload directly into the mapped I/O Region of the destination CSD, achieving true zero-copy data transfer.
• Control Plane via Host Metadata Manager: While massive data payloads traverse directly between endpoints, the execution logic and dependency synchronization are elegantly maintained by the Host Metadata Manager. As illustrated in Figure 6b, control signaling is kept deliberately lightweight. Upon completing a P2P DMA transfer, the source CSD merely dispatches a tiny completion interrupt (status flag) to the Host. The Host rapidly updates its global routing metadata and immediately “rings the doorbell” for the destination CSD, signaling that its input buffers are ready and it can proceed with downstream RNN computations.

By centralizing ultra-lightweight control flows while offloading heavy tensor routing to the existing PCIe hardware fabric, M-DGNN successfully collapses communication latencies without necessitating any bespoke hardware modifications.

4.5. Host-Pinned Memory Extension (HPME)

As illustrated in Figure 7a, relying exclusively on the highly constrained onboard memory of CSDs invariably leads to systemic throughput collapse. Processing massive intermediate embeddings rapidly triggers aggressive cache thrashing. This forces the embedded runtime to evict the required data and repeatedly fetch raw structural topologies from the sluggish backend NAND flash, effectively starving the high-performance DNN Accelerator. To mitigate the flash I/O and break the vicious cycle of memory constraints (Challenge 2), M-DGNN implements the HPME mechanism, as shown in Figure 7b. Rather than treating host memory as a detached, distant resource, M-DGNN seamlessly provisions it as an expansive extension buffer. The HPME operates through three orchestrated stages:

Figure 7. The M-DGNN caching architecture leveraging host-side buffers to mask flash I/O.

(1)

Asynchronous Feature Offloading: Prior to training, a dedicated Buffer Pool is allocated within the Host DRAM using page-locked (pinned) memory, ensuring it is physically contiguous and safe for direct DMA access. During the forward pass of a newly ingested temporal snapshot, the CSD performs initial neighbor sampling directly from the flash. Instead of discarding these aggressively sampled intermediate results (e.g., Target Node IDs mapped to Aggregated Features), the CSD asynchronously streams them across the PCIe bus into the host Buffer Pool via DMA.

(2)

Temporal Reuse via Sliding Windows: This design intrinsically exploits the temporal locality of DGNNs. Because the sliding window mechanism dictates that identical temporal snapshots are repeatedly processed across consecutive overlapping frames, subsequent training iterations frequently demand the exact same topological data. For these subsequent requests, the CSD bypasses the backend flash array entirely, natively querying the Buffer Pool in the Host DRAM. Fetching cached features via standard PCIe DMA delivers bandwidths orders of magnitude higher than generating highly randomized flash read sequences.

(3)

Computation-I/O Overlapping (Fetch Overlap): To perfectly hide the PCIe transmission latency, M-DGNN introduces a double-buffering prefetch strategy. As the execution unfolds, while the CSD’s DNN Accelerator is actively traversing the currently sampled subgraphs, the localized DMA concurrently pre-fetches the required topological features for the next layer or batch from the Host DRAM. As vividly depicted by the “Saved Time” gap in Figure 7b, this computation-I/O overlapping deeply masks the memory access tail latency, ensuring the embedded computational units remain constantly saturated.

5. Experimental Evaluation

In this section, we evaluate M-DGNN from multiple perspectives. We first evaluate its end-to-end performance advantage over host-centric and conventional multi-CSD baselines. We then evaluate the effectiveness of the inter-drive P2P routing mechanism in reducing communication overhead, followed by an analysis of the HPME mechanism in improving temporal data reuse and reducing flash I/O stalls. Finally, we evaluate the scalability of M-DGNN with increasing numbers of CSDs and analyze its energy efficiency benefits.

For clarity, the compared systems include both existing host-centric DGNN frameworks and staged near-data baselines. ESDG, BLAD, and DGC represent prior host-centric frameworks. M-Base denotes a naive multi-CSD near-data baseline, M-P2P adds only the proposed inter-drive P2P routing mechanism, and M-DGNN integrates both P2P routing and HPME. This setup allows us to isolate the contribution of each major component of M-DGNN.

5.1. Experiment Setup

Evaluation Platforms and Baselines. To rigorously evaluate the system-level efficiency and scalability of M-DGNN, we configure six distinct DGNN acceleration pipelines across two fundamentally different hardware computing paradigms. To isolate and validate the pure architectural benefits of our decoupled data–plane design, we enforce an intentionally conservative hardware alignment based on an equal number of processing nodes (N). The fundamental difference lies in their compute capacity (as detailed in Table 1): the host-centric platform is equipped with heavy-duty server-grade discrete accelerators (TPUs), possessing significantly higher compute density than the power-constrained, embedded accelerators inside the CSD platform.

Table 1. Simulation parameters for M-DGNN.

Component	Parameters
Controller and Host I/F	4 ARM Cortex-A9 Cores, Non-Volatile Memory Express (NVMe) 2.0, PCIe Gen 6.0 × 4
Local CSD DRAM	DDR4-3200, 25.6 GB/s, 4 GB
Flash Memory	16 Channels, 4 Packages, 2 Dies, 2 Planes
	1024 Blocks, 1024 Pages, Page size = 4 KB
	NV-DDR3 800 MT/s, 8-bit Channel Bus width
SSD-level Accelerator	$64\times 64$ systolic array, 64-width vector unit
(Constrained Embedded)	6 MB shared scratchpad, precision FP16
	Area 27.2 mm2, Frequency 800 MHz
Discrete Accelerator	$256\times 256$ systolic array, 256-width vector unit
(Baseline Heavy Host)	64 MB shared scratchpad, precision FP16
	Frequency 1 GHz
Timing and Latency	Raw flash page read latency = 12 μs
	Host-side software stack latency = 10 μs
PCIe Switch Topography	PCIe Gen 6.0 protocol with native P2P routing enabled
	256-lane switch configured, 64 GT/s per lane
	4 MB port buffer
Energy and Power	Operation voltage = 3.3 V
	$I_{\mathrm{READ}}$, $I_{\mathrm{PROG}}$, $I_{\mathrm{ERASE}}$ = 32 mA
	$I_{\mathrm{BUSIDLE}}$ = 5 mA, $I_{\mathrm{STBY}}$ = 12 μA
	PCIe PHY = 8.5 pJ/bit, Host memory R/W = 48 pJ/bit
	Flash Sampler Logic = 5.23 mW

(1) Host-Centric Platform (Executed on N SSDs + N Server TPUs):

This category represents conventional host-driven paradigms. Despite utilizing powerful host accelerators, they suffer from standard PCIe host–device data movement bottlenecks.

• ESDG [29]: A mainstream snapshot-partitioned system that pulls dynamic graphs from SSDs into host memory via conventional block I/O.
• BLAD [31]: A state-of-the-art frame-partitioned distributed pipeline optimized for temporal batching, but still penalized by PCIe traversal overheads.
• DGC [32]: A cutting-edge, graph chunk-partitioned system designed for spatial locality, yet physically constrained by the host-centric memory hierarchy.

(2) In-Storage Computing Platform (executed exclusively on N CSDs):

This category represents the near-data processing evolution, configured as an ablation study to validate our specific hardware–software co-designs.

• M-Base (Naive Multi-CSD): The foundational near-data baseline (introduced in Section 3). It offloads subgraph sampling/computation to N CSDs but strictly relies on host-bounced memory copies for inter-layer RNN sequence communication.
• M-P2P (Addressing Comm. Bottleneck): An architectural extension of M-Base enabling the Control Data Decoupled P2P Routing Fabric. It utilizes zero-copy PCIe BAR mappings and lightweight doorbell signaling to bypass host.
• M-DGNN (The Complete System): Our definitive proposed architecture. It integrates M-P2P with the HPME mechanism, leveraging asynchronous DMA tunneling to utilize host DRAM as buffer.

Microarchitectural Performance Modeling. To accurately capture the complex hardware–software interactions, we utilized ASTRA-SIM [38,39], a highly validated cycle-level distributed system simulation framework. Standard simulators often lack native support for advanced near-data PCIe semantics. Our current design assumes a PCIe-connected multi-CSD platform in which all CSD endpoints are attached to a single non-blocking switch and can communicate through 1-hop P2P TLP forwarding. We use this topology as a target design point to isolate the architectural benefit of direct inter-CSD communication, rather than to imply that all commodity PCIe fabrics provide identical connectivity or contention characteristics. In more constrained PCIe deployments, such as oversubscribed or multi-level switch fabrics, the latency and queuing overhead of inter-CSD transfers may increase. Likewise, if full P2P reachability is unavailable for some endpoint pairs, part of the communication may need to fall back to host-mediated routing. Under such conditions, the quantitative speedup of M-DGNN may be reduced, although its qualitative advantage remains rooted in reducing repeated flash accesses and unnecessary host-side data movement. We extended the simulation backend to capture three critical structural behaviors:

(i)

PCIe P2P Routing Canvas: Memory traces were strictly recalibrated to intercept direct Base Address Register (BAR)-mapped transactions. This captures the hardware-level delays of PCIe switch routing while strictly excluding context-switches.

(ii)

Non-Blocking Switch Fabric: We leveraged the analytical network backend to simulate a non-blocking PCIe Gen 6.0 topological switch, modeling native 1-hop Transaction Layer Packet (TLP) switching capabilities between all CSD endpoints.

(iii)

HPME Cache Interactions: We integrated accurate models for PCIe bus contention, iATU (Input/Output Address Translation Unit) mapping cycles, and the exact DMA burst transfer latency associated with asynchronous reads/writes to the pinned Host DRAM pools (simulating the HPME mechanism).

The localized embedded computation within the CSDs is parameterized based on BeaconGNN-style near-storage accelerator logic [17], including implementation-oriented constraints on accelerator footprint, operating frequency, and local scratchpad capacity under SSD form-factor limits. For the persistent storage tier, M-DGNN timing parameters align with commercial ultra-low-latency NAND technology (e.g., Samsung Z-NAND [40,41]). We further clarify that the PCIe Gen6 × 4 configuration for each CSD is used as a forward-looking target design point for a PCIe-connected multi-CSD platform. It is not intended to imply that identical deployments are already standard across all commodity servers today. We adopt this configuration to study whether the proposed host-bypass communication mechanisms remain beneficial under emerging high-bandwidth PCIe fabrics. Similarly, the switch port buffer size is used as a system-level modeling parameter to capture transaction queuing and contention behavior during inter-CSD P2P transfers. To enforce a fair and rigorous comparison, the embedded CSD accelerator footprints are tightly bound by standardized M.2/U.2 SSD thermal and area constraints [42], whereas the baseline discrete host accelerators mirror the massive specifications of Google TPU v4 [43]. Power modeling combines McPAT [44] and DRAMPower [45] for control/storage logic, and CACTI-7 [46] for embedded matrix SRAM units. We note that the reported performance numbers correspond to the above design point and should be interpreted accordingly. Our goal is to evaluate the architectural effect of combining storage-side execution, PCIe P2P communication, and HPME-assisted buffering under a consistent hardware configuration, rather than to claim that the exact speedup value is invariant across all possible system parameter choices. Because the main benefit of M-DGNN comes from reducing host-mediated communication and repeated flash accesses, its qualitative advantage is rooted in the system organization itself rather than in any single timing parameter. We also note that the practical deployment of PCIe P2P communication depends on platform-level support for peer routing and peer memory exposure, including appropriate address translation and access control configuration (e.g., IOMMU-related settings, outbound address translation where applicable, BAR-space visibility, and PCIe ACS/ARI policies). Our evaluation models a target platform where such support is provisioned. Finally, our current conclusions are bounded to the evaluated topology and scale. We do not claim that switch buffer contention and control plane coordination overhead remain negligible for arbitrarily larger deployments; as the number of CSDs increases beyond the evaluated 16-device setting, these factors may become more pronounced and deserve separate analysis.

Workload Topology and Workload Selection. To rigorously stress test the execution limits and scalability of M-DGNN under extreme storage and network bottlenecks, we purposefully deploy a comprehensive suite encompassing two structural models and seven massive-scale datasets. For the spatial topological extraction (the GNN phase), we utilize GraphSage [22], an industry-standard inductive representation learner known for its intensive random walk memory access patterns. For the temporal dependency modeling (the RNN phase), Long Short-Term Memory (LSTM) [47,48] serves as the primary backbone. Notably, while we evaluate using LSTM, the foundational P2P communication mechanism introduced in M-DGNN operates at the hardware data-flow level and remains entirely architecturally agnostic to the specific recurrent formulation.

To invoke realistic flash I/O thrashing and fully saturate the PCIe P2P fabric, we scale our evaluation across dynamic graphs containing up to tens of billions of edges (summarized in Table 2). We incorporate two authentic real-world temporal benchmarks, Flickr (FR) and YouTube (YE), sourced from the ESDG framework [29,49]. To deeply evaluate the HPME mechanism under extreme-scale regimes that dwarf localized CSD DRAM capacities, we synthesized five massive temporal topologies (Reddit, Amazon, Movielens, OGBN, and PPI) leveraging static base structures from prior in-storage baseline evaluations. Temporal dynamism was procedurally injected into these giant graphs via a meticulously controlled random edge deletion heuristic, ensuring the inter-snapshot structural evolution strictly adheres to the empirical density shifts observed in the real-world Flickr dataset.

Table 2. Massive-Scale Dynamic graph workloads for evaluation. (N: vertices, E: edges, S: temporal snapshots, and D: dense feature dimension across all temporal states).

Dataset	N	E	S	D
Flickr (FR)	2.3 M	33.1 M	132	32
Youtube (YE)	3.2 M	602.0 K	198	32
Reddit (RD)	37.3 M	53.9 B	132	602
Amazon (AM)	265.9 M	79.8 B	132	200
Movielens (ML)	22.2 M	59.2 B	132	30
OGBN (OG)	179.1 M	5.0 B	132	32
PPI (PP)	9.1 M	8.8 B	132	256

Hyperparameter Boundaries and Precision Mapping: To ensure strictly fair comparisons against software-based baselines, simulation invariants are parameterized as follows: minibatch workload size is clamped at 128 targets (consistent with BeaconGNN bounding), temporal hidden state granularity is mapped to a dimension of six (structurally aligned with BLAD), and topological node identifiers are encoded as INT-32 scalars. Dense feature vectors are precision-scaled to FP-16 to maximize vector unit throughput within the constrained CSD SRAM. All convergent evaluations capture the steady-state throughput over a minimum of 5 continuous training epochs.

5.2. End-to-End Performance

This experiment evaluates the overall end-to-end training performance benefit of M-DGNN compared with representative host-centric frameworks and multi-CSD baselines.

Figure 8 illustrates the normalized end-to-end training performance across all evaluated architectures using four CSDs. To study how temporal dependency depth affects system performance, we evaluate models with 4, 16, and 32 layers. All throughput metrics are normalized against M-Base. Observing the traditional host-based software frameworks (ESDG, BLAD, and DGC), their relative performance noticeably trails that of the near-data paradigms. For instance, on the AM dataset with four stacked layers, BLAD achieves only about 0.46× the throughput of M-Base. This persistent bottleneck indicates that relying on host-to-device data transfers causes massive intermediate graph structures to repeatedly congest the PCIe bus, leaving discrete accelerators fundamentally underfed. By shifting to the near-data computational storage paradigm, our integrated design effectively mitigates these structural inefficiencies. As depicted in Figure 8a, by combining direct inter-CSD communication (M-P2P) with the HPME caching mechanism, the architecture delivers its best end-to-end performance under the four-layer configuration. M-DGNN attains a peak end-to-end speedup of about 6.2× over BLAD on the AM dataset and consistently provides up to a 2.85× improvement over M-Base. A detailed micro-architectural breakdown of P2P routing and caching is provided in Section 5.3.

Figure 8. Normalized end-to-end training speedup across different model layers using four CSDs. Results are evaluated under (a) 4 layers, (b) 16 layers, and (c) 32 layers respectively, with performance normalized against the M-Base configuration.

However, from a system-level scalability perspective, the empirical results also reveal an intrinsic architectural limitation associated with temporal dependencies in dynamic graphs. As the number of stacked layers increases from 4 to 16 and 32 (Figure 8b,c), the relative acceleration margins gradually shrink. At 32 stacked layers, the peak speedup of M-DGNN over BLAD drops from 6.2× to about 3.9×, while its gain over M-Base decreases to 2.15×. This reduction in speedup at larger layer counts is consistent with Amdahl’s Law: while our architecture effectively alleviates spatial I/O bottlenecks, the sequential dependencies across deep RNN layers inevitably force compute units to idle while waiting for temporally preceding features. These pipeline bubbles reduce overall hardware utilization and fundamentally limit the achievable end-to-end scalability of our PCIe-based multi-CSD system at higher layer counts. Overall, the end-to-end gains of M-DGNN arise from the combination of near-data execution, P2P-based hidden state transfer, and HPME-assisted reduction of flash-access stalls.

5.3. Effectiveness of Proposed Designs

This subsection evaluates the effectiveness of the key mechanisms in M-DGNN by isolating the individual contributions of the hardware P2P DMA routing engine and the HPME caching mechanism.

5.3.1. Hardware P2P DMA Routing Engine

This experiment isolates the contribution of the hardware P2P DMA routing engine and evaluates the effectiveness of direct inter-CSD communication.

We evaluate the effectiveness of the hardware P2P DMA routing engine using a four-CSD setup, comparing the naive multi-CSD baseline (M-Base) against the P2P-optimized architecture (M-P2P). Figure 9 breaks down the end-to-end execution time into GNN, RNN, and Other phases across ten configurations. We use varying model layers ranging from 4 to 32 on the Reddit (RD) and Amazon (AM) datasets. Note that the RNN phase encapsulates both the sequential feature computation and the overhead of inter-layer hidden state transmission. As illustrated by the left bars (M-Base) in the figure, the traditional host-bounced memory copy mechanism causes severe latency during hidden state exchanges. Consequently, the RNN phase dominates the total runtime, taking up to 75.59% in the AM-L32 configuration and averaging 64.07% across all evaluated setups, making it the primary performance bottleneck. By integrating the P2P DMA routing engine, M-P2P enables direct device-to-device data transfers. This entirely bypasses host interrupts, context switches, and redundant DMA initialization. Because this P2P optimization is orthogonal to the spatial graph computation, the absolute execution times of the GNN and other phases remain unchanged between M-Base and M-P2P. The P2P mechanism exclusively mitigates the inter-device routing latency within the RNN phase. As a result, it shrinks the average RNN time proportion from 64.07% to just 31.62%, yielding an average end-to-end speedup of 1.44× over M-Base (with a peak acceleration of 1.55× on AM-L8). These results indicate that the P2P routing engine is the main contributor to reducing hidden-state communication overhead in the RNN phase.

Figure 9. Execution time breakdown isolating the effect of the P2P DMA routing engine (left stack: M-Base; right stack: M-P2P).

5.3.2. HPME Caching Mechanism

Next, we evaluate the effectiveness of the HPME caching mechanism. This experiment isolates the contribution of the HPME caching mechanism. Figure 10 normalizes the execution time against the transmission-optimized architecture (M-P2P, set as the 1.0 baseline) and displays three main execution phases: RNN, GNN, and Other. In this context, the GNN phase time accounts for both the actual GNN logic computation and the high latency of physical NAND flash array accesses. As shown in the left bars (M-P2P), fetching scattered neighbor features directly from flash drives hits a severe memory wall. Specifically, flash I/O overhead consumes an average of 69.72% of the GNN phase, making the aggregated GNN operation a major performance bottleneck. By integrating hardware-assisted caching (M-DGNN; right bars), the system intercepts most locality-rich feature requests, successfully avoiding slow flash I/O. Similar to the P2P optimization, this caching mechanism is completely orthogonal to temporal sequence modeling. Therefore, the absolute elapsed time on the RNN and Other phases remains unchanged between M-P2P and M-DGNN. The buffer pool exclusively accelerates the GNN phase by significantly reducing synchronous flash I/O stalls. Ultimately, it drops the average flash I/O proportion during the GNN phase down to 44.68%, delivering an average end-to-end speedup of 1.30× strictly over the M-P2P baseline. These results indicate that HPME is the main contributor to reducing flash access overhead in the GNN phase.

Figure 10. Execution time breakdown isolating the effect of the HPME caching mechanism (left stack: M-P2P; right stack: M-DGNN).

5.4. Scalability Analysis

This experiment evaluates the scalability of M-DGNN and evaluates how its end-to-end throughput changes as the number of CSDs increases. To evaluate the scale-out capability of M-DGNN, we measure the end-to-end throughput of M-Base and M-DGNN on the RD and AM datasets as the number of CSDs scales from 1 to 16. We test both shallow (4-layer; L4) and deep (32-layer; L32) model configurations, which represent common industrial and academic workloads. All the performance metrics are normalized to their respective single-CSD execution throughput. As shown in Figure 11, for the shallow RD-L4 model, M-Base achieves only a 3.15× speedup when scaled to 16 CSDs, whereas M-DGNN reaches a peak speedup of 5.11×. For the deep RD-L32 model, the serial dependency of the RNN dominates the execution timeline, restricting M-DGNN’s 16-CSD speedup to 4.38×. A similar trend occurs on the larger AM dataset, where M-DGNN obtains 4.57× and 3.92× speedups for the L4 and L32 configurations at 16 CSDs, respectively. Ultimately, while M-DGNN successfully breaks the communication and I/O bottlenecks of naive multi-CSD schemes, its maximum scalability remains fundamentally bounded by the unoptimized sequential dependencies of RNNs. Addressing these temporal dependencies through parallel scheduling strategies will be a critical direction for future near-data DGNN accelerators. While M-DGNN offloads bulk data transfers to the PCIe P2P fabric, the host metadata manager remains responsible for lightweight doorbell signaling and routing metadata updates. For the 16-CSD configuration evaluated in this work, the control plane overhead remains negligible (accounting for less than 0.45% of the total execution time).

Figure 11. Scalability analysis of M-DGNN. End-to-end throughput is evaluated on the Reddit (RD) and Amazon (AM) datasets under shallow (L4) and deep (L32) model configurations as the system scales from 1 to 16 CSDs. Results are normalized to the single-CSD execution. (a) RD-L4; (b) RD-L32; (c) AM-L4; and (d) AM-L32.

5.5. Energy Efficiency

This experiment analyzes the energy efficiency benefit of M-DGNN compared with the host-centric baseline and its intermediate variants. Beyond raw throughput, the near-data computing paradigm offers fundamental advantages in energy efficiency. Figure 12 illustrates the normalized end-to-end energy consumption across the BLAD, M-Base, M-P2P, and full M-DGNN architectures. All the results are strictly normalized against the state-of-the-art BLAD framework. By eliminating power-hungry, long-distance data shuttling between host memory and standard storage devices, the naive near-data baseline (M-Base) reduces overall energy consumption by 68.27% compared to BLAD. Building upon this, our P2P routing engine (M-P2P) delivers an additional 19.34% energy reduction by natively bypassing redundant host-bounced memory copies. Finally, the fully tailored M-DGNN architecture achieves a further 26.37% drop in energy usage via the localized caching mechanism, which effectively replaces power-intensive flash I/O accesses with highly efficient host DRAM lookups.

Figure 12. Normalized end-to-end energy consumption comparison among the BLAD framework, naive near-data baseline (M-Base), P2P-optimized architecture (M-P2P), and the fully integrated M-DGNN system. All results are normalized to the BLAD baseline.

6. Related Work

There is extensive research on GNN acceleration for both inference and training. Some design specialized compute engines and dataflows for GNNs’ irregular patterns [50,51,52,53]. Others co-design algorithms and hardware using GNNs’ feature sparsity and degree distribution to cut data access and redundancy [54,55,56,57]. Certain studies employ emerging hardware like ReRAM for parallelism and energy efficiency [58,59]. More broadly, recent studies on in-memory computing further expand the design space for efficient data-intensive computation through emerging device modeling [60,61]. For ISC in GNNs, SmartSage [18] and GList [62] do not fully offload the taskflow and have discussed limitations. GraphStore [63] stores data in SSD but computes on an FPGA, facing bandwidth constraints and lacking host control removal or flash backend I/O optimization. BeaconGNN [17] optimizes channel transfer, I/O communication, and computation, thereby addressing the aforementioned limitations and setting a new standard for ISC in GNN applications.

Regarding DGNN models, prior research has primarily focused on developing efficient and scalable training schemes under compute scaling. ESDG [29] proposed a snapshot partition scheme to overcome the issue of vertex feature movement in DGNNs. They introduced a hybrid technique that combines data parallelism and model parallelism to ensure the scalability of training dynamic temporal graph networks. PiPAD [33] enables pipelined and parallel training for single accelerator training. Cambricon-G [64] proposed an efficient accelerator designed to avoid unnecessary graph topology updates for DGNNs. Dynagraph [65] and TGOpt [54] analyze and reduce redundant graph feature computations in the training pipeline. BLAD [31] proposed a snapshot group partition scheme and focuses on achieving load balance between different stages of the training pipeline for DGNNs. DGC [32] mitigates communication overhead and optimizes scheduling by organizing snapshots into graph chunks. However, these systems all result in significant data movement overhead. As dynamic graph data grows, TB-level server memory cannot hold the full dataset. These systems must access dynamic graphs from SSDs, causing data movement issues via the narrow PCIe bus. M-DGNN focuses on providing system support and performance optimization for DGNNs under dynamic graph scaling [12].

To better position M-DGNN with respect to representative prior systems, we further clarify the differences in targeted workloads, execution models, and optimization scope. Prior near-storage and in-storage studies such as BeaconGNN, SmartSAGE, and FlashGNN show the benefit of moving graph-learning-related processing closer to storage, which helps reduce host-side data movement and improve data locality. However, these efforts mainly focus on static graph settings or on offloading selected parts of the GNN pipeline, and therefore differ from our focus on large-scale DGNN training. In contrast, prior DGNN systems such as ESDG, BLAD, and DGC are designed for DGNN training, but they adopt a host-centric execution model in which the host remains the main coordination and data movement point. Compared with these two lines of work, M-DGNN targets DGNN training on PCIe-interconnected multi-CSD systems and combines storage-side execution, direct inter-device communication through PCIe P2P DMA, and host-pinned-memory-assisted caching through HPME within a unified hardware–software co-design framework. This distinction is important because our goal is not only to move selected computation closer to data, but also to mitigate the communication and flash-access bottlenecks that emerge during distributed DGNN training across multiple CSDs.

7. Conclusions

In this work, we present M-DGNN, a pragmatic hardware–software co-designed architecture crafted to execute massive-scale DGNN training across standard PCIe-interconnected topologies. By systematically eliminating disruptive host-side routing interventions via P2P mapping, and insulating the embedded NAND flash media from I/O thrashing through an asynchronous HPME strategy, M-DGNN effectively dismantles the pervasive I/O and communication bottlenecks that limit the performance of naive multi-CSD DGNN setups. Extensive cycle-level experimental evaluations across seven large-scale dynamic graph datasets demonstrate that M-DGNN provides up to a 6.2× end-to-end acceleration compared to state-of-the-art distributed DGNN frameworks, achieves a 5.11× peak throughput speedup when scaled horizontally to 16 CSDs, and also delivers significant holistic energy efficiency improvements over conventional host-centric DGNN systems by reducing power-intensive data shuttling and redundant flash accesses. While our design effectively breaks the core I/O and communication barriers of multi-CSD based DGNN acceleration, the inherent rigid sequential dependencies of RNN layers in temporal modeling still limit the ultimate horizontal scalability of the architecture as device count increases. Future research will focus on designing parallel scheduling and pipeline orchestration strategies for these temporal dependencies to further improve the overall hardware utilization and scalability of large-scale DGNN training.