STAGE: LLM-Driven Semantic and Topological Augmented Graph Embedding for Text-Attributed Graphs

Abstract

Text-attributed graphs (TAGs) require models to jointly exploit node text and graph structure, yet doing so effectively remains difficult when node text is sparse and the structural context is large. Here, we propose STAGE (Semantic and Topological Augmented Graph Embedding), a two-stage framework for representation learning on TAGs. In Stage I, a frozen large language model is used offline to generate explanatory text that enriches compressed node attributes without introducing online LLM training cost. In Stage II, STAGE performs structure-aware representation learning under a fixed global token budget by combining random-walk-based structural context with graph-conditioned token reduction before PLM encoding. This design preserves informative semantic content while preventing unconstrained sequence expansion. Experiments on seven benchmark datasets show that STAGE consistently outperforms strong baselines under the same evaluation setting and maintains favorable efficiency under bounded input-length constraints.

1. Introduction

Graphs serve as a fundamental data structure for modeling entities and their complex interrelationships through nodes and edges [1,2,3,4]. Specifically, many real-world graphs are associated with rich node text, such as paper abstracts, product descriptions, or user-generated content. These data are commonly studied as text-attributed graphs (TAGs), where nodes carry textual attributes and edges describe relations between them. For this setting, Graph Neural Networks (GNNs) have become a standard choice because they can aggregate information over local neighborhoods and make effective use of graph connectivity, especially in low-label settings [5,6,7,8].

A practical difficulty, however, lies in how node text is represented before graph propagation begins. Early TAG methods often encode text with shallow features such as Bag-of-Words or Skip-gram embeddings [5]. These representations are easy to use, but they usually miss contextual meaning and domain-specific distinctions. Once such weak features are fed into a GNN, message passing may propagate irrelevant or weakly informative features across neighboring nodes, which limits the quality of the learned node representations [9]. Later methods improve the text encoder itself. For example, PLM-based models such as BERT can be combined with GNNs, and approaches like GraphBridge [10] further incorporate neighboring text during encoding. These models provide substantially stronger contextual representations than shallow text features. However, they still rely mainly on the original node text as the primary semantic source. In many specialized TAGs, that text is short, technical, and semantically compressed, so even a strong pretrained encoder may receive insufficient evidence to recover the missing background knowledge from the input alone [9,10].

Large Language Models (LLMs) make this limitation more tractable because they can supply missing context, definitions, and semantic expansion that are difficult to obtain directly from raw node attributes alone [11,12]. However, using LLMs inside TAG pipelines is not straightforward. Direct fine-tuning is expensive, and prompt-based use does not automatically solve how graph structure should be incorporated. Existing LLM-based methods can enrich node features, yet they often process nodes one by one and leave structural interaction to a later stage [13]. In addition, LLM outputs may be unreliable in highly specialized domains, and standard self-attention is not well suited to large graph neighborhoods without substantial computational cost [13].

Figure 1 illustrates the practical tension behind current approaches. Methods built on shallow or handcrafted features are lightweight, but they leave much of the node semantics under-modeled. Joint text structure models improve structural reasoning, yet they are still constrained by the quality of the original text. Generative LLM-based methods can enrich semantics, but the generated information is often attached to nodes independently, with limited interaction between semantic augmentation and graph propagation. In other words, current TAG methods tend to be stronger on either semantic enrichment or structural modeling, but less convincing when both are needed at the same time.

We address this issue with STAGE (Semantic and Topological Augmented Graph Embedding), a decoupled framework for TAG representation learning. STAGE separates semantic enrichment from structure-aware encoding. In the first stage, a frozen LLM is used offline to generate explanatory text that complements sparse node text. In the second stage, the enriched text is encoded together with structural context, but under an explicit token budget. To make this feasible, we introduce a graph-conditioned token reduction module that keeps informative tokens from the structural neighborhood before PLM encoding. This design allows the model to use richer semantic inputs without letting sequence length grow uncontrollably as more graph context is considered.

The main contributions of this paper are as follows:

• We present STAGE, a decoupled framework that combines offline LLM-based semantic enrichment with GNN-based structural propagation for text-attributed graphs.
• We introduce a graph-conditioned token reduction mechanism that performs token selection before deep encoding. Unlike topology-agnostic pruning, it uses structural context to retain informative semantic anchors while keeping the PLM input length within a fixed computational budget.
• We evaluate STAGE on seven benchmark datasets and show that it outperforms strong baselines under our evaluation protocol, while maintaining favorable efficiency on graphs with different characteristics.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the proposed STAGE framework. Section 4 reports the experimental results. Section 5 provides additional discussion and analysis. Finally, Section 6 concludes the paper.

2. Related Work

2.1. TAG Representation Learning

Research on text-attributed graphs (TAGs) has progressed from relatively separate modeling of text and structure to more tightly coupled architectures. Early graph-based methods such as GCN [5] and GraphSAGE [14] are effective at neighborhood aggregation, but they typically rely on shallow text features such as Bag-of-Words. As a result, their initial node representations may miss contextual meaning and domain-specific terminology [15]. In contrast, text-centric encoders based on PLMs such as BERT [16] provide stronger contextual representations, but they process nodes independently and therefore do not directly model graph connectivity [10,17].

Subsequent work moved toward the joint modeling of node text and graph structure. Methods such as GLEM [17] and SimTeG [18] combine PLMs and GNNs through iterative training or distillation. GraphBridge [10] further allows the language encoder to incorporate neighboring text during encoding, making structure available earlier in the representation process. These methods improve the interaction between text and topology, but they still depend heavily on the quality of the original node attributes. In specialized TAG settings, the raw text is often short, technical, and incomplete, which limits how much semantic information can be recovered from the input alone [9,10]. Related work has also explored language model pretraining on text-rich networks [19], but this line of work does not focus on offline semantic generation under a fixed token budget.

2.2. LLM-Based Semantic Augmentation for Graph Learning

Recent work has used LLMs to enrich graph data before downstream learning. TAPE [13], for example, generates rationales or predictions that are then attached to nodes as additional features. KEA [15] extracts entities and generates short definitions to reduce the risks associated with direct zero-shot use of LLMs, while still using the generated content as semantic augmentation rather than end-to-end graph reasoning. More broadly, graph LLM studies have explored prompting, explanation generation, and graph-aware adaptation as ways to inject richer semantics into graph learning pipelines [20,21,22].

These methods improve the semantic content of node attributes, but the generation step is typically performed for each node separately. As a result, the added information is not tightly coupled with graph structure at the time it is produced or encoded [23]. In other words, they strengthen semantic enrichment, but they do not fully address how the enriched text should be filtered and integrated once structural context becomes large.

Recent LLM applications have shown that LLMs can support a variety of text-centered analysis and decision support tasks, such as mental health text analysis [24], cyber threat intelligence [25], fake or real tweet classification [26], virtual assistant command classification [27], and LLM-based evaluation [28]. These studies suggest that LLMs can provide useful semantic cues for downstream models, but they also indicate that LLM-generated information should be used carefully in specialized or high-stakes domains. This observation further motivates our decoupled design: STAGE uses the LLM as an offline semantic enhancer, while the downstream PLM and GNN components remain responsible for supervised structure-aware representation learning.

2.3. Efficient Transformers and Token Reduction

The Transformer architecture has emerged as the de facto standard for representation learning across a wide array of domains [29,30,31]. Recently, a separate line of research studies how to reduce the cost of Transformer encoding. Since standard self-attention scales quadratically with sequence length [32], prior research has explored sparse attention, low-rank approximations, and token reduction methods such as BigBird [33], PoWER-BERT [34], ToMe [35], and learned token pruning [36,37]. However, most of these methods are designed for plain text or image inputs, where token importance is determined mainly by redundancy within a single sequence.

In TAGs, the situation is different because token importance also depends on how the text relates to structural context. This is particularly relevant when semantic augmentation and graph propagation are combined: broader neighborhoods can provide useful context, but they can also quickly increase sequence length and introduce irrelevant tokens. STAGE is designed for this setting. Compared with joint text structure models, it supplements sparse node text with offline LLM-generated explanations. Compared with existing LLM-based augmentation methods, it introduces graph-conditioned token selection before PLM encoding so that structural context affects which tokens are retained under a fixed computational budget.

3. Materials and Methods

3.1. Preliminaries

We first define the text-attributed graph setting studied in this paper and summarize the modeling components used in STAGE. These include Graph Neural Networks for structural propagation, Pretrained Language Models for text encoding, and prompted large language models for offline semantic augmentation. We also briefly describe common ways of incorporating LLMs into graph learning to clarify where STAGE fits.

• Text-Attributed Graphs. A text-attributed graph is denoted as ${\mathcal{G}}_T=(\mathcal{V},\mathcal{E},\mathcal{T})$, where $\mathcal{V}$ is the node set and $\mathcal{E}$ is the edge set. Each node $v\in \mathcal{V}$ is associated with a raw text sequence $t_v\in \mathcal{T}$ and a corresponding discrete label $y_v\in \mathcal{Y}$. In this work, we focus on semi-supervised node classification, which aims to predict the labels of unlabeled nodes ${\mathcal{V}}_U$ given the graph topology, textual attributes, and a small subset of labeled nodes ${\mathcal{V}}_L$.
• GNN-Based Paradigm. Graph Neural Networks (GNNs) learn node representations by recursively aggregating information from topological neighbors. Formally, the hidden representation of node v at the l-th layer is updated as

  $${\mathbf{h}}_v^{\left(l\right)}=\mathrm{UPD}\left({\mathbf{h}}_v^{(l-1)},\mathrm{AGG}\left(\{{\mathbf{h}}_u^{(l-1)}:u\in \mathcal{N}\left(v\right)\}\right)\right),$$

  (1)

  where $\mathcal{N}\left(v\right)$ denotes the set of neighbors of v, and $\mathrm{AGG}(·)$ and $\mathrm{UPD}(·)$ represent the aggregation and update functions, respectively. The initial feature ${\mathbf{h}}_v^{\left(0\right)}$ is typically derived from the textual attribute $t_v$.
• Pretraining of PLMs. Pretrained Language Models (PLMs), such as BERT and RoBERTa, generate contextualized text representations via self-attention mechanisms. Given a raw text $t_v$, it is first tokenized into a sequence $S_v={\left\{s_{v,i}\right\}}_{i=1}^{|S_v|}$. A PLM encodes this sequence into contextualized hidden states:

  $${\mathbf{H}}_v=\mathrm{PLM}\left(S_v\right)\in {\mathbb{R}}^{|S_v|\times d}.$$

  (2)
  Instandard sequence classification tasks, the embedding of the special [CLS] token is commonly extracted as the global semantic representation of the entire text.
• LLMs with Prompting. Unlike PLMs that are typically fine-tuned to adapt to downstream tasks, Large Language Models (LLMs) are often utilized in a frozen state via prompting. Let $\mathcal{M}$ denote a frozen LLM. A transformation function $\mathcal{P}(·)$ wraps the raw text $t_v$ into a natural language prompt. The model then generates tokens autoregressively according to

  $$y_t\sim P_{\mathcal{M}}(y_t\mid \mathcal{P}\left(t_v\right),y_{<t}).$$

  (3)
  This formulation highlights the role of prompting in extracting or generating semantic information without updating the parameters of the LLM.
• Taxonomy of LLM Integration. Existing work uses LLMs in graph learning in different ways [15,20,21,22]. One distinction is whether the language model is directly tunable or only accessible through prompting. Another is whether the model is used to predict labels directly or to enrich node attributes before downstream learning. STAGE belongs to the latter setting. It uses a prompted, frozen LLM to generate additional semantic context, while a trainable PLM and a GNN are responsible for structure-aware representation learning.

3.2. The Proposed STAGE Framework

Figure 2 shows the overall design of STAGE. Instead of coupling semantic augmentation and graph propagation in a single training loop, STAGE separates them into two stages. The first stage enriches node text offline with LLM-generated explanations. The second stage learns structure-aware representations from the enriched text under a fixed token budget. This separation keeps the semantic augmentation step simple to apply while making the subsequent encoding and propagation stages easier to control computationally.

3.2.1. Stage I: Generative Semantic Injection

In many TAG datasets, the raw text attached to each node is short and highly compressed. Citation graphs are a typical example: abstracts often contain technical terms but leave out the background needed to interpret them reliably. To supplement this missing context, we use a frozen LLM offline to generate explanatory text for each node. Because this stage is detached from the downstream training loop, it enriches node attributes without introducing the cost of LLM fine-tuning into the representation learning process.

• Knowledge Extraction via Prompting. For each node v, we construct a prompt $\mathcal{P}\left(t_v\right)$ based on its raw text $t_v$. The prompt asks the LLM to identify technical terms in the input text and generate a short description for each term. To keep the prompting strategy consistent across datasets with different text styles, we use a general extraction template rather than a dataset-specific domain instruction. The prompt template used in our implementation was as follows:

  You should work like a named entity recognizer. Text: [text]. Extract the technical terms from this text and output a description for each term in the format of a Python (version 3.10) dictionary, e.g., {’XX’: ’XXX’, ’YY’: ’YYY’}.

The generation process can be written in shorthand as

$$e_v=\mathcal{M}\left(\mathcal{P}\left(t_v\right)\right),$$

(4)

where $\mathcal{M}$ denotes the frozen LLM introduced in the preliminaries, and $e_v$ is the generated explanatory text. In practice, this step makes the semantics of brief or specialized node text more explicit before graph-based learning begins.

• Implementation Details of Stage I. In our implementation, Stage I uses a unified general-text prompt for all datasets. For each node, we apply a lightweight input-length control step before prompting and then submit the processed text to the frozen LLM using the same extraction template. In our OpenRouter-based API calls, we did not explicitly set temperature, max_tokens, top_p, frequency penalty, or presence penalty. Therefore, these values followed the model/provider-side default behavior. To avoid introducing undocumented manual settings, we now report this configuration explicitly. The same API configuration is applied to all nodes and datasets. The generated response is post-processed into plain explanatory text and concatenated with the original node attribute. Malformed outputs and empty generations are filtered by simple format checking, and nodes that fail this step fall back to their original text only. Since Stage I is executed offline, the generation cost is incurred once and does not appear in the optimization of the PLM or GNN components.
• Semantic Enrichment. We concatenate the original node text with the generated explanation to obtain an enriched attribute:

  $${\widehat{t}}_v=[t_v\oplus e_v],$$

  (5)

  where ⊕ denotes concatenation. The resulting set $\widehat{\mathcal{T}}={\left\{{\widehat{t}}_v\right\}}_{v\in \mathcal{V}}$ is then used in the second stage. Since the LLM is only used offline, its computation does not appear in the training loop of the PLM or the GNN.

3.2.2. Stage II: Structure-Aware Representation Learning

Although the enriched attributes $\widehat{\mathcal{T}}$ contain more semantic information, using them together with large structural neighborhoods can quickly make PLM encoding too expensive. A straightforward concatenation of neighbor text increases sequence length, which in turn raises the cost of self-attention quadratically [32]. Existing token reduction methods such as ToMe [35] can shorten sequences, but they are usually designed for plain sequences and do not consider graph structure. STAGE addresses this issue with a graph-conditioned token reduction module. Before PLM encoding, the module selects tokens from the structural context under a fixed budget, so the model can incorporate neighborhood information without letting the input length grow unchecked.

The selector is intentionally lightweight because it is applied before PLM encoding and is used to score candidate tokens under a fixed budget. A more expressive cross-attention module could model richer token–neighbor interactions, but it would also introduce additional computation before token reduction and weaken the efficiency motivation of the proposed design.

• Random Walk Context Sampling. To capture local structural context, we construct a walk-based subgraph for each target node v. Starting from v, we perform a random walk with restart and collect the visited nodes into a sampled structural view ${\mathcal{G}}_v^{\left(L\right)}$, where L denotes the walk depth. Compared with BFS-style neighborhood expansion, this strategy provides a more controllable way to explore local context without deterministically including all nodes within a fixed radius. Compared with attention-based neighbor sampling, it does not introduce an additional trainable sampling module. In practice, the sampled subgraph is later linearized into a sequence by concatenating the reduced texts of its constituent nodes under a fixed global token budget.
• Graph-Conditioned Token Selection. To enforce a fixed computational budget $L_{max}$, STAGE introduces a graph-conditioned token selector ${\mathcal{S}}_{\varphi }(·)$. In our implementation, the selector is trained at the node level. For each target node v, let ${\left\{{\mathbf{e}}_{v,k}\right\}}_{k=1}^{|t_v|}$ denote the token embeddings of its enriched text, and let ${\mathbf{n}}_v$ denote a structural conditioning vector obtained from multi-hop neighborhood aggregation. The selector then produces a relevance score for each token:

  $${\alpha }_{v,k}={\mathcal{S}}_{\varphi }({\mathbf{e}}_{v,k},{\mathbf{n}}_v),$$

  (6)

  where ${\alpha }_{v,k}\in (0,1)$ reflects the importance of token k in the presence of graph context.

During selector training, we use a soft aggregation scheme rather than hard truncation. Specifically, the token embeddings are summarized as

$${\mathbf{s}}_v=\sum _k{\alpha }_{v,k}{\mathbf{e}}_{v,k},$$

(7)

and the selector parameters are optimized with a supervised classification loss together with a regularization term:

$${\mathcal{L}}_{sel}={\mathcal{L}}_{cls}+\beta {\mathcal{L}}_{reg},$$

(8)

where ${\mathcal{L}}_{cls}$ is the cross-entropy loss on the node label and ${\mathcal{L}}_{reg}$ regularizes the score distribution over valid tokens.

In practice, the regularization term encourages the selector to remain well behaved over the valid token positions instead of collapsing prematurely to an overly sharp distribution. Let ${\mathbf{m}}_v$ denote the binary attention mask over valid tokens. We define a normalized reference distribution over valid positions as

$${\mathbf{u}}_v={\displaystyle \frac{{\mathbf{m}}_v}{{\sum }_j{m}_{v,j}}},$$

(9)

Let ${\overline{\mathit{\alpha }}}_v$ denote the normalized selector distribution over valid tokens:

$${\overline{\alpha }}_{v,k}={\displaystyle \frac{{\alpha }_{v,k}{m}_{v,k}}{{\sum }_j{\alpha }_{v,j}{m}_{v,j}}}.$$

(10)

We then define the regularization term as

$${\mathcal{L}}_{reg}=\mathrm{KL}\left({\overline{\mathit{\alpha }}}_v\parallel {\mathbf{u}}_v\right)=\sum _{k:m_{v,k}=1}{\overline{\alpha }}_{v,k}log{\displaystyle \frac{{\overline{\alpha }}_{v,k}}{u_{v,k}}},$$

(11)

where ${\overline{\mathit{\alpha }}}_v$ denotes the normalized selector distribution over the valid tokens of node v.

After the selector is trained and frozen, we use it during sequence construction. For each sampled subgraph ${\mathcal{G}}_v^{\left(L\right)}$, we allocate a per-node token quota under the fixed global budget $L_{max}$, retain the highest-scoring tokens for each node, and concatenate the reduced node texts with separator tokens. In this way, the final PLM input remains bounded even when the sampled structural context becomes larger.

• Hierarchical Encoding and Aggregation. The term hierarchical refers to the fact that the structure of the STAGE models has two levels: sequence-level encoding and graph-level aggregation. After token selection, we obtain a reduced text sequence ${\tilde{t}}_v$ from the enriched attribute and its sampled structural context. The PLM then encodes this reduced sequence to produce a dense node embedding ${\mathbf{h}}_v\in {\mathbb{R}}^d$:

  $${\mathbf{h}}_v=\mathrm{Encoder}\left({\tilde{t}}_v\right).$$

  (12)
  Because the input sequence already integrates filtered random walk contexts, ${\mathbf{h}}_v$ captures local structural information at the sequence level. Subsequently, these embeddings serve as initial features for a GNN aggregator, which propagates them over the graph topology to capture higher-order dependencies:

  $${\mathbf{z}}_v=\mathrm{GNN}\left({\mathbf{h}}_v,\mathbf{A};{\mathsf{\Theta }}_G\right).$$

  (13)
  For node classification, we further apply a linear classifier on top of ${\mathbf{z}}_v$ to obtain the prediction

  $${\widehat{\mathbf{y}}}_v=\mathrm{softmax}({\mathbf{W}}_c{\mathbf{z}}_v+{\mathbf{b}}_c),$$

  (14)

  and optimize the GNN stage with the supervised cross-entropy loss

  $${\mathcal{L}}_{gnn}=-\sum _{v\in {\mathcal{V}}_L}{\mathbf{y}}_v^{\top }log{\widehat{\mathbf{y}}}_v,$$

  (15)

  where ${\mathcal{V}}_L$ denotes the set of labeled nodes, and ${\mathbf{y}}_v$ is the one-hot ground-truth label of node v.

3.2.3. Optimization and Complexity Analysis

Algorithm 1 summarizes the training pipeline of STAGE.

• Cascaded Optimization Strategy. We train STAGE in three steps instead of optimizing all components jointly. This design keeps the pipeline manageable and avoids passing gradients through the full LLM–PLM–GNN stack at once.

• Step 1: Train the Token Selector. We first optimize the token selector ${\mathcal{S}}_{\varphi }$ at the node level. For each target node, the selector receives token embeddings from the enriched text together with a structural conditioning vector derived from neighborhood aggregation. Because hard Top-K truncation is not differentiable, we train the selector with a soft aggregation scheme and optimize it using the loss ${\mathcal{L}}_{sel}={\mathcal{L}}_{cls}+\beta {\mathcal{L}}_{reg}$. The selector is frozen after this stage.
• Step 2: Fine-Tune the PLM Encoder. After freezing the selector, we construct a walk-based subgraph for each target node and allocate a per-node token quota under the fixed global budget $L_{max}$. The selector scores are then used to retain the most informative tokens for each node, and the resulting reduced node texts are concatenated into a bounded PLM input sequence. The PLM encoder is fine-tuned on these reduced inputs with the classification loss ${\mathcal{L}}_{enc}$.
• Step 3: Train the GNN Aggregator. We then freeze the PLM encoder, use it to produce node representations ${\mathbf{h}}_v$, and train the GNN aggregator on top of these representations with the supervised downstream loss ${\mathcal{L}}_{gnn}$. Since only one trainable module is optimized at a time, the overall training process is easier to control in memory and implementation complexity than a fully joint alternative.

Algorithm 1 Training Pipeline of the Proposed STAGE

Require: Text-attributed graph ${\mathcal{G}}_T=(\mathcal{V},\mathcal{E},\mathcal{T})$, frozen LLM $\mathcal{M}$, token selector ${\mathcal{S}}_{\varphi }$, PLM   encoder, GNN aggregator, global token budget $L_{max}$.   1: % Stage I: Offline Generative Semantic Injection   2: for each node $v\in \mathcal{V}$ do   3:     Construct a prompt $\mathcal{P}\left(t_v\right)$ from raw text $t_v$   4:     Generate explanatory text $e_v\leftarrow \mathcal{M}\left(\mathcal{P}\left(t_v\right)\right)$   5:     Form enriched attribute ${\widehat{t}}_v\leftarrow [t_v\oplus e_v]$   6: end for   7: % Stage II: Structure-Aware Representation Learning   8: Step 1: Train the token selector   9: for each target node v in batch do 10:     Encode the enriched text of v into token embeddings $\left\{{\mathbf{e}}_{v,k}\right\}$ 11:     Construct a structural conditioning vector ${\mathbf{n}}_v$ from multi-hop neighborhood aggregation 12:     Compute token relevance scores ${\alpha }_{v,k}\leftarrow {\mathcal{S}}_{\varphi }({\mathbf{e}}_{v,k},{\mathbf{n}}_v)$ 13:     Form a soft summary ${\mathbf{s}}_v\leftarrow {\sum }_k{\alpha }_{v,k}{\mathbf{e}}_{v,k}$ 14:     Update $\varphi$ using selector loss ${\mathcal{L}}_{sel}$ 15: end for 16: Step 2: Fine-tune the PLM encoder 17: for each target node v in batch do 18:     Sample a walk-based structural view ${\mathcal{G}}_v^{\left(L\right)}$ 19:     Allocate a per-node token quota under the fixed global budget $L_{max}$ 20:     Retain the highest-scoring tokens for each node in ${\mathcal{G}}_v^{\left(L\right)}$ 21:     Concatenate the reduced node texts into a bounded sequence ${\tilde{t}}_v$ 22:     Encode ${\tilde{t}}_v$ with the PLM encoder to obtain ${\mathbf{h}}_v$ 23:     Update the encoder using classification loss ${\mathcal{L}}_{enc}$ 24: end for 25: Step 3: Train the GNN aggregator 26: for each target node v in batch do 27:     Propagate PLM-based node representations through the graph: 28:          ${\mathbf{z}}_v\leftarrow \mathrm{GNN}({\mathbf{h}}_v,\mathbf{A};{\mathsf{\Theta }}_G)$ 29:     Update GNN parameters using downstream supervision ${\mathcal{L}}_{gnn}$ 30: end for 31: return Trained selector ${\mathcal{S}}_{\varphi }$, PLM encoder, and GNN aggregator

Table 1. Analytical comparison of representative TAG encoding strategies.

Strategy	Structural Text Usage	PLM-Side Sequence Cost	Main Limitation
Node-text-only PLM	Target node text only	$O\left(\right|t_v{|}^{2}d)$	Does not encode structural context in the PLM input
Unconstrained joint context encoding	Raw target and neighbor text	$O\left(R_v^{2}d\right)$	Cost grows rapidly with sampled context length
Topology-agnostic truncation	Truncated context sequence	$O\left(L_{max}^{2}d\right)$	Retention ignores graph-conditioned token relevance
STAGE	Graph-conditioned reduced context	$O\left(L_{max}^{2}d\right)$	Requires preprocessing for sampling and token scoring

provides an analytical comparison of representative TAG encoding strategies.

• Formal Properties of Graph-Conditioned Reduction. Let $R_v$ denote the number of raw candidate tokens collected from the sampled structural view ${\mathcal{G}}_v^{\left(L\right)}$ for node v, and let ${\tilde{L}}_v$ denote the final number of tokens retained for PLM encoding after graph-conditioned reduction. By construction, STAGE enforces the bounded-input property:

  $${\tilde{L}}_v\le L_{max},\forall v\in \mathcal{V}.$$

  (16)
  This property implies that the PLM encoder never receives an input sequence whose length grows without bound as the sampled structural context expands. In particular, although $R_v$ may increase substantially with walk depth L, the final encoded sequence is always upper-bounded by the global token budget.

A second implication is a decoupling effect between context growth and PLM-side cost. Increasing L enlarges the sampled subgraph and therefore increases graph construction and sequence assembly overhead, but it does not allow the quadratic self-attention cost of the PLM to scale directly with the full raw neighborhood text. In this sense, graph-conditioned reduction changes the dominant cost from unconstrained PLM attention over long sequences to a bounded-input encoding stage together with a larger preprocessing stage.

• A Compositional View of STAGE. STAGE can be viewed as a composition of four operators over graph–text inputs. Let $\mathcal{A}$ denote the offline semantic augmentation operator, $\mathcal{R}$ the graph-conditioned reduction operator, $\mathcal{E}$ the PLM encoding operator, and ${\mathcal{P}}_G$ the graph propagation operator. Then, the overall pipeline can be written schematically as

  $${\mathcal{G}}_T\stackrel{\mathcal{A}}{⟶}\widehat{\mathcal{T}}\stackrel{\mathcal{R}}{⟶}\tilde{\mathcal{T}}\stackrel{\mathcal{E}}{⟶}\mathbf{H}\stackrel{{\mathcal{P}}_G}{⟶}\mathbf{Z},$$

  (17)

  where $\widehat{\mathcal{T}}$ denotes the semantically enriched node text, $\tilde{\mathcal{T}}$ the reduced structure-aware text sequences, $\mathbf{H}$ the PLM-based node embeddings, and $\mathbf{Z}$ the final graph-aware representations. This view highlights that STAGE is not simply a sequential engineering pipeline, but a compositional mapping in which semantic expansion, structure-aware token filtering, contextual encoding, and graph propagation play distinct roles.
• Time and Space Complexity. We now summarize the main computational costs of STAGE.

• Stage I: Offline semantic augmentation. Let $C_{\mathrm{LLM}}$ denote the average generation cost per node. Since Stage I is executed once for each node and is detached from the downstream training loop, its total complexity is

  $$O\left(\right|\mathcal{V}|·C_{\mathrm{LLM}}).$$

  (18)
• Stage II: Structure-aware representation learning. For each target node v, let $R_v$ denote the number of raw candidate tokens collected from the sampled subgraph. The graph-conditioned selector scores these tokens and constructs a reduced sequence of length ${\tilde{L}}_v\le L_{max}$. The token scoring and sequence construction costs therefore scale with the candidate context size, which may increase with walk depth. In contrast, the dominant self-attention cost of the PLM with respect to sequence length is bounded by the fixed token budget:

  $$O\left({\tilde{L}}_v^{2}d\right)\le O\left(L_{max}^{2}d\right),$$

  (19)

  where d is the hidden dimension of the PLM. Thus, compared with an unconstrained concatenation strategy whose self-attention cost would scale as $O\left(R_v^{2}d\right)$, STAGE replaces the dependence on the full raw neighborhood text with a dependence on the fixed budget $L_{max}$.
• GNN propagation. After PLM encoding, the graph propagation stage follows the complexity of a standard message-passing GNN. For a K-layer GNN with hidden dimension $d_h$, the propagation cost is proportional to the graph size and can be written in the usual form as

  $$O\left(K\right|\mathcal{E}\left|d_h\right).$$

  (20)

Overall, STAGE does not remove all cost associated with larger structural exploration. Instead, it bounds the PLM-side sequence cost while allowing the preprocessing overhead—including graph construction, token scoring, and sequence assembly—to absorb most of the growth in structural context. This analytical view is consistent with the empirical scalability trends reported in Section 4.5.

4. Results

In this section, we evaluate STAGE and answer the following research questions:

• RQ1: Can STAGE outperform existing baselines across diverse text-attributed graphs?
• RQ2: How do semantic injection and token retention strategies affect the performance of STAGE?
• RQ3: How sensitive is STAGE to the balance between random walk depth and a fixed token budget?
• RQ4: How does STAGE scale in preprocessing cost, PLM-side cost, and downstream performance as the structural context grows?

4.1. Implementation Details

• Datasets and Baselines. We evaluate STAGE on seven benchmark datasets from citation, social, and e-commerce domains. Our experimental protocol follows the settings used in GraphBridge [10] and TAPE [13]. We compare STAGE with 14 baselines from three groups: traditional methods (e.g., GCN, GraphSAGE, BERT, and RoBERTa), joint text–structure methods (e.g., GLEM, ENGINE, and GraphBridge), and LLM-augmented methods (e.g., TAPE and KEA). Dataset statistics are reported in

  Table 2. Statistics of the seven datasets used in our experiments.

  Dataset	# Nodes	# Edges	# Avg. Tokens	# Classes
  Cora	2708	5429	194	7
  WikiCS	11,701	215,863	545	10
  CiteSeer	3186	4277	196	6
  ArXiv-2023	46,198	78,543	253	40
  Ele-Photo	48,362	500,928	185	12
  OGBN-Products (subset)	54,025	74,420	163	47
  OGBN-ArXiv	169,343	1,166,243	231	40

  # denotes the number of the corresponding item.

  . For all seven datasets, we use the same dataset versions and train/validation/test splits as GraphBridge [10] (following TAPE [13] where applicable), without introducing any additional re-splitting. Following GraphBridge [10], Table 2 reports the number of nodes, edges, classes, and the average number of tokens per node.
• Experimental Setup. All experiments were conducted on a single NVIDIA GeForce RTX 4090 GPU with 24 GB VRAM. In Stage I, we use GPT-3.5-turbo to generate the offline semantic explanations. To keep the semantic augmentation procedure consistent across datasets with different text styles, we use a unified general-text extraction prompt for all datasets. The generated outputs are post-processed into explanatory text, and malformed or empty generations are filtered before concatenation. In Stage II, RoBERTa-base is adopted as the PLM backbone and GraphSAGE as the GNN aggregator. The hyperparameters of STAGE are tuned separately for each dataset based on validation performance. We tune the reduction module, the PLM fine-tuning stage, and the GNN stage independently, and report the final test results using the configuration that achieves the best validation accuracy for each dataset. When prior work provides established search ranges, we use them as references for the tuning procedure. Unless otherwise specified, we report mean accuracy and standard deviation over ten runs with different random seeds.

4.2. Overall Comparison (RQ1)

Table 3. Node classification accuracy (%) comparisons. The best, second-best, and third-best results are highlighted in bold and red, underlined and blue, and italicized and green, respectively. “–” indicates out-of-memory (OOM) or incompatibility. Traditional methods include GNNs and PLMs; joint methods model structure and text together; LLM-Aug. methods utilize generative augmentation.

Category	Method	Cora	WikiCS	CiteSeer	ArXiv-2023	Ele-Photo	OGBN-Prod.	OGBN-ArXiv
Traditional (GNN-based)	MLP	76.12 ± 1.51	68.11 ± 0.76	70.28 ± 1.13	65.41 ± 0.16	62.21 ± 0.17	58.11 ± 0.23	62.57 ± 0.11
	GCN	88.12 ± 1.13	76.82 ± 0.62	71.98 ± 1.32	66.99 ± 0.19	80.11 ± 0.09	69.84 ± 0.52	70.78 ± 0.10
	GraphSAGE	87.60 ± 1.40	76.65 ± 0.84	72.44 ± 1.11	68.76 ± 0.51	79.79 ± 0.23	70.64 ± 0.20	71.72 ± 0.21
	GAT	85.13 ± 0.95	77.04 ± 0.55	72.73 ± 1.18	67.61 ± 0.24	80.38 ± 0.37	69.70 ± 0.25	70.85 ± 0.17
	NodeFormer	88.48 ± 0.33	75.47 ± 0.46	75.74 ± 0.54	67.44 ± 0.42	77.30 ± 0.06	67.26 ± 0.71	69.60 ± 0.08
Traditional (PLM-based)	BERT	79.70 ± 1.70	78.13 ± 0.63	71.92 ± 1.07	77.15 ± 0.09	68.79 ± 0.11	76.23 ± 0.19	72.75 ± 0.09
	RoBERTa-base	78.49 ± 1.36	76.91 ± 0.69	71.66 ± 1.18	77.33 ± 0.16	69.12 ± 0.15	76.01 ± 0.14	72.51 ± 0.03
	RoBERTa-large	79.79 ± 1.31	77.79 ± 0.89	72.26 ± 1.80	77.70 ± 0.35	71.22 ± 0.09	76.29 ± 0.27	73.20 ± 0.13
Joint Structure–Text	GLEM	87.61 ± 0.19	78.11 ± 0.61	77.51 ± 0.63	79.18 ± 0.21	81.47 ± 0.52	76.15 ± 0.32	74.46 ± 0.27
	SimTeG	86.85 ± 1.81	79.77 ± 0.68	78.69 ± 1.12	79.31 ± 0.49	81.61 ± 0.18	76.46 ± 0.55	74.31 ± 0.14
	ENGINE	87.56 ± 1.48	77.97 ± 0.94	76.79 ± 1.38	78.34 ± 0.15	80.50 ± 0.33	77.80 ± 1.20	73.59 ± 0.14
	GraphBridge	92.14 ± 1.03	80.59 ± 0.47	85.32 ± 1.39	84.07 ± 0.34	83.84 ± 0.07	79.80 ± 0.19	74.89 ± 0.23
LLM- Augmented	TAPE	87.82 ± 0.91	–	–	80.11 ± 0.20	–	79.46 ± 0.11	74.66 ± 0.07
	KEA	90.44 ± 1.62	80.48 ± 0.31	75.55 ± 1.24	85.23 ± 0.47	78.27 ± 0.21	76.99 ± 0.37	73.40 ± 0.19
Ours	STAGE	92.80 ± 1.65	81.99 ± 0.33	86.18 ± 1.36	85.66 ± 0.48	84.01 ± 0.22	81.08 ± 0.11	75.47 ± 0.16

Bold: best; Underlined with blue: second-best; Italicized with green: third-best. Gray background highlights our proposed method.

reports the node classification accuracy on all seven datasets. Several patterns are worth noting.

• Observation 1: Both text semantics and graph structure matter for TAG learning. The results show that models relying mainly on one source of information are often limited. On ArXiv-2023, for example, structure-oriented methods such as GCN remain well below the strongest results, suggesting that shallow initial features are not sufficient for this dataset. In contrast, text-based PLMs capture richer semantics, but they do not fully exploit graph connectivity. This gap is consistent with the nature of TAGs, where useful signals come from both node text and graph structure rather than either one alone.
• Observation 2: Semantic enrichment is more useful when it is followed by structure-aware learning. Generative baselines such as TAPE enrich node attributes with additional semantic content, but the generated information is typically produced for each node independently. Joint models such as GraphBridge make stronger use of structural information, yet they are still constrained by the quality of the original text. STAGE combines these two steps in sequence: it first enriches the node text offline and then learns graph-aware representations from the enriched attributes. The comparison suggests that this separation is effective in practice.
• Observation 3: STAGE achieves the strongest overall results under our evaluation protocol. Across the seven benchmarks, STAGE achieves the best overall results among the methods compared in our experiments. The gains are visible across all seven datasets, although their magnitude varies by dataset, with the largest margins on WikiCS and CiteSeer. In absolute terms, STAGE reaches 92.80% on Cora and 85.66% on ArXiv-2023, indicating that the proposed design remains effective across graphs with different scales and characteristics.

4.3. Component Ablation and Diagnostic Analysis (RQ2)

We conducted a controlled component ablation study to examine the individual contributions of the two main components of STAGE: offline semantic injection and graph-conditioned token reduction. Since the benefit of token reduction is expected to become more evident when the sampled structural context is larger, we report the complete ablation on OGBN-Products under a larger walk depth. This setting creates a larger and noisier candidate token pool, making it suitable for evaluating whether graph-conditioned token selection provides additional benefit over simple retention strategies.

All variants use the same train/validation/test split, PLM backbone, GNN architecture, token budget, walk depth, and downstream training configuration. Only the corresponding component is removed or replaced. Specifically, STAGE w/o Semantic Injection removes the LLM-generated explanatory text and uses only the original node text; STAGE w/o Token Selector replaces the learned graph-conditioned selector with head truncation; and STAGE w/ Random Token Selection randomly retains tokens under the same token budget.

Table 4. Complete component ablation on OGBN-Products under a larger walk depth. Accuracy (%) is reported as mean ± standard deviation over ten runs.

Variant	Main Difference	Accuracy (%)
STAGE w/o Semantic Injection	Original text only	78.59 ± 0.22
STAGE w/o Token Selector	Head truncation	79.66 ± 0.17
STAGE w/ Random Token Selection	Random retention	80.32 ± 0.09
Full STAGE	Graph-conditioned selector	81.08 ± 0.11

Bold value indicates the best performance.

shows that removing semantic injection leads to the lowest performance, indicating that the offline LLM-generated explanations provide useful semantic context. Replacing the learned selector with head truncation improves over the non-enriched variant, but remains lower than the full model. Random token selection performs better than head truncation in this setting, suggesting that fixed positional retention may become less reliable when more neighbor tokens are included. The full STAGE model achieves the best accuracy, improving over the no-semantic variant by 2.49 percentage points, over head truncation by 1.42 percentage points, and over random selection by 0.76 percentage points. These results support the joint contribution of semantic enrichment and graph-conditioned token reduction under larger structural contexts.

4.4. Parameter Sensitivity (RQ3)

We now study how STAGE behaves under different combinations of random walk length L and a fixed global token budget of 512. This setting reflects the trade-off between using more structural context and preserving more tokens from the central node.

• Effect of graph characteristics. Figure 3 shows that the preferred walk length depends on the dataset. On Cora, performance improves as L becomes larger and reaches its best value at $L=32$, suggesting that broader structural context is useful in this graph. On ArXiv-2023, the best results appear at smaller values such as $L\in \{0,8\}$. A likely reason is that, once the node text has already been enriched by the LLM, allocating too much of the fixed token budget to distant neighbors can remove useful local content from the target node itself.
• Interpreting the fluctuations. The performance curves are not perfectly monotonic. This is expected because the hard truncation step is discrete: when L changes, the token allocation across the target node and its neighbors can shift abruptly. As a result, some informative local tokens may be replaced with neighbor tokens at certain settings. Even so, the overall trends remain stable enough to show that the selector can work under a fixed budget across graphs with different properties.

4.5. Scalability Analysis (RQ4)

We examine how the computational cost of STAGE changes as the structural context grows under a fixed global token budget. Since the selector enforces an upper bound before PLM encoding, increasing the walk depth does not allow the PLM input length to grow freely. Instead, it enlarges the sampled structural view and the raw candidate context, while the final PLM input remains bounded by the token budget.

• Resource profile under increasing structural context. 

  Table 5. Scalability statistics of STAGE on OGBN-ArXiv under different walk depths. The reduction stage is trained once offline and shared across all walk-depth settings.

  Walk Steps	Avg. Nodes	Avg. Raw Tokens	Avg. Final Tokens	Graph Build (s)	Seq. Build (s)	Token Sel. (s)	LM Epoch (s)	LM Mem. (GB)	GNN Acc. (%)
  0	1.00	336.46	337.38	27.51	49.08	15.74	1121.81	11.78	76.07 ± 0.19
  8	3.13	1066.49	486.49	52.17	87.49	35.43	1132.67	11.78	75.26 ± 0.19
  16	4.91	1678.03	503.46	59.07	98.51	44.73	1128.82	11.78	75.09 ± 0.15
  24	6.48	2216.42	506.47	63.99	121.53	57.74	1131.99	11.78	75.32 ± 0.14
  32	7.87	2692.87	507.01	61.42	128.88	67.49	1334.78	11.78	75.16 ± 0.17

  reports the scalability statistics of STAGE on OGBN-ArXiv under different walk depths. As the walk depth increases from 0 to 32, the sampled subgraph becomes larger, with the average number of nodes per graph increasing from 1.00 to 7.87, and the average raw candidate tokens per graph increasing from 336.46 to 2692.87. This confirms that larger walk depths indeed expose the model to substantially broader structural context.

At the same time, the final PLM input remains bounded. Although the raw candidate context grows by nearly eight times, the average final tokens per graph increase only from 337.38 to 507.01, approaching but not exceeding the fixed token budget. This indicates that the graph-conditioned reduction stage effectively constrains the input length before PLM encoding.

The additional cost introduced by larger walk depths is mainly reflected in preprocessing. Graph construction time, sequence build time, and token selection time all increase as the sampled context grows. In contrast, the peak GPU memory used by the LM stage remains stable at 11.78 GB across all tested settings. The LM-side epoch time is also nearly stable for moderate walk depths, and only increases at the largest settings. These results suggest that STAGE does not eliminate all costs of larger structural exploration, but shifts the dominant overhead from PLM self-attention to the graph construction and sequence assembly stages.

From the downstream performance perspective, broader structural context does not improve accuracy on OGBN-ArXiv under this bounded-input regime. The best GNN result is achieved at walk depth 0, while deeper walks yield slightly lower but broadly similar performance. This suggests that, for this dataset, allocating more of the fixed budget to distant neighbors may reduce the amount of useful local semantic content preserved for the target node.

The reduction module is trained once offline and kept fixed across the walk-depth settings; its one-time training cost is 10,642.37 s with a peak GPU memory of 5.64 GB.

5. Discussion

5.1. Discussion on LLM Choices and Prompt Design

Since STAGE uses an LLM only in the offline generation stage, the framework is not tied to gradient-based fine-tuning of a particular LLM. In our experiments, GPT-3.5-turbo is used to generate explanatory text for all datasets under a unified extraction prompt. This design keeps the semantic augmentation stage detached from downstream PLM and GNN training, and the generated explanations are treated as additional node attributes rather than as direct predictions.

Nevertheless, the choice of the generative model and the prompt template may still affect the quality of the generated explanations. For example, a stronger LLM may produce more accurate definitions for specialized terms, while a weaker model may generate incomplete or noisy descriptions. Similarly, prompts that are too domain-specific may improve one dataset but reduce generality across heterogeneous TAGs. For this reason, we adopted a simple and label-free extraction prompt in the main experiments, which asks the LLM to identify technical terms and provide short descriptions without using class labels or downstream supervision.

We do not claim that STAGE is insensitive to all possible LLM backbones or prompt formulations. Instead, the current design aims to reduce this dependency by using the LLM as an offline semantic enhancer rather than an end-to-end predictor. Conducting a more systematic evaluation of different LLM backbones, decoding settings, and prompt templates remains an important direction for future work.

5.2. Why Graph-Conditioned Token Reduction Matters

Section 4.5 shows that token reduction helps control the cost of PLM encoding as the structural context grows. Beyond efficiency, this step also matters for representation quality.

A natural alternative would be to encode each node’s enriched text into a single dense vector first (for example, with the [CLS] token) and then apply graph propagation on top of these node-level embeddings. This late-fusion design is simpler, but it compresses the enriched text before the model has a chance to decide which parts of the structural context are worth preserving. In settings where node text is short, technical, and semantically uneven, this compression can remove useful lexical cues too early.

This issue becomes more pronounced in graphs with heterophily or weak graph signal, where neighboring text may be topically misaligned with the target node [38,39,40]. A related phenomenon has also been observed in long-context NLP, where irrelevant context can distract attention-based models [41].

STAGE instead applies token selection before PLM encoding. The selector operates under graph context, so the retained tokens are not chosen only by local redundancy within a sequence. They are chosen with respect to the structural neighborhood from which the sequence was constructed. This gives the encoder access to a more targeted input: it can preserve informative local content together with structurally relevant neighbor tokens, while still staying within a fixed token budget.

In this sense, graph-conditioned token reduction is not only a way to save memory. It is also part of how STAGE combines semantic enrichment with structure-aware learning. The offline generation stage adds missing context to node text, and the selector determines which parts of that enriched context should remain available when structural information is introduced.

5.3. Failure Case Analysis and Limitations

Although STAGE performs well on the evaluated benchmarks, it still has several limitations.

First, the quality of Stage I depends on the generated explanations. If the LLM provides incomplete, ambiguous, or misleading descriptions for key terms, the enrichment stage may introduce noise rather than useful context. This issue is more likely in highly specialized domains where the relevant terminology is uncommon or rapidly evolving.

Second, the token budget in Stage II creates an explicit trade-off. A larger structural context allows the model to consider more neighbors, but it also leaves fewer tokens available for the target node under a fixed budget. As shown in Section 4.4, this trade-off does not have the same optimum on every dataset. The best setting depends on how much signal comes from local text versus broader graph context.

Third, STAGE is decoupled by design. This makes the training process easier to manage, but it also means that the offline semantic generation stage is not updated using downstream feedback. In some cases, a tighter interaction between generation and encoding could produce better task-specific explanations than a frozen offline stage.

These identified limitations are consistent with broader observations on hallucination in language generation and graph-aware LLM reasoning [42,43]. One possible extension is to ground the generation stage with retrieval-augmented generation [44], so that the offline explanations are supported by external evidence rather than the parametric knowledge of the LLM alone.

Another limitation is that the current framework is evaluated on static TAG benchmarks. In dynamic or time-varying TAGs, node text, edge text, and graph topology may evolve over time, which would require incremental semantic enrichment and token selection rather than one-time offline generation. Extending STAGE to dynamic TAGs is therefore an important direction for future work.

In addition, this study focuses on semi-supervised node classification, which is the standard setting for the evaluated TAG benchmarks. Whether the enriched representations produced by STAGE can generalize to auxiliary tasks such as link prediction or graph classification remains to be further investigated.

6. Conclusions

This paper presents STAGE, a decoupled framework for representation learning on text-attributed graphs. The framework uses a frozen LLM offline to generate explanatory text that supplements sparse node attributes, and then learns structure-aware representations from the enriched text with a graph-conditioned token reduction module. This design makes it possible to introduce additional semantic context without letting the PLM input length grow freely with the structural neighborhood.

Across seven benchmark datasets, STAGE shows consistent improvements over strong baselines under our evaluation protocol. The results suggest that separating semantic enrichment from graph-based propagation is a practical way to balance representation quality and computational cost in TAG learning.

There are several directions for future work. One is to study whether tighter interaction between the semantic augmentation stage and the downstream encoder can further improve performance. Another is to examine how stable the framework remains under stronger domain shifts or more challenging low-resource settings.