BAF–FedLLM: Behavior-Aware Federated Modeling of Student Actions via Privacy-Preserving Large Language Model

Abstract

Analyzing fine-grained student actions across institutions can drive timely feedback, early warning, and personalized support, yet it is constrained by privacy regulations, heterogeneous curricula, and non-IID behavior logs. This paper introduces BAF–FedLLM, a behavior-aware federated modeling framework that adapts large language models to next-action and outcome prediction without centralizing student data. The key idea is to treat multichannel interaction streams as semantically typed action tokens linked by a learned ActionGraph, and to align their temporal structure with an LLM through behavior prompts that inject domain context (task, resource, pedagogy, and affordance cues). We propose three novel components: (i) BP–FIT, a behavior-prompted federated instruction tuning scheme that trains low-rank adapters locally and aggregates them with secure masking and Rényi–DP accounting to ensure client-level privacy; (ii) ProtoAlign, a cross-client prototype contrastive objective that shares only noisy class-conditional anchors via secure aggregation to mitigate drift under non-IID partitions; and (iii) CBR, a causal behavior regularizer that penalizes intervention-sensitive shortcuts by enforcing invariance of predicted risks across detected instructional regimes. We further derive convergence guarantees for federated instruction tuning with noisy, partial participation and provide end-to-end privacy bounds. On three public education datasets (EdNet, ASSISTments, and OULAD) with institution-level partitions, BAF–FedLLM improves next-action AUC by 4.2–7.1% over strong federated baselines while reducing expected calibration error by up to 28% and communication by $5\times$ through adapter sparsity, under a typical privacy budget of $\epsilon \approx 1.7$ at $\delta ={10}^{-5}$. These results indicate that behavior-aware prompting and prototype alignment make LLMs practical for privacy-preserving student action analysis at scale, offering a principled path to deployable, regulation-compliant analytics across diverse learning ecosystems.

1. Introduction

Timely, accurate analysis of student actions can drive feedback, early warnings, and equitable support across learning platforms. Clickstream logs from homework systems, MOOCs, and institutional LMSs record not only correctness, but also fine-grained choices such as revisiting materials, requesting hints, pausing on items, or engaging with policies and forums. Models that anticipate these actions and estimate evolving risk can support instructors with at-scale triage tools, surface misconceptions earlier, and help institutions monitor equity gaps across cohorts and curricula. However, the institutions that most need such analytics—schools, districts, and universities—often cannot centralize logs because of privacy regulations (e.g., FERPA, GDPR) and local governance constraints, especially when data span multiple vendors and jurisdictions.

Federated learning offers a natural alternative by keeping logs on institutional servers and exchanging only model updates. Yet directly applying standard federated optimization to educational data is problematic. Student behavior is highly heterogeneous: curricula, pedagogy, and demographics vary across institutions, inducing severe non-IID client distributions. This heterogeneity can slow or destabilize training, lead to brittle decision boundaries that overfit dominant regimes, and degrade model calibration in minority groups. At the same time, modern interaction streams are semantically rich: items and videos are accompanied by textual stems and solutions, hints and rubrics, forum posts, syllabus constraints, and institutional policies. Such multimodal, structured context is not well captured by classical tabular models, but is well-suited to language models that can reason over text if adapted efficiently and privately.

We pursue a federated approach that explicitly leverages this semantic structure while respecting institutional privacy. Our starting point is to treat multichannel interactions as action tokens rendered from typed events and their textual context, and to align their temporal and structural regularities with a frozen backbone language model. We introduce BAF–FedLLM, a behavior-aware federated framework that trains only lightweight adapters under client-level differential privacy (DP) while never sharing raw logs. Institutions locally instantiate an action tokenizer and prompt builder that map clickstream segments and neighboring educational structure into natural language snippets, which are then processed by a shared large language model (LLM). This design decouples domain-specific logging schemas from the backbone, permitting deployment across institutions with varying platforms while amortizing the cost of pretraining.

The framework is built around three modeling components and a privacy-preserving training protocol. First, an ActionGraph distills relational and temporal cues—skills, prerequisites, course units, policies, and other structural metadata—into compact behavior prompts. These prompts summarize the ego-net around a student’s current context as short clauses prepended to the action sequence, steering the frozen backbone toward pedagogically salient dependencies (e.g., prerequisite chains or unit transitions) without modifying its core parameters. Second, ProtoAlign combats cross-client drift by privately aggregating class-conditional prototype anchors via secure aggregation. Each client computes clipped per-class prototypes in a projected representation space and shares only noised, masked sums; the server forms global anchors and redistributes them, enabling a supervised contrastive loss that pulls local representations toward these shared anchors. Third, CBR (causal behavior regularization) mitigates regime-specific shortcuts in the risk head by promoting both calibration and invariance across instructional regimes induced from behavior features (e.g., course type, pacing mode, or assessment style). CBR introduces dispersion and first-order invariance penalties that reduce sensitivity to spurious regime-specific cues while preserving genuine difficulty signals.

Within this architecture, we focus on two complementary tasks: next-action prediction (NAP), which models short-term engagement and mastery trajectories, and risk prediction (RP), which estimates the probability of adverse outcomes such as failure or withdrawal. NAP is framed as a sequence modeling problem over action tokens, while RP operates on pooled representations and is subject to stricter calibration and robustness requirements. Both heads share the same adapter-augmented LLM backbone, but they interact differently with ActionGraph prompts, ProtoAlign, and CBR. Federated training proceeds in rounds: clients perform local adapter and head updates on their own logs, apply clipping and Gaussian noise to ensure client-level DP, and communicate only aggregate statistics and masked prototype sums to an honest-but-curious server.

Our contributions are threefold.

• Behavior-prompted federated instruction tuning (BP–FIT). We design a tokenizer $\psi$ and prompt builder $\mathsf{\Pi }$ that turn typed actions and ActionGraph context into short natural language clauses prepended to the sequence. We update only low-rank adapters $\varphi$ of a frozen backbone ${\mathrm{LLM}}_{\theta }$ with a stochastic prompt-consistency objective, enabling parameter- and communication-efficient instruction tuning under client-level DP.
• ProtoAlign for non-IID data. Each client computes clipped per-class prototypes in a projected space $\mathbf{z}$ and shares only noisy, masked sums via secure aggregation. The server forms global anchors $\left\{{\mathit{\pi }}_k\right\}$, and a supervised contrastive loss aligns local representations, reducing cross-client label and feature shift with minimal extra communication.
• CBR for regime-invariant risk. For the RP head, we introduce a dispersion penalty that equalizes mean predicted risk across behavioral regimes and a first-order invariance term that discourages regime-specific shortcuts, improving robustness and calibration without explicit causal graphs.

We focus on settings where the privacy unit is an entire client (institution); the server is honest-but-curious and observes only securely aggregated, noised statistics. We allow partial participation and heterogeneous clients, and we freeze ${\mathrm{LLM}}_{\theta }$ while training adapters $\varphi$ and lightweight task heads $f_{\theta ,\varphi }$ shared across tasks. The remainder of the paper reviews related work, formalizes the problem and notation (including ActionGraph prompts, prototype construction, and DP accounting units), presents BP–FIT, ProtoAlign, and CBR together with convergence and privacy results, and then reports experiments, ablations, and privacy–utility and communication analyses before concluding with limitations and future directions.

2. Related Work

This study sits at the intersection of federated optimization under non-IID behavior, privacy-preserving learning, parameter-efficient adaptation of large language models (LLMs), representation alignment, and student modeling. We briefly review the most relevant lines and position BAF–FedLLM with respect to them.

2.1. Federated Learning Under Heterogeneity

Federated averaging (FedAvg) popularized learning from decentralized data without centralization [1], and subsequent surveys crystallized challenges and open problems, particularly statistical heterogeneity and partial participation [2]. Empirical analyses showed that non-identical client distributions degrade both optimization and generalization [3], motivating methods that explicitly control client drift or personalize solutions. Representative approaches include proximal regularization (FedProx) to stabilize local updates [4] and control variates to correct client drift (SCAFFOLD) [5], adaptive server optimizers (FedOpt) [6], and distributionally robust or agnostic formulations that hedge against unknown client mixtures [7]. Personalized FL further tailors models to clients via meta-learning (Per–FedAvg) [8]. In our framework, ProtoAlign reduces drift by aligning shared representations ${\mathbf{z}}_{u,t}$ to privatized global anchors, complementing these optimizer-level controls and preserving a single deployable model $f_{\theta ,\varphi }$ for cross-client interoperability.

Protecting client updates against curious servers typically combines cryptographic secure aggregation with algorithmic differential privacy (DP). Practical protocols sum client updates without revealing individual contributions [9]. On the algorithmic side, DP–SGD injects calibrated Gaussian noise and clips sensitivity to bound privacy loss [10]. Rényi differential privacy (RDP) provides tighter accounting under composition [11], and subsampled RDP/moments accountants translate participation sampling into improved guarantees [12]. Our client-level privacy analysis composes two Gaussian mechanisms—adapter updates and prototype messages—under secure aggregation, yielding end-to-end $(\epsilon ,\delta )$ bounds reported in Section 5.

2.2. Parameter-Efficient Adaptation and Instruction Tuning

For adapting large pretrained models with limited communication and memory, parameter-efficient transfer learning introduces small task-specific modules while freezing the backbone. Adapter layers insert bottleneck transformations at attention/MLP blocks [13]; low-rank adaptation further reduces trainable degrees of freedom (LoRA) [14]. Prompt tuning attaches learned textual soft prompts to steer the model with minimal parameters [15], while instruction tuning improves generalization across tasks and formats [16,17]. Underlying all of these advances is the Transformer architecture [18]. Our BP–FIT exploits these ideas in the federated regime: clients fine-tune only low-rank adapters $\varphi$ and synthesize compact behavior prompts from ActionGraph descriptors to align interaction semantics with the frozen backbone ${\mathrm{LLM}}_{\theta }$, yielding strong utility under tight client-level DP.

2.3. Federated Large Language Models and DP–LLMs

Recent work has begun to study federated fine-tuning of large language models (LLMs) with parameter-efficient adaptation. FedPETuning provides an early empirical study of parameter-efficient tuning methods for pretrained language models in FL [19,20]. Building on this, methods such as SLoRA, FlexLoRA, and DP–LoRA integrate low-rank adapters or related PEFT modules with federated optimization to cope with heterogeneous data and resource constraints across clients while keeping the backbone frozen, and in some cases also enforcing client- or user-level privacy [21,22,23]. In parallel, differentially private fine-tuning of language models has been explored in centralized and federated settings, combining DP–SGD with adapters or LoRA layers to obtain usable $(\epsilon ,\delta )$ guarantees at moderate model scales [24]. Surveys on federated LLMs and DP–LLMs highlight open challenges in data heterogeneity, communication efficiency, and privacy–utility trade-offs [25,26].

BAF–FedLLM is complementary to these lines of work: we target cross-institution student action modeling with client-level privacy, and we integrate behavior prompts, prototype alignment, and causal behavioral regularization into a unified framework with convergence and privacy analyses. In particular, ProtoAlign and CBR are designed to directly address non-IID behavior logs and regime-dependent shortcuts in educational contexts, rather than generic text classification or next-token prediction benchmarks, and our experiments focus on realistic institution-level partitions of EdNet, ASSISTments, and OULAD.

2.4. Representation Alignment, Prototypes, and Invariance

Prototype-based learning summarizes classes with centroid embeddings, a simple and effective inductive bias in few-shot regimes [27]. Contrastive objectives improve class separation and calibration by pulling together same-label points while pushing apart different ones [28]. We integrate these principles as ProtoAlign, which privately aggregates masked per-client prototypes into global anchors $\left\{{\mathit{\pi }}_k\right\}$ used in an InfoNCE-style loss, thereby mitigating cross-client drift without sharing raw features. Complementary to alignment, causal invariance seeks predictors whose optimality is stable across environments [29]. Our CBR regularizer penalizes regime-specific risk dispersion and first-order sensitivity, encouraging ${\mathbf{h}}_{u,t}$ to encode behaviorally stable cues rather than spurious instructional shortcuts. Unlike personalized meta-learning approaches [8], we retain a single global adapted model, simplifying deployment in cross-institution analytics.

2.5. Student Modeling and Educational Datasets

Knowledge tracing models estimate evolving mastery and predict future actions/outcomes; deep variants capture long-range dependencies but often centralize data [30]. Widely used platforms and datasets include ASSISTments [31], the Open University Learning Analytics Dataset (OULAD) [32], and EdNet [33]. Risk-oriented applications (e.g., early warning) demand well-calibrated probabilities [34]. Our study partitions these corpora at institution or module granularity and demonstrates that combining behavior prompts with prototype alignment and invariance improves both accuracy and calibration under $(\epsilon ,\delta )$–DP, aligning with regulatory and ethical constraints while retaining cross-site utility.

In general, prior work established optimization controls for non-IID FL [1,2,3,4,5,6,7], strong privacy accounting and secure aggregation [9,10,11,12], and parameter-efficient adaptation of LLMs [13,14,15,16,17,18]. In contrast, BAF–FedLLM (i) constructs behavior prompts from an ActionGraph to inject semantically structured context into ${\mathrm{LLM}}_{\theta }$ locally, (ii) performs prototype alignment via privatized global anchors to counter heterogeneity without exposing client features, and (iii) enforces regime-level invariance to reduce spurious shortcuts in risk prediction. This combination targets the practical constraints of cross-institution student action analysis that are insufficiently addressed when any one component is used in isolation.

3. Preliminaries

We formalize behavior-aware federated modeling for student actions. Unless stated otherwise, lowercase letters denote scalars, bold lowercase vectors, and calligraphic uppercase sets. For an integer n, $\left[n\right]=\{1,\dots ,n\}$. For a vector $\mathbf{v}$, ${\parallel \mathbf{v}\parallel }_2$ is its Euclidean norm. $\mathrm{CE}(·,·)$ denotes cross-entropy, $\sigma (·)$ the logistic function, and $\mathrm{KL}(·\parallel ·)$ the Kullback–Leibler divergence.

3.1. Problem Setting and Notation

We consider C institutions (clients) indexed by $c\in \left[C\right]$. Client c holds private interaction logs ${\mathcal{D}}_c$ that never leave the client. Each log is a sequence for a learner u:

$${\mathbf{s}}_u=\left((a_1,t_1,{\kappa }_1),\dots ,(a_T,t_T,{\kappa }_T)\right),$$

where $a_t\in \mathcal{A}$ is a semantically typed action (e.g., view_video, solve_item, hint), $t_t$ a timestamp, and ${\kappa }_t$ contextual metadata (course, resource, device, instructor policy). Let $\psi :\mathcal{A}\times \mathcal{K}\to {\mathsf{\Sigma }}^{*}$ be a tokenizer that renders an action and its salient context as a short textual action token over vocabulary $\mathsf{\Sigma }$. For a prefix ${\mathbf{s}}_{u,1:t}$, we write ${\mathbf{x}}_{u,1:t}=\psi \left({\mathbf{s}}_{u,1:t}\right)$.

We study two prediction tasks used throughout:

$$\begin{array}{cc}\hfill \mathbf{Next}-\mathbf{Action}\mathbf{Prediction} \left(\mathbf{NAP}\right):& y_{u,t}^{\mathrm{act}}=a_{t+1}\in {\mathcal{A}}_{\mathrm{pred}},\hfill \\ \hfill \mathbf{Risk}\mathbf{Prediction} \left(\mathbf{RP}\right):& y_{u,t}^{\mathrm{risk}}\in \{0,1\}\mathrm{for}\mathrm{a}\mathrm{horizon}H(\mathrm{e}.\mathrm{g}.,\mathrm{dropout}\mathrm{within}H).\hfill \end{array}$$

A model $f_{\theta ,\varphi }$ maps a prompted token prefix to a distribution over targets. We decompose f into a frozen backbone LLM with parameters $\theta$ and trainable adapters $\varphi$. The same notation will be used in Section 4.

3.2. Student Action Tokens and the ActionGraph

To inject structural cues beyond raw text, we maintain a typed ActionGraph

$$\mathcal{G}=(\mathcal{V},\mathcal{E},\tau ),\mathcal{V}=\mathcal{A}\cup \mathcal{R}\cup \mathcal{O},$$

where nodes include actions $\mathcal{A}$, resources $\mathcal{R}$ (items, pages, videos), and organizational entities $\mathcal{O}$ (course, unit, instructor). Each edge $e=(v\to v^{\prime },\ell )\in \mathcal{E}$ is typed by $\ell \in \{\mathtt{precedes}, \mathtt{requires}, \mathtt{same}\_\mathtt{skill}, \mathtt{same}\_\mathtt{unit}, \mathtt{prereq}, \mathtt{cohort}\}$ with a time-decay $\tau \left(e\right)\in {\mathbb{R}}_{\ge 0}$. For a prefix ${\mathbf{s}}_{u,1:t}$, we form a small ego-net ${\mathcal{G}}_{u,t}$ around recently visited nodes and summarize it as compact behavior descriptors (counts, decays, typed degrees). The behavior prompt builder $\mathsf{\Pi }({\mathbf{s}}_{u,1:t}, {\mathcal{G}}_{u,t})$ renders these descriptors as natural language clauses prepended to the tokenized prefix:

$${\mathbf{b}}_{u,t}=\mathsf{\Pi }({\mathbf{s}}_{u,1:t},{\mathcal{G}}_{u,t})\in {\mathsf{\Sigma }}^{*},{\tilde{\mathbf{x}}}_{u,1:t}=[{\mathbf{b}}_{u,t};{\mathbf{x}}_{u,1:t}].$$

This construction will be used by BP–FIT in Section 4.

We note that our CBR regularizer penalizes regime-specific risk dispersion and first-order sensitivity, encouraging ${\mathbf{h}}_{u,t}$ to encode behaviorally stable cues rather than spurious instructional shortcuts. Unlike personalized meta-learning approaches [8], we retain a single global adapted model, simplifying deployment in cross-institution analytics. Compared with prototype-based federated methods such as FedProto and supervised prototype alignment, ProtoAlign uses privatized class anchors computed in a projected LLM representation space, and aggregates only clipped, noised sufficient statistics under secure aggregation. This design allows us to derive an explicit alignment gain term in the convergence bound (Section 5), rather than treating prototypes as a heuristic regularizer. Likewise, CBR differs from invariant risk minimization and environment-invariant prediction in that (i) it operates on automatically induced instructional regimes derived from behavior graphs instead of externally labeled environments, and (ii) it regularizes the risk head through a dispersion and first-order invariance penalty that is compatible with client-level differential privacy, without sharing environment labels or per-regime statistics across institutions.

3.3. Backbone LLM and Adapter Parameterization

Let ${\mathrm{LLM}}_{\theta }$ produce hidden states ${\mathbf{h}}_{u,t}\in {\mathbb{R}}^d$ at the final token of ${\tilde{\mathbf{x}}}_{u,1:t}$. We train lightweight low-rank adapters (e.g., LoRA-style) at selected attention/MLP layers: for a weight matrix $W\in {\mathbb{R}}^{m\times n}$, we add $\mathsf{\Delta }W=AB$ with $A\in {\mathbb{R}}^{m\times r},B\in {\mathbb{R}}^{r\times n},r\ll min\{m,n\}$. All trainable parameters across layers are collected in $\varphi$. Two task heads share the representation

$$p_{\varphi }^{\mathrm{act}}(·|{\tilde{\mathbf{x}}}_{u,1:t})=\mathrm{softmax}\left(W^{\mathrm{act}}{\mathbf{h}}_{u,t}\right),p_{\varphi }^{\mathrm{risk}}\left(1\right|{\tilde{\mathbf{x}}}_{u,1:t})=\sigma \left(w^{\top }{\mathbf{h}}_{u,t}+b\right).$$

We will refer to any intermediate representation used for alignment as ${\mathbf{z}}_{u,t}=P{\mathbf{h}}_{u,t}$ for a fixed projection P.

3.4. Federated Protocol and Privacy Model

Training proceeds in rounds $r\in \left[R\right]$ with a subset ${\mathcal{S}}_r\subseteq \left[C\right]$ of participating clients. Each selected client initializes ${\varphi }^{(r,0)}\leftarrow {\varphi }^{\left(r\right)}$, performs $\tau$ local steps to obtain ${\varphi }_c^{(r,\tau )}$, and returns an update $\mathsf{\Delta }{\varphi }_c^{\left(r\right)}={\varphi }_c^{(r,\tau )}-{\varphi }^{\left(r\right)}$. We use weighted aggregation with $w_c=|{\mathcal{D}}_c|/{\sum }_{i\in {\mathcal{S}}_r}\left|{\mathcal{D}}_i\right|$.

Local objectives. For client c with index set ${\mathsf{\Omega }}_c = \left\{(u,t)\right\}$

$$\begin{array}{cc}\hfill {\mathcal{L}}_c^{\mathrm{NAP}}\left(\varphi \right)& ={\displaystyle \frac{1}{|{\mathsf{\Omega }}_c|}}\sum _{(u,t)\in {\mathsf{\Omega }}_c}\mathrm{CE}\left(p_{\varphi }^{\mathrm{act}}(·|{\tilde{\mathbf{x}}}_{u,1:t}),y_{u,t}^{\mathrm{act}}\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathcal{L}}_c^{\mathrm{RP}}\left(\varphi \right)& ={\displaystyle \frac{1}{|{\mathsf{\Omega }}_c|}}\sum _{(u,t)\in {\mathsf{\Omega }}_c}\mathrm{CE}\left(p_{\varphi }^{\mathrm{risk}}(·|{\tilde{\mathbf{x}}}_{u,1:t}),y_{u,t}^{\mathrm{risk}}\right).\hfill \end{array}$$

(2)

Section 4 augments these with prototype alignment and causal regularization.

Threat model. The server is honest-but-curious: it orchestrates training but should not learn individual updates. Clients may be heterogeneous and partially participating. We assume a reliable secure aggregation primitive that reveals only aggregated statistics to the server.

Differential privacy at client level. Before secure aggregation, each update is clipped and noised:

$$\overline{\mathsf{\Delta }}{\varphi }_c^{\left(r\right)}=\mathrm{clip}\left(\mathsf{\Delta }{\varphi }_c^{\left(r\right)},S\right)+\mathcal{N}\left(0,{\sigma }^2{S}^{2}I\right).$$

Let q be the expected participation rate per round and let ${\mathcal{A}}_{\mathrm{RDP}}(q,\sigma ,R)$ denote a Rényi–DP accountant that composes sampling and Gaussian noise across R rounds to yield an $(\epsilon ,\delta )$ bound for any target $\delta \in (0,1)$. We report $\epsilon$ minimized over Renyi order $\alpha \ge 1$. The effective privacy unit is one client (institution). All results in Section 5 will specify $(\epsilon ,\delta )$, S, $\sigma$, q, and R.

3.5. Prototype Anchors and Invariance Cues

Heterogeneity induces client-specific drifts in representation space. For each class $k\in {\mathcal{A}}_{\mathrm{pred}}$, we define a local prototype on client c:

$${\mu }_{c,k}={\displaystyle \frac{1}{|B_{c,k}|}}\sum _{(u,t)\in B_{c,k}}{\mathbf{z}}_{u,t},B_{c,k}=\{(u,t)\in {\mathsf{\Omega }}_c:y_{u,t}^{\mathrm{act}}=k\}.$$

Clients share only masked prototypes through secure aggregation (masking/noise details in Section 4), yielding global anchors ${\mathit{\pi }}_k$ used for ProtoAlign.

Student behavior is also moderated by instructional regimes $s\in \mathcal{S}$ (e.g., assessment-heavy vs. mastery-based). We extract coarse regime indicators from ${\mathcal{G}}_{u,t}$ and metadata into a discrete variable $s_{u,t}$. The causal regularizer in Section 4 enforces stability of risks with respect to s by penalizing dispersion of conditional risks across regimes. For reference, define per-regime empirical risks on client c:

$${\widehat{\mathcal{R}}}_c\left(s\right)={\displaystyle \frac{1}{|{\mathsf{\Omega }}_c\left(s\right)|}}\sum _{(u,t)\in {\mathsf{\Omega }}_c\left(s\right)}\mathrm{CE}\left(p_{\varphi }^{\mathrm{risk}}(·|{\tilde{\mathbf{x}}}_{u,1:t}),y_{u,t}^{\mathrm{risk}}\right),{\mathsf{\Omega }}_c\left(s\right)=\{(u,t)\in {\mathsf{\Omega }}_c:s_{u,t}=s\}.$$

3.6. Tasks, Metrics, and Heterogeneity Index

We evaluate NAP with top-1 accuracy, macro-F1, and ROC–AUC. RP is evaluated with ROC–AUC, AUPRC, and calibration. Given M probability bins ${\left\{B_m\right\}}_{m=1}^M$, the expected calibration error is

$$\mathrm{ECE}=\sum _{m=1}^M{\displaystyle \frac{|B_m|}{N}}|\mathrm{acc}\left(B_m\right)-\mathrm{conf}\left(B_m\right)|.$$

To quantify non-IIDness across clients, we report a label skew index based on total variation of next-action marginals:

$$\mathrm{TV}–\mathrm{Skew}={\displaystyle \frac{2}{C(C-1)}}\sum _{1\le i<j\le C}{\displaystyle \frac{1}{2}}\sum _{k\in {\mathcal{A}}_{\mathrm{pred}}}|p_i\left(k\right)-p_j\left(k\right)|,$$

where $p_c\left(k\right)$ is the empirical frequency of class k on client c. We additionally state the Dirichlet concentration ${\alpha }_{\mathrm{part}}$ used to synthesize heterogeneous partitions when needed in Section 5.

Summary of symbols. C (clients), ${\mathcal{D}}_c$ (local data), ${\mathbf{s}}_u$ (sequence), $\psi$ (action tokenizer), $\mathcal{G}, {\mathcal{G}}_{u,t}$ (ActionGraph and ego-net), $\mathsf{\Pi }$ (behavior prompt), $\theta$ (frozen LLM), $\varphi$ (adapters), ${\mathbf{h}}_{u,t}$/${\mathbf{z}}_{u,t}$ (hidden/representation), ${\mathit{\mu }}_{c,k}$/${\mathit{\pi }}_k$ (local/global prototypes), $S, \sigma , q, R$ (DP/participation parameters), s (instructional regime), and evaluation metrics as above. These notations are used consistently in Section 4 and Section 5.

4. Methodology

We present BAF–FedLLM, a three-part framework built on the preliminaries of Section 3: BP–FIT for behavior-prompted federated instruction tuning of adapters, ProtoAlign for cross-client prototype alignment under secure aggregation, and CBR for causal behavior regularization that enforces regime-invariant risk. We then provide a unified training algorithm, convergence and privacy guarantees, and a communication analysis. Throughout, the backbone ${\mathrm{LLM}}_{\theta }$ is frozen and we optimize only the adapter parameters $\varphi$ and task heads. Algorithm 1 and Figure 1 show a general pipeline of our proposal.

Algorithm 1 Training and deployment of the privacy-preserving federated LLM.

1: Input: Frozen LLM backbone $\theta$, adapter parameters $\varphi$, privacy budget $(\epsilon ,\delta )$, clip norm S, noise multiplier $\sigma$, number of rounds R   2: for each round $r=1,\dots ,R$ do   3:    Server samples a subset of clients ${\mathcal{C}}_r$   4:    for each client $c\in {\mathcal{C}}_r$ in parallel do   5:      Construct behavior graph and behavior prompts from local logs   6:      Form model input (system prompt + behavior prompt + query)   7:      Initialize local adapter parameters ${\varphi }_c\leftarrow \varphi$   8:      for local steps $t=1,\dots ,\tau$ do   9:         Compute task loss on local mini-batch 10:         Take gradient step on ${\varphi }_c$ (backbone $\theta$ is frozen) 11:      end for 12:      Clip update: $\mathsf{\Delta }{\varphi }_c\leftarrow \mathrm{clip}({\varphi }_c-\varphi ,S)$ 13:      Add noise: ${\widetilde{\mathsf{\Delta }\varphi }}_c\leftarrow \mathsf{\Delta }{\varphi }_c+\mathcal{N}(0,{\sigma }^2{S}^{2}I)$ 14:      Send ${\widetilde{\mathsf{\Delta }\varphi }}_c$ to server (via secure aggregation) 15:    end for 16:    Server aggregates: $\varphi \leftarrow \varphi +{\displaystyle \frac{1}{|{\mathcal{C}}_r|}}{\sum }_{c\in {\mathcal{C}}_r}{\widetilde{\mathsf{\Delta }\varphi }}_c$ 17: end for 18: Output: Global privacy-preserving model $(\theta ,\varphi )$ deployed to institutions

Figure 1. High-level overview of the proposed privacy-preserving federated LLM framework. Clients construct behavior prompts from local interaction logs, adapt frozen LLM backbones via parameter-efficient adapters under client-level differential privacy, and share only clipped and noised aggregates with the server. The server updates the global adapters and redistributes the model without accessing raw texts or identifiers.

4.1. BP–FIT: Behavior-Prompted Federated Instruction Tuning

BP–FIT adapts a frozen backbone ${\mathrm{LLM}}_{\theta }$ to student interaction streams by prepending compact behavior prompts to the tokenized action prefix and training only low-rank adapters $\varphi$ locally under clipping and noise. The design emphasizes three properties: (i) semantic grounding via ActionGraph-derived descriptors, (ii) robustness to missing or noisy metadata through stochastic prompting with self-distillation, and (iii) privacy/efficiency from parameter-efficient updates and client-level differential privacy.

4.1.1. Action-to-Text Tokenization and Behavior Prompt Construction

Given a learner prefix ${\mathbf{s}}_{u,1:t}=\left((a_1,t_1,{\kappa }_1),\dots ,(a_t,t_t,{\kappa }_t)\right)$ on client c, the tokenizer $\psi$ renders a short textual action token sequence ${\mathbf{x}}_{u,1:t}=\psi \left({\mathbf{s}}_{u,1:t}\right)$ over vocabulary $\mathsf{\Sigma }$ (Section 3). To expose structural regularities from the ActionGraph $\mathcal{G}$ without leaking raw identifiers, a prompt builder $\mathsf{\Pi }$ maps $({\mathbf{s}}_{u,1:t},{\mathcal{G}}_{u,t})$ to a fixed-form clause string:

$${\mathbf{b}}_{u,t}=\mathtt{skill}=s^{★};\mathtt{recent}\_\mathtt{correct}={\rho }_t;\mathtt{same}\_\mathtt{unit}\_\mathtt{ratio}=u_t;\mathtt{recency}=d_t;\mathtt{policy}=p_t.$$

Here $s^{★}\in {\mathcal{S}}_{\mathrm{skills}}$ is the modal skill in a sliding window; ${\rho }_t={\displaystyle \frac{1}{W}}{\sum }_{j=1}^W\mathbb{I}\left[{\mathrm{correct}}_{t-j+1}\right]$; $u_t={\displaystyle \frac{\#\left\{\mathrm{actions}\mathrm{in}\mathrm{current}\mathrm{unit}\right\}}{W}}$; $d_t$ summarizes inter-event gaps via multi-scale decay $d_t={\sum }_{\ell =1}^L{\alpha }_{\ell }exp(-{\lambda }_{\ell }\mathsf{\Delta }{t}_t)$ with ${\sum }_{\ell }{\alpha }_{\ell }=1$ and $\mathsf{\Delta }{t}_t=t_t-t_{t-1}$; and $p_t\in \{\mathtt{assessment},$$\mathtt{practice}, \mathtt{mastery}, \mathtt{mixed}\}$ is an instructional regime cue derived from $\mathcal{O}$ nodes linked to recent actions. To limit cardinality and prevent identifier leakage, continuous fields are discretized by a public quantizer R with bin set $\mathcal{B}$, and high-cardinality categories (e.g., resource IDs) are mapped to stable hashed buckets $h(·)$ with $K_h$ bins. The prompted sequence is

$${\tilde{\mathbf{x}}}_{u,1:t}=[{\mathbf{b}}_{u,t};{\mathbf{x}}_{u,1:t}],$$

with a strict token budget for ${\mathbf{b}}_{u,t}$ to preserve context for ${\mathbf{x}}_{u,1:t}$. The builder caches rolling statistics of ${\mathcal{G}}_{u,t}$ (typed degrees, recency counts) to keep amortized cost $O\left(1\right)$ per step.

In all experiments, the ActionGraph $\mathcal{G}$ contains nodes for actions, resources (items, videos, pages), and organizational entities (course, unit, instructor). We keep edge types $\{\mathtt{precedes}, \mathtt{same}\_\mathtt{skill}, \mathtt{same}\_\mathtt{unit}, \mathtt{prereq}, \mathtt{cohort}\}$ and apply an exponential time decay with ${\lambda }_{\ell }\in \{{10}^{-2},{10}^{-3},{10}^{-4}\}$ for short-, medium-, and long-range dependencies, respectively. Behavior descriptors such as recent correctness ${\rho }_t$ and same-unit ratio $u_t$ are computed over a sliding window of size $W=20$. Continuous quantities are discretized into 8 bins using a public quantile estimator fitted on a held-out validation subset, and high-cardinality identifiers (e.g., resource IDs) are mapped to $K_h=512$ hashed buckets to avoid leaking raw identifiers. The behavior prompt budget is fixed to $B_{\mathrm{prompt}}$ tokens per dataset, and prompts are truncated from the least informative fields if they exceed this budget.

4.1.2. Stochastic Prompting and Self-Distilled Objective

Institutional logs vary in fidelity; some clients may lack policy tags or fine-grained skills. To make predictions stable under missing fields, BP–FIT applies prompt dropout. For each of the five fields, we sample a Bernoulli mask $\mathbf{m}\in {\{0,1\}}^5$ with rate $p_{\mathrm{drop}}$ and elide masked clauses, producing two paired views per example: an on view with mask $\mathbf{m}$ and an off view with the complementary mask $\overline{\mathbf{m}}$. Let ${\mathbf{h}}_{u,t}$ denote the final hidden state of ${\mathrm{LLM}}_{\theta }$ on ${\tilde{\mathbf{x}}}_{u,1:t}$ and P a fixed projection yielding ${\mathbf{z}}_{u,t}=P{\mathbf{h}}_{u,t}$. The action and risk heads share ${\mathbf{h}}_{u,t}$:

$$p_{\varphi }^{\mathrm{act}}(·|{\tilde{\mathbf{x}}}_{u,1:t})=\mathrm{softmax}\left(W^{\mathrm{act}}{\mathbf{h}}_{u,t}\right),p_{\varphi }^{\mathrm{risk}}\left(1\right|{\tilde{\mathbf{x}}}_{u,1:t})=\sigma (w^{\top }{\mathbf{h}}_{u,t}+b).$$

Denoting predictions from the on and off views by $p_{\varphi }^{\mathrm{act},\mathrm{on}/\mathrm{off}}$ and $p_{\varphi }^{\mathrm{risk},\mathrm{on}/\mathrm{off}}$, the local BP–FIT loss on client c combines task cross-entropy with a self-distillation term that enforces prompt invariance:

$$\begin{array}{cc}\hfill {\mathcal{L}}_c^{\mathrm{BP}}\left(\varphi \right)& ={\mathcal{L}}_c^{\mathrm{NAP}}\left(\varphi \right)+{\lambda }_{\mathrm{risk}}{\mathcal{L}}_c^{\mathrm{RP}}\left(\varphi \right)\hfill \\ \hfill & +{\lambda }_{\mathrm{cons}}·{\displaystyle \frac{1}{|{\mathsf{\Omega }}_c|}}\sum _{(u,t)\in {\mathsf{\Omega }}_c}\left[\mathrm{KL}\left(p_{\varphi }^{\mathrm{act},\mathrm{on}}\parallel p_{\varphi }^{\mathrm{act},\mathrm{off}}\right)+\mathrm{KL}\left(p_{\varphi }^{\mathrm{risk},\mathrm{on}}\parallel p_{\varphi }^{\mathrm{risk},\mathrm{off}}\right)\right].\hfill \end{array}$$

(3)

The KL terms encourage consistency when salient descriptors are absent, improving calibration and reducing shortcutting on spurious prompt fields. We linearly warm up ${\lambda }_{\mathrm{cons}}$ over the first $R_{\mathrm{warm}}$ rounds to avoid underfitting early in training.

BP–FIT trains only low-rank adapters injected at selected attention and MLP layers. For a backbone weight $W\in {\mathbb{R}}^{m\times n}$, the trainable delta is $\mathsf{\Delta }W=AB$ with $A\in {\mathbb{R}}^{m\times r}$, $B\in {\mathbb{R}}^{r\times n}$, $r\ll min\{m,n\}$. The parameter set $\varphi$ aggregates all adapter pairs and the two task heads. Unless otherwise stated, local optimization uses Adam with learning rate $5\times {10}^{-4}$, weight decay ${10}^{-2}$, and mini-batch size 64, with $T_{\mathrm{local}}=\tau =5$ local steps per communication round. We apply gradient clipping $\parallel {\nabla }_{\varphi }{\parallel }_2 \le S$ with $S=1.0$ and label smoothing of $0.1$ on NAP to temper overconfidence. Only the clipped update $\mathsf{\Delta }{\varphi }_c$ is released; before secure aggregation, it is perturbed by Gaussian noise with multiplier $\sigma$, yielding client-level $(\epsilon ,\delta )$ guarantees after composition across rounds. In all reported configurations, we use $\sigma =1.2$ for adapter updates and $(S_{\pi },{\sigma }_{\pi },{\sigma }_n)=(1.0,0.8,0.5)$ for prototype messages, resulting in $\epsilon \in [1.61,1.72]$ at $\delta ={10}^{-5}$. No tokens, prompts, counts, or ActionGraph features leave the client. Communication per round is $O\left(r\right(m+n\left)\right)$ numbers for adapters plus a small overhead for prototypes, independent of raw sequence length.

By canonically rendering a few stable descriptors from ${\mathcal{G}}_{u,t}$ and enforcing invariance to their omission, BP–FIT steers ${\mathrm{LLM}}_{\theta }$ toward pedagogically meaningful structure while remaining resilient to heterogeneous logging. The projected states ${\mathbf{z}}_{u,t}$ produced here are subsequently consumed by ProtoAlign for cross-client alignment, and the regime indicators implicit in $p_t$ integrate with CBR to regularize risk across instructional regimes, ensuring methodological cohesion across the next subsections.

4.2. ProtoAlign: Cross-Client Prototype Alignment

Objective. ProtoAlign mitigates representation drift caused by non-IID client distributions by aligning projected states ${\mathbf{z}}_{u,t}=P{\mathbf{h}}_{u,t}$ to global class anchors ${\left\{{\mathit{\pi }}_k\right\}}_{k\in {\mathcal{A}}_{\mathrm{pred}}}$ that are computed from privatized, securely aggregated local prototypes. Unlike methods that exchange gradients or example features, ProtoAlign communicates only clipped, noisy sufficient statistics. This preserves client-level privacy and adds a small, count-proportional overhead to adapter updates.

4.2.1. Private Prototype Messaging and Anchor Estimation

On client c, the per-class prototype is ${\mathit{\mu }}_{c,k}={\displaystyle \frac{1}{n_{c,k}}}{\sum }_{(u,t)\in B_{c,k}}{\mathbf{z}}_{u,t}$ with $n_{c,k}=\left|B_{c,k}\right|$. To bound sensitivity, we clip it to radius $S_{\pi }$, obtaining ${\widehat{\mathit{\mu }}}_{c,k}={\mathit{\mu }}_{c,k}·min\left(1,S_{\pi }/{\parallel {\mathit{\mu }}_{c,k}\parallel }_2\right)$. Clients send only masked sums through secure aggregation:

$${\mathbf{s}}_{c,k}=n_{c,k}{\widehat{\mathit{\mu }}}_{c,k}+\mathcal{N}(0,{\sigma }_{\pi }^2{S}_{\pi }^{2}I),{\tilde{n}}_{c,k}=n_{c,k}+\mathcal{N}(0,{\sigma }_n^2).$$

The server receives ${\tilde{\mathbf{s}}}_k={\sum }_{c\in {\mathcal{S}}_r}{\mathbf{s}}_{c,k}$ and ${\tilde{n}}_k={\sum }_{c\in {\mathcal{S}}_r}{\tilde{n}}_{c,k}$ (but never per-client values) and forms a debiased, stabilized anchor estimate

$${\widehat{\mathit{\pi }}}_k^{\left(r\right)}={\alpha }_k·{\displaystyle \frac{{\tilde{\mathbf{s}}}_k}{max\{1,{\tilde{n}}_k\}}},{\alpha }_k={\displaystyle \frac{{\tilde{n}}_k}{{\tilde{n}}_k+{\varsigma }_n^2}},$$

where ${\varsigma }_n^2\approx \left|{\mathcal{S}}_r\right|{\sigma }_n^2$ is the aggregate count noise variance (known to the server from the number of participating clients), and ${\alpha }_k\in (0,1]$ attenuates anchors estimated from few or noisy counts. To reduce high-variance fluctuations from ${\sigma }_{\pi }$, anchors are exponentially averaged across rounds and ${\ell }_2$-normalized to match cosine similarity used downstream:

$${\mathit{\pi }}_k\leftarrow (1-\beta ){\mathit{\pi }}_k+\beta {\displaystyle \frac{{\widehat{\mathit{\pi }}}_k^{\left(r\right)}}{\parallel {\widehat{\mathit{\pi }}}_k^{\left(r\right)}{\parallel }_2+ϵ}},\beta \in (0,1],ϵ>0.$$

Sensitivity and privacy. Clipping by $S_{\pi }$ ensures the per-client contribution to ${\tilde{\mathbf{s}}}_k$ has ${\ell }_2$ sensitivity at most $S_{\pi }$; the Gaussian mechanisms with multipliers $({\sigma }_{\pi },{\sigma }_n)$ therefore compose (via subsampled RDP in Section 3) with the noised adapter updates to yield client-level $(\epsilon ,\delta )$ bounds. Because only sums are visible under secure aggregation, no per-client prototype or count is ever exposed.

4.2.2. Alignment Loss with Prior Correction and Hard Negatives

Given privatized anchors, each client aligns its representations via a supervised contrastive objective. For a labeled pair $({\mathbf{z}}_{u,t},k)$ with $k=y_{u,t}^{\mathrm{act}}$, define

$${\mathcal{L}}_c^{\mathrm{PA}}\left(\varphi \right)=-{\displaystyle \frac{1}{|{\mathsf{\Omega }}_c|}}\sum _{(u,t)\in {\mathsf{\Omega }}_c}log{\displaystyle \frac{exp\left(\mathrm{sim}({\mathbf{z}}_{u,t},{\mathit{\pi }}_k)/{\tau }_{\mathrm{PA}}\right)}{{\sum }_{j\in {\mathcal{A}}_{\mathrm{pred}}}{\omega }_{c,j}exp\left(\mathrm{sim}({\mathbf{z}}_{u,t},{\mathit{\pi }}_j)/{\tau }_{\mathrm{PA}}\right)}},$$

(4)

where $\mathrm{sim}(\mathbf{z},\mathit{\pi })={\displaystyle \frac{{\mathbf{z}}^{\top }\mathit{\pi }}{{\parallel \mathbf{z}\parallel }_2{\parallel \mathit{\pi }\parallel }_2}}$ and ${\tau }_{\mathrm{PA}}>0$. To counter label–prior mismatch across clients, we introduce prior correction weights

$${\omega }_{c,j}={\left({\displaystyle \frac{\widehat{P}\left(j\right)}{{\widehat{p}}_c\left(j\right)+ϵ}}\right)}^{\gamma },{\widehat{p}}_c\left(j\right)={\displaystyle \frac{n_{c,j}}{{\sum }_{\ell }{n}_{c,\ell }}},\widehat{P}\left(j\right)\propto \sum _{c\in {\mathcal{S}}_r}{\tilde{n}}_{c,j},$$

with exponent $\gamma \in [0,1]$ controlling the strength of reweighting. This emphasizes underrepresented classes on each client without revealing exact priors. For hard negatives, we optionally sharpen the denominator by upweighting anchors j whose similarity exceeds a margin $\kappa$, i.e., set ${\omega }_{c,j}\leftarrow {\omega }_{c,j}(1+{\lambda }_{\mathrm{hard}}\mathbb{I}[\mathrm{sim}(\mathbf{z},{\pi }_j)>\kappa ])$, which accelerates separation under heavy class overlap.

The anchor-based contrast pulls representations toward class-level centroids that average away client-specific nuisance variability while pushing away other classes globally. This reduces the effective smoothness constant and the drift term in the federated convergence bound (Section 3), improving both optimization and calibration under non-IID partitions.

5. Theoretical Analysis

This section revises and expands the guarantees that underlie BAF–FedLLM. We first formalize assumptions and the unified objective that couples BP–FIT, ProtoAlign, and CBR. We then present non-asymptotic convergence bounds that make explicit the roles of DP noise, client drift, and alignment/invariance. Next, we provide client-level differential privacy accounting that composes noised adapter updates with privatized prototype messaging under secure aggregation. Finally, we analyze mixture shift generalization, calibration, and the effect of low-rank adapters on optimization bias.

5.1. Setup, Assumptions, and Unified Objective

Notation 1. 

There are C clients indexed by $c\in \left[C\right]$. Client c holds data ${\mathcal{D}}_c$ and induces a loss $F_c\left(\varphi \right)={\mathbb{E}}_{\xi \sim {\mathcal{D}}_c}\left[{\mathcal{L}}_c(\varphi ;\xi )\right]$, where ${\mathcal{L}}_c$ combines BP–FIT, ProtoAlign, and CBR. The global objective is the mixture

$$F\left(\varphi \right)=\sum _{c=1}^C{p}_c{F}_c\left(\varphi \right),p_c={\displaystyle \frac{|{\mathcal{D}}_c|}{{\sum }_{j=1}^C\left|{\mathcal{D}}_j\right|}}.$$

Per round r, a subset ${\mathcal{S}}_r$ participates with rate q. Each selected client takes τ local steps, returning a clipped and noised update $\overline{\mathsf{\Delta }}{\varphi }_c^{\left(r\right)}$, and the server applies the weighted update. Prototypes are shared as noised sums through secure aggregation (Section 3).

Assumption 1. 

We adopt standard FL conditions:

• F is L smooth: $\parallel \nabla F\left(\varphi \right)-\nabla F\left({\varphi }^{\prime }\right){\parallel }_2 \le L{\parallel \varphi -{\varphi }^{\prime }\parallel }_2$.
• Stochastic gradients have bounded variance: $\mathbb{E}\parallel \nabla {\mathcal{L}}_c(\varphi ;\xi )-\nabla F_c{\left(\varphi \right)\parallel }_2^2\le {\sigma }_g^2$.
• Client drift at a minimizer ${\varphi }^{★}$ is ${\zeta }^2={\sum }_c{p}_c{\parallel \nabla F_c\left({\varphi }^{★}\right)-\nabla F\left({\varphi }^{★}\right)\parallel }_2^2$.
• Adapter updates are clipped at radius S and perturbed by Gaussian noise $\mathcal{N}(0,{\sigma }^2{S}^{2}I)$ prior to secure aggregation. Prototype messages are clipped at $S_{\pi }$ and noised with multipliers $({\sigma }_{\pi },{\sigma }_n)$.

Unified objective. For each client

$$\begin{array}{cc}\hfill {\mathcal{L}}_c\left(\varphi \right)& =\underset{\mathrm{BP}–\mathrm{FIT}\mathrm{tasks}}{\underbrace{{\mathcal{L}}_c^{\mathrm{NAP}}\left(\varphi \right)+{\lambda }_{\mathrm{risk}}{\mathcal{L}}_c^{\mathrm{RP}}\left(\varphi \right)}}\hfill \\ \hfill & +{\lambda }_{\mathrm{cons}}{\mathcal{D}}_{\mathrm{KL}}^{\mathrm{prompt}}\left(\varphi \right)+{\lambda }_{\mathrm{PA}}{\mathcal{L}}_c^{\mathrm{PA}}\left(\varphi \right)+{\lambda }_{\mathrm{disp}}{\mathcal{L}}_c^{\mathrm{disp}}\left(\varphi \right)+{\lambda }_{\mathrm{inv}}{\mathcal{L}}_c^{\mathrm{inv}}\left(\varphi \right),\hfill \end{array}$$

(5)

where ${\mathcal{D}}_{\mathrm{KL}}^{\mathrm{prompt}}$ is the BP–FIT consistency term, ${\mathcal{L}}^{\mathrm{PA}}$ is the anchor-based InfoNCE loss, and ${\mathcal{L}}^{\mathrm{disp}}$, ${\mathcal{L}}^{\mathrm{inv}}$ are the CBR penalties (all defined in Section 3 and Section 4).

We also define two helpful margins that quantify the regularizing effect of ProtoAlign and CBR:

$${\mathsf{\Delta }}_{\mathrm{align}}=\mathbb{E}\left[\mathrm{sim}(\mathbf{z},{\pi }_y)-\underset{j\ne y}{max}\mathrm{sim}(\mathbf{z},{\pi }_j)\right],{\mathsf{\Delta }}_{\mathrm{inv}}=\sum _{s\in \mathcal{S}}{∥{\nabla }_{\nu }{\widehat{\mathcal{R}}}_c^{\left(s\right)}(\nu w,b){|}_{\nu =1}∥}_2^2.$$

Larger ${\mathsf{\Delta }}_{\mathrm{align}}$ and smaller ${\mathsf{\Delta }}_{\mathrm{inv}}$ both improve optimization, as formalized next.

5.2. Convergence with Non-IID Data, DP Noise, and Alignment/Invariance

Let $T=qCR\tau$ be the expected number of local steps across R rounds and keep step size $\eta \le 1/\left(2L\right)$.

Theorem 1 

(Stationarity under alignment and invariance). Under the assumptions above

$${\displaystyle \frac{1}{R}}\sum _{r=0}^{R-1}\mathbb{E}\left[\parallel \nabla F\left({\varphi }^{\left(r\right)}\right){\parallel }_2^2\right]\le \underset{\mathit{optimization}}{\underbrace{{\displaystyle \frac{2\left(F\left({\varphi }^{\left(0\right)}\right)-F^{★}\right)}{\eta T}}}}+\underset{\mathit{stochastic}+\mathit{DP}\mathit{noise}}{\underbrace{2L\eta \left({\sigma }_g^2+{\sigma }^2{S}^2\right)}}+\underset{\mathit{client}\mathit{drift}}{\underbrace{2L^2{\eta }^2{\tau }^2{\zeta }^2}}-\underset{\mathit{alignment}\mathit{gain}}{\underbrace{c_{\mathrm{PA}}{\lambda }_{\mathrm{PA}}{\mathsf{\Delta }}_{\mathrm{align}}}}-\underset{\mathit{invariance}\mathit{gain}}{\underbrace{c_{\mathrm{CBR}}\left({\lambda }_{\mathrm{disp}}+{\lambda }_{\mathrm{inv}}\right)\mathsf{\Gamma }}}.$$

Here $c_{\mathrm{PA}},c_{\mathrm{CBR}}>0$ are constants depending on temperature ${\tau }_{\mathrm{PA}}$ and the smoothness of the logistic head, and Γ is a regime variance surrogate that decreases as ${\mathsf{\Delta }}_{\mathrm{inv}}$ decreases.

Sketch. The proof adapts the standard descent lemma for noisy local SGD with partial participation, adding two strongly concave regularizers: (i) a cosine similarity neighborhood induced by ${\mathcal{L}}^{\mathrm{PA}}$ that increases the margin ${\mathsf{\Delta }}_{\mathrm{align}}$ and tightens the smoothness constant observed by the optimizer; (ii) CBR terms that reduce cross-regime variance in the gradients of the risk head, effectively shrinking the drift seen by the global model. The negative terms in (1) quantify these gains.

Corollary 1 

(PL condition). If F satisfies the Polyak–Łojasiewicz (PL) inequality with parameter $\mu >0$, then for $\eta \le min\{1/\left(2L\right),\mu /\left(4L^2{\tau }^2\right)\}$,

$$\begin{array}{cc}\hfill \mathbb{E}\left[F\left({\varphi }^{\left(R\right)}\right)-F^{★}\right]& \le {(1-\eta \mu )}^T·\left(F\left({\varphi }^{\left(0\right)}\right)-F^{★}\right)+{\displaystyle \frac{\eta L}{\mu }}\left({\sigma }_g^2+{\sigma }^2{S}^2\right)+{\displaystyle \frac{L\eta {\tau }^2}{\mu }}{\zeta }^2\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & -{\displaystyle \frac{\eta }{\mu }}\left(c_{\mathrm{PA}}{\lambda }_{\mathrm{PA}}{\mathsf{\Delta }}_{\mathrm{align}}+c_{\mathrm{CBR}}({\lambda }_{\mathrm{disp}}+{\lambda }_{\mathrm{inv}})\mathsf{\Gamma }\right).\hfill \end{array}$$

(7)

Thus alignment and invariance accelerate convergence and reduce the error floor created by DP noise and client drift.

Remark 1 

(Low-rank approximation bias). Let $\mathsf{\Delta }{W}^{★}$ be the full-rank optimal weight delta for a backbone layer and let ${\mathsf{\Pi }}_r$ denote the best rank r approximation operator. If adapters implement $\mathsf{\Delta }W={\mathsf{\Pi }}_r\left(\mathsf{\Delta }{W}^{★}\right)$, then the approximation residual contributes an additive bias $B_r\propto {\parallel \mathsf{\Delta }{W}^{★}-{\mathsf{\Pi }}_r\left(\mathsf{\Delta }{W}^{★}\right)\parallel }_F^2$ to the optimization term. Empirically, modest ranks (e.g., $r=8$) keep $B_r$ small relative to the alignment/invariance gains.

5.3. Client-Level Differential Privacy with Secure Prototype Messaging

We analyze the composite mechanism releasing (i) adapter updates and (ii) prototype messages each round.

Mechanism M_upd. Each participating client releases $\overline{\mathsf{\Delta }}{\varphi }_c^{\left(r\right)}=\mathrm{clip}(\mathsf{\Delta }{\varphi }_c^{\left(r\right)},S)+\mathcal{N}(0,{\sigma }^2{S}^{2}I)$ via secure aggregation. With subsampling rate q and R rounds, its Rényi DP cost is ${\mathcal{A}}_{\mathrm{RDP}}(q,\sigma ,R)$.

Mechanism M_proto. For each class k, client c sends ${\mathbf{s}}_{c,k}=n_{c,k}{\widehat{\mathit{\mu }}}_{c,k}+\mathcal{N}(0,{\sigma }_{\pi }^2{S}_{\pi }^{2}I)$ and ${\tilde{n}}_{c,k}=n_{c,k}+\mathcal{N}(0,{\sigma }_n^2)$ through secure aggregation. Clipping at $S_{\pi }$ bounds sensitivity and yields Rényi costs ${\mathcal{A}}_{\mathrm{RDP}}(q,{\sigma }_{\pi },R)$ and ${\mathcal{A}}_{\mathrm{RDP}}(q,{\sigma }_n,R)$.

Theorem 2 

(Client–DP composition). For any target $\delta \in (0,1)$, the combined mechanism $(M_{\mathit{upd}},M_{\mathit{proto}})$ is $(\epsilon ,\delta )$ client–DP with

$$\epsilon \le \underset{\alpha >1}{min}\left\{{\mathcal{A}}_{\mathrm{RDP}}(q,\sigma ,R)+{\mathcal{A}}_{\mathrm{RDP}}(q,{\sigma }_{\pi },R)+{\mathcal{A}}_{\mathrm{RDP}}(q,{\sigma }_n,R)\right\}\Rightarrow (\epsilon ,\delta ).$$

Secure aggregation ensures the server observes only sums across clients; per-client messages remain hidden even from an honest-but-curious server. The privacy unit is an entire institution.

Corollary 2 

(Typical budgets). With $(q,R)=(0.1,150)$ and multipliers $(\sigma ,{\sigma }_{\pi },{\sigma }_n)=(1.2,0.8,0.5)$, the above yields ε in the range $1.6$–$1.8$ for $\delta ={10}^{-5}$, consistent with Section 5.

5.4. Mixture Shift Generalization, Calibration, and Communication Complexity

Generalization across client mixtures. Let P be the training mixture over clients and Q an unseen evaluation mixture (e.g., different cohort proportions). Consider a linear action head on unit norm features with Lipschitz constant $L_{\mathrm{act}}={\parallel W^{\mathrm{act}}\parallel }_2$. Define class conditional prototype drift

$${\delta }_k=\parallel {\mathbb{E}}_P\left[\mathbf{z}\right|y=k]-{\mathbb{E}}_Q\left[\mathbf{z}\right|y=k]{\parallel }_2\mathrm{and}{\mathsf{\Delta }}_{\mathrm{prior}}={\parallel {\mathbf{p}}_P-{\mathbf{p}}_Q\parallel }_1,$$

where ${\mathbf{p}}_P$, ${\mathbf{p}}_Q$ are label marginals. Then for cross-entropy loss ℓ

$$\underset{\mathrm{mixture}\mathrm{shift}\mathrm{gap}}{\underbrace{|{\mathbb{E}}_Q[\ell ]-{\mathbb{E}}_P[\ell ]|}}\le L_{\mathrm{act}}\sum _k{\pi }_k{\delta }_k+C{\mathsf{\Delta }}_{\mathrm{prior}},$$

for a universal constant C. ProtoAlign contracts ${\delta }_k$ by aligning client-level features to common anchors, while BP–FIT reduces sensitivity to metadata gaps, jointly tightening this bound.

Calibration. Let logits for the risk head be $z=w^{\top }\mathbf{h}+b$ with Lipschitz constant $L_z$ in $\mathbf{h}$. If CBR reduces the across-regime logit variance by a factor $\rho <1$, then for standard binning estimators, the expected calibration error satisfies

$${\mathrm{ECE}}_{\mathrm{with}\mathrm{CBR}}\le \rho {\mathrm{ECE}}_{\mathrm{without}\mathrm{CBR}}+O\left({\displaystyle \frac{1}{\sqrt{N}}}\right),$$

where N is the number of evaluation points. Empirically, we observe $\rho \approx 0.72$–$0.76$ (Section 5), matching the 24–$28\%$ ECE reductions.

Communication complexity. Each round, a client transmits

$$\underset{\mathrm{adapter}\mathrm{update}}{\underbrace{p_{\mathrm{adpt}}·b}}+\underset{\mathrm{prototype}\mathrm{sums}}{\underbrace{Kd·b}}+\underset{\mathrm{noisy}\mathrm{counts}}{\underbrace{K·b}}\mathrm{bits},$$

with bitwidth b, class count K, and representation dimension d. Since $p_{\mathrm{adpt}}\ll \left|\theta \right|$ and K is modest, adapter updates dominate; ProtoAlign adds a small fixed overhead. Low-rank adapters thus allow stronger DP noise at a similar bandwidth compared to full fine-tuning, without materially changing convergence (Remark above).

6. Experiments

We empirically validate BAF–FedLLM on three public education datasets with institution-level partitions and non-IID behavior: EdNet (clickstream-based K–12) (https://github.com/riiid/ednet, accessed on 9 October 2025), ASSISTments (K–12 tutoring) (https://sites.google.com/site/assistmentsdata/, accessed on 9 October 2025), and OULAD (open university) (https://analyse.kmi.open.ac.uk/open-dataset, accessed on 9 October 2025 or https://doi.org/10.1038/sdata.2017.171, accessed on 9 October 2025). We evaluate the two tasks defined in Section 3: next-action prediction (NAP) and risk prediction (RP). All numbers are averaged over five seeds; we report means and note significance via paired t-tests across clients ($p<0.01$ unless stated). The same notation as in Section 3 and Section 4 is used throughout.

6.1. Setup

Corpora and tasks. We evaluate BAF–FedLLM on three public learning platforms with heterogeneous behavior and governance constraints: EdNet (clickstream-based K–12), ASSISTments (tutoring), and OULAD (open university). Following Section 3, each interaction is rendered to an action token and augmented by a compact behavior prompt derived from the ActionGraph ego-net. The two prediction tasks are (i) next-action prediction (NAP) over a fixed action alphabet ${\mathcal{A}}_{\mathrm{pred}}$ and (ii) risk prediction (RP) for near-term disengagement. For K–12 (EdNet/ASSISTments), we label RP for a $H=28$ day horizon; for OULAD, whose modules are longer, we use $H=42$ days. Table 1 summarizes the corpora after standard cleaning (de-duplication, timestamp sanity checks, anonymization) and tokenization. We cap sequences at $L_{max}$ tokens via head–tail truncation and constrain prompts to a budget $B_{\mathrm{prompt}}$ to preserve context length for actions.

Table 1. Corpora and tokenization. C clients are derived from institutional metadata; $L_{max}$ is the sequence cap; $B_{\mathrm{prompt}}$ caps the behavior prompt. NAP uses $K=|{\mathcal{A}}_{\mathrm{pred}}|=10$ action types across all datasets. RP horizon H is 28 days (EdNet/ASSISTments) and 42 days (OULAD).

Dataset	C	Learners	Interactions	Resources	Avg Seq (p50/p95)	${\mathit{L}}_{max}$	${\mathit{B}}_{\mathbf{prompt}}$
EdNet	32	52,130	4.21 M	78,400	81 (48/230)	256	18
ASSISTments	24	41,905	2.31 M	57,900	57 (37/160)	256	18
OULAD	10	26,103	1.04 M	12,500	39 (25/120)	192	16

Federated partitioning and non-IIDness. Clients correspond to institution-level units (schools/classes in EdNet/ASSISTments; module presentations in OULAD). Where metadata is sparse, we synthesize heterogeneity by Dirichlet partitioning with concentration ${\alpha }_{\mathrm{part}}$ over next-action classes. We quantify label skew by the TV–skew index from Section 3; empirical values are reported in Table 2 and visualized in Figure 2. To probe sensitivity to heterogeneity, we also generate auxiliary splits at ${\alpha }_{\mathrm{part}}\in \{0.1,0.3,0.5,1.0\}$ and track the induced skew (Figure 3).

Table 2. Federated protocol and privacy configuration. Shared defaults across datasets unless noted: $R=150$, $q=0.1$, $\tau =5$, rank $r=8$, learning rate $\eta =5\times {10}^{-4}$, EMA $\beta =0.5$, temperature ${\tau }_{\mathrm{PA}}=0.2$. TV–skew estimated on training splits; $\epsilon$ at $\delta ={10}^{-5}$.

Dataset	TV–Skew	${\mathit{\alpha }}_{\mathbf{part}}$	S	$\mathit{\sigma }$	${\mathit{S}}_{\mathit{\pi }}$	${\mathit{\sigma }}_{\mathit{\pi }}$	${\mathit{\sigma }}_{\mathit{n}}$	$\mathit{\epsilon }$
EdNet	0.31	0.30	1.0	1.2	1.0	0.8	0.5	1.72
ASSISTments	0.28	0.30	1.0	1.2	1.0	0.8	0.5	1.67
OULAD	0.22	0.30	1.0	1.2	1.0	0.8	0.5	1.61

Figure 2. Measured non-IID label skew. Bars show TV–skew for each dataset (left to right: EdNet, ASSISTments, OULAD). No legend is shown by design; bar ordering conveys the dataset identity.

Figure 3. Heterogeneity calibration. Solid curve: empirical TV–skew observed on EdNet under synthetic partitions at ${\alpha }_{\mathrm{part}}\in \{0.1,0.2,0.3,0.5,1.0\}$. Dashed curve: analytical expectation for Dirichlet sampling with the same class prior. No legend is shown; solid vs. dashed are described here.

Backbone, adapters, and privacy. The backbone ${\mathrm{LLM}}_{\theta }$ is frozen; we train LoRA-style adapters of rank $r=8$ on selected attention/MLP layers with shared task heads. Training proceeds for $R=150$ rounds with partial participation $q=0.1$ and $\tau =5$ local steps. Client updates are clipped at $S=1.0$ and noised with multiplier $\sigma =1.2$; prototype messages use $(S_{\pi },{\sigma }_{\pi },{\sigma }_n)=(1.0,0.8,0.5)$. Subsampled Rényi accountants yield client-level budgets $\epsilon \in [1.61,1.72]$ at $\delta ={10}^{-5}$; details per dataset appear in Table 2. All evaluations use five seeds and report mean ± std over clients; calibration uses $M=15$ equal-width bins.

Evaluation protocol. For NAP, we report top-1 accuracy, macro-F1, and ROC–AUC; for RP, we report ROC–AUC, AUPRC, and ECE with binning-corrected estimates. Paired t-tests across clients (per seed) assess significance at $p<0.01$. Unless noted, hyperparameters are fixed across datasets (Table 2); ablations vary one factor at a time. All results are computed per client and averaged to respect federated evaluation, with stratified confidence intervals across seeds. (Our empirical reliability analysis centers on accuracy, calibration, and robustness to client sampling and label skew; more fine-grained LLM-specific probes such as template sensitivity, sequence perturbations, and held-out institution generalization are left to future work (Section 7).)

Optimization and hyperparameters. For all datasets, we keep the federated protocol fixed: $R=150$ rounds, participation rate $q=0.1$, and $\tau =5$ local steps with adapter rank $r=8$. Clients use Adam with learning rate $5\times {10}^{-4}$, weight decay ${10}^{-2}$, and batch size 64. Prompt dropout is set to $p_{\mathrm{drop}}=0.15$, the alignment temperature to ${\tau }_{\mathrm{PA}}=0.2$, and the invariance weight to ${\lambda }_{\mathrm{inv}}=0.1$, matching the configuration used in the ablations of Section 6.3. All default hyperparameters are summarized in Table 3. More details about our experimental settings are given in Appendix A.

Table 3. Main notation and default hyperparameters used in experiments.

Symbol	Description	Default Value
C	Number of clients (institutions)	dataset–dependent
r	LoRA rank	8
R	Communication rounds	150
q	Participation rate	$0.1$
$\tau$	Local steps per round	5
S	Clip norm for updates	$1.0$
$\sigma$	Noise multiplier (updates)	$1.2$
$S_{\pi }$	Prototype clip norm	$1.0$
${\sigma }_{\pi },{\sigma }_n$	Prototype noise multipliers	$0.8,0.5$
${\lambda }_{\mathrm{risk}}$	Weight on RP loss	$1.0$
${\lambda }_{\mathrm{cons}}$	Prompt consistency weight	warm–up to $0.5$
${\lambda }_{\mathrm{PA}}$	ProtoAlign weight	$0.5$
${\lambda }_{\mathrm{disp}},{\lambda }_{\mathrm{inv}}$	CBR dispersion/invariance weights	$0.1,0.1$
${\tau }_{\mathrm{PA}}$	ProtoAlign temperature	$0.2$
$p_{\mathrm{drop}}$	Prompt dropout rate	$0.15$

6.2. Main Results

Overall performance. Table 4 and Figure 4 summarize next-action prediction (NAP). Across EdNet/ASSISTments/OULAD, BAF–FedLLM improves ROC–AUC over the strongest baseline by $+0.041/+0.039/+0.039$ absolute (relative +5.0%/+4.9%/+5.2%). These gains translate to $+2.8$–$3.4$ points in top-1 accuracy and $+3.0$–$3.8$ points in macro-F1, reflecting better class balance and fewer false positives on rare actions. Pairwise client-level t-tests (five seeds) show significance at $p<0.01$ on all datasets.

Table 4. NAP results (mean over five seeds; client-averaged). Bold indicates the best per dataset.

	EdNet		ASSISTments		OULAD
Method	ROC–AUC	Acc	ROC–AUC	Acc	ROC–AUC	Acc
Local	0.804	0.742	0.761	0.701	0.728	0.668
FedAvg–Adapter	0.812	0.750	0.784	0.723	0.743	0.681
FedProx–Adapter	0.815	0.754	0.789	0.728	0.748	0.686
FedProto	0.821	0.759	0.790	0.730	0.756	0.692
BAF–FedLLM	0.862	0.787	0.829	0.756	0.795	0.721

Figure 4. NAP ROC–AUC by dataset and method. Within each dataset group, the five bars (left to right) correspond to Local, FedAvg–Adapter, FedProx–Adapter, FedProto, and BAF–FedLLM. No legend is displayed by design; bar order conveys method identity.

Risk prediction and calibration. Table 5 and Figure 5 report risk prediction (RP). Relative to the best baseline ROC–AUC of 0.801/0.772/0.744, our method reaches 0.838/0.814/0.785. AUPRC improves by +0.051/+0.044/+0.046 absolute, and expected calibration error (ECE) drops by 28%/24%/24% (from 4.9/5.4/6.2 to 3.5/4.1/4.7). Improvements are consistent across clients (interquartile gains: AUC $+0.026$ to $+0.051$; ECE $-0.7$ to $-1.9$ points). These results are aligned with the design: BP–FIT stabilizes predictions under missing metadata, ProtoAlign reduces cross-client drift, and CBR suppresses regime-specific shortcuts, sharpening recall at low false positive rates and improving probability calibration.

Table 5. RP results (mean over five seeds; client-averaged). ECE is reported in percentage points with $M=15$ bins. Bold indicates the best value per dataset. In the rest of our paper, the ↓ (or ↑) symbol indicates that lower (or higher) values are better.

Dataset	Method	ROC–AUC	AUPRC	ECE ↓
EdNet	Local	0.786	0.407	6.1
	Best Baseline	0.801	0.412	4.9
	BAF–FedLLM	0.838	0.463	3.5
ASSISTments	Local	0.748	0.365	6.8
	Best Baseline	0.772	0.389	5.4
	BAF–FedLLM	0.814	0.433	4.1
OULAD	Local	0.715	0.332	7.4
	Best Baseline	0.744	0.356	6.2
	BAF–FedLLM	0.785	0.402	4.7

Figure 5. RP precision–recall quality. Bars per dataset show AUPRC for the best baseline other than ours (left bar) and BAF–FedLLM (right bar). No legend is displayed; the caption states the bar semantics.

Effect sizes and robustness. On EdNet, per-client Cohen’s d for NAP AUC between BAF–FedLLM and the best baseline is $0.71$ (medium–large). Under resampling of client participation ($q\in [0.05,0.2]$), average NAP AUC varies by $\le 0.006$, indicating robustness to partial participation. Across seeds, standard deviations remain small (AUC $\le 0.006$, AUPRC $\le 0.008$, ECE $\le 0.3$), consistent with reduced variance from prototype alignment and invariance regularization.

Our baseline suite focuses on local training, FedAvg/FedProx with adapters, and prototype-based FedProto because these methods are representative of global model federated learning under non-IID behavior and can be implemented with frozen LLM backbones and client-level DP using clipped/noised adapter updates. In contrast, drift correction via control variates (e.g., SCAFFOLD-style methods), adaptive server optimizers (FedOpt), and fully personalized FL often assume full-model updates, additional global control states, or client-specific heads, which complicate our privacy and communication budget and depart from the single deployable global model considered here. We view ProtoAlign and CBR as complementary to these optimizer- and personalization-level techniques, and we leave systematic comparison and integration with FedOpt/SCAFFOLD/personalized FL as important directions for future work.

Interpretation. NAP gains are largest in clients with high label skew and sparser prompts, consistent with ProtoAlign contracting inter-client prototype dispersion while BP–FIT enforces prompt invariance. RP benefits most on cohorts exhibiting assessment-heavy regimes; CBR dampens regime-specific overconfidence, lowering ECE without sacrificing discrimination. Together these effects yield better calibrated, more transferable predictors under client-level DP.

6.3. Ablations and Analyses

Module contributions. We ablate each component of BAF–FedLLM on all datasets by removing ProtoAlign (–PA), BP–FIT (–BP), or CBR (–CBR), and by removing both alignment and invariance (–PA, –CBR). As a sanity check, we also report a non-private variant (No DP, $\sigma =0$) to quantify the privacy cost. Table 6 shows that ProtoAlign delivers the largest accuracy gain under label skew (AUC drops by $1.2$–$1.5$ points when removed), BP–FIT adds robustness to missing metadata (AUC drops by $0.6$–$0.9$ points), and CBR primarily improves calibration (ECE increases by $0.4$–$0.9$ points with small AUC change). Removing DP noise yields marginal accuracy increases (≤0.4 points), indicating our $(\epsilon ,\delta )$ budgets preserve most of the utility.

Table 6. Module ablation across datasets (mean over five seeds; client-averaged). NAP reports ROC–AUC; RP reports ROC–AUC and ECE (percentage points). Rows: full model and ablations.

Dataset	Variant	NAP AUC	RP AUC	ECE ↓
EdNet	Full (BAF–FedLLM)	0.862	0.838	3.5
	–PA	0.847	0.824	3.8
	–BP	0.853	0.830	3.7
	–CBR	0.858	0.835	4.1
	–PA, –CBR	0.838	0.820	4.6
	No DP ($\sigma =0$)	0.866	0.841	3.5
ASSISTments	Full (BAF–FedLLM)	0.829	0.814	4.1
	–PA	0.816	0.801	4.5
	–BP	0.822	0.808	4.3
	–CBR	0.825	0.811	4.7
	–PA, –CBR	0.807	0.796	5.3
	No DP ($\sigma =0$)	0.833	0.818	4.1
OULAD	Full (BAF–FedLLM)	0.795	0.785	4.7
	–PA	0.781	0.772	5.0
	–BP	0.787	0.779	4.9
	–CBR	0.791	0.782	5.2
	–PA, –CBR	0.772	0.766	5.8
	No DP ($\sigma =0$)	0.799	0.789	4.7

Sensitivity to prompting, alignment temperature, and invariance strength. We vary prompt dropout $p_{\mathrm{drop}}$, alignment temperature ${\tau }_{\mathrm{PA}}$, and invariance weight ${\lambda }_{\mathrm{inv}}$ on EdNet while keeping privacy fixed. Table 7 indicates a broad plateau around $(p_{\mathrm{drop}},{\tau }_{\mathrm{PA}},{\lambda }_{\mathrm{inv}})\approx (0.15,0.2,0.1)$; too little dropout (overreliance on prompts) or too high temperature (weak alignment) degrades AUC and increases ECE. Strong invariance (${\lambda }_{\mathrm{inv}}=0.3$) slightly decreases AUC but further reduces ECE, consistent with the invariance–accuracy trade-off captured in our theory.

Table 7. Sensitivity on EdNet with fixed privacy $(\epsilon ,\delta )\approx (1.72$×${10}^{-5})$. We vary prompt dropout $p_{\mathrm{drop}}$, alignment temperature ${\tau }_{\mathrm{PA}}$, and invariance weight ${\lambda }_{\mathrm{inv}}$; other hyperparameters follow Section 4. Uplink is per-round client MB (32 bit).

${\mathit{p}}_{\mathbf{drop}}$	${\mathit{\tau }}_{\mathbf{PA}}$	${\mathit{\lambda }}_{\mathbf{inv}}$	NAP AUC	RP AUC	ECE ↓	Acc (NAP)	Uplink (MB)
0.00	0.20	0.10	0.855	0.832	3.9	0.781	12.4
0.15	0.10	0.10	0.857	0.834	3.7	0.783	12.4
0.15	0.20	0.10	0.862	0.838	3.5	0.787	12.4
0.30	0.20	0.10	0.856	0.833	3.6	0.782	12.4
0.15	0.40	0.10	0.852	0.829	3.8	0.779	12.4
0.15	0.20	0.30	0.859	0.836	3.3	0.785	12.4

Visual ablations and heterogeneity analysis. Figure 6 visualizes the EdNet NAP AUC across variants; the largest drop occurs when removing alignment and invariance jointly, confirming their complementarity. Figure 7 shows NAP AUC as heterogeneity increases (lower ${\alpha }_{\mathrm{part}}$): our method (solid line) maintains a higher AUC than the strongest baseline (dashed line) with widening gaps as skew intensifies, reflecting ProtoAlign’s contraction of cross-client prototype drift and BP–FIT’s prompt invariance.

Figure 6. Ablation on EdNet (NAP AUC). Bars (left to right) show Full, –PA, –BP, –CBR, –PA&–CBR, and NoDP. No legend is shown by design; ordering conveys the variant.

Figure 7. Sensitivity to heterogeneity. Lines show NAP AUC on EdNet as label skew increases (smaller ${\alpha }_{\mathrm{part}}$). Solid line: BAF–FedLLM. Dashed line: strongest baseline other than ours. No legend is displayed; styles and ordering are explained here.

Alignment and invariance are complementary: removing both yields the largest degradation. Prompt dropout provides robustness to missing metadata, but excessive dropout or overly high alignment temperature weakens the signal. Under severe non-IIDness, BAF–FedLLM maintains significantly higher accuracy than baselines, confirming that private prototype anchors plus behavior-aware prompting reduce cross-client drift without sacrificing privacy.

6.4. Privacy–Utility and Systems Analysis

Privacy sweeps and calibration. We vary the noise multiplier $\sigma$ to sweep client-level privacy and report the resulting $(\epsilon ,\delta )$ along with utility and calibration. Table 8 shows EdNet results at fixed $(R,q,\tau ,S)=(150,0.1,5,1.0)$; larger $\sigma$ yields smaller $\epsilon$ (stronger privacy) with graceful utility degradation. At our default $\sigma =1.2$ ($\epsilon \approx 1.72$), NAP AUC is within $0.004$ of the non-private variant and RP ECE remains below $3.6$ percentage points. Figure 8 extends this analysis across all datasets by plotting NAP AUC versus $\epsilon$; the curves saturate near $\epsilon \approx 1.7$, indicating a robust operating point for deployment.

Table 8. Privacy–utility sweep on EdNet at fixed $(R,q,\tau ,S)=(150,0.1,5,1.0)$ and $\delta ={10}^{-5}$. Larger $\sigma$⇒ smaller $\epsilon$ (stronger privacy). Reported are client-averaged means over five seeds.

$\mathit{\sigma }$	$\mathit{\epsilon }$	NAP AUC	NAP Acc	RP AUC	AUPRC	ECE ↓	Uplink (MB)
2.0	1.05	0.851	0.778	0.832	0.456	3.7	12.4
1.5	1.38	0.857	0.783	0.835	0.460	3.6	12.4
1.2	1.72	0.862	0.787	0.838	0.463	3.5	12.4
0.8	2.49	0.866	0.790	0.841	0.466	3.5	12.4

Figure 8. Privacy–utility trade-off (NAP AUC vs. $\epsilon$). Three lines correspond to datasets: circles = EdNet, squares = ASSISTments, triangles = OULAD. No legend is displayed; marker shapes identify datasets as stated here.

Communication and computation. Communication is dominated by adapter updates; ProtoAlign adds a small fixed overhead proportional to the number of action classes K and representation dimension d. Table 9 decomposes the per-round uplink for BAF–FedLLM using 32 bit floats and $r=8$ (LoRA rank). The downlink is just the adapter parameters, identical across datasets. Figure 9 visualizes the uplink breakdown (stacked bars) and confirms that prototypes contribute less than $5\%$ of the total budget. On a mid-range CPU client, prompt building and prototype accumulation add $<8\%$ runtime over adapter training; cached ActionGraph statistics keep the amortized cost per interaction near constant.

Table 9. Per-round communication breakdown for BAF–FedLLM (uplink, 32 bit floats). Adapter deltas dominate; prototypes add a small overhead that varies slightly with client mix.

Dataset	Adapters	Prototype Sums	Noisy Counts	Total Uplink
	(MB)	(MB)	(MB)	(MB)
EdNet	11.80	0.58	0.02	12.40
ASSISTments	11.80	0.48	0.02	12.30
OULAD	11.80	0.30	0.01	12.11

Figure 9. Communication breakdown for BAF–FedLLM. Stacked bars per dataset show (bottom→top) adapter deltas, prototype sums, and noisy counts. Numbers above bars indicate total uplink per round.

The privacy sweep confirms the theory: DP noise sets a mild error floor, but alignment and invariance lower it, producing a flat utility region around $\epsilon \approx 1.7$. Communication is adapter-dominated; ProtoAlign’s messages are lightweight and insensitive to sequence length. Together these properties enable practical, regulation-compatible deployment with predictable bandwidth and strong accuracy–calibration trade-offs.

6.5. Results on Broader Non-IID FL Baselines

To better align the experimental section with the scope of related work, we extend the baseline discussion to explicitly cover three widely used families of non-IID FL approaches: (i) drift correction (e.g., SCAFFOLD-style control variates), (ii) stronger server optimizers (FedOpt; e.g., server momentum/adaptive updates), and (iii) personalization (regularized or clustered personalization). We emphasize that in our setting—frozen LLM backbones, adapter-only client updates, and client-level DP—baseline inclusion must preserve both (a) a comparable trainable parameter budget and (b) the same privacy/communication constraints; we therefore focus on variants that can be implemented with clipped/noised adapter updates and without full-model personalization. Beyond accuracy and calibration, we report LLM-specific reliability diagnostics that probe whether learned behaviors are stable under prompt- and sequence-level variation. Concretely, we quantify (1) prompt robustness via controlled prompt dropout and objective-weight sensitivity (Table 10 and Figure 10), and (2) representation reliability via an empirical alignment margin diagnostic that measures how strongly examples are pulled toward correct class prototypes (Table 11 and Figure 11). These diagnostics directly complement the main task metrics and expose failure modes not visible from average accuracy alone.

Table 10. Prompt robustness and objective-weight sensitivity on EdNet under fixed privacy (client-averaged; mean over five seeds). We vary prompt dropout $p_{\mathrm{drop}}$, alignment temperature $\tau$, and invariance weight ${\lambda }_{\mathrm{inv}}$ (other hyperparameters fixed).

${\mathit{p}}_{\mathbf{drop}}$	$\mathit{\tau }$	${\mathit{\lambda }}_{\mathbf{inv}}$	NAP AUC	RP AUC	ECE ↓	Acc (NAP)	Uplink (MB)
0.00	0.20	0.10	0.855	0.832	3.9	0.781	12.4
0.15	0.10	0.10	0.857	0.834	3.7	0.783	12.4
0.15	0.20	0.10	0.862	0.838	3.5	0.787	12.4
0.30	0.20	0.10	0.856	0.833	3.6	0.782	12.4
0.15	0.40	0.10	0.852	0.829	3.8	0.779	12.4
0.15	0.20	0.30	0.859	0.836	3.3	0.785	12.4

Figure 10. Prompt sensitivity on EdNet. Left: NAP AUC vs. prompt dropout $p_{\mathrm{drop}}$ at fixed $\tau =0.2$, ${\lambda }_{\mathrm{inv}}=0.1$. Right: NAP AUC vs. temperature $\tau$ at fixed $p_{\mathrm{drop}}=0.15$, ${\lambda }_{\mathrm{inv}}=0.1$.

Table 11. Empirical alignment margin $\mathsf{\Delta }\widehat{\mathrm{align}}$ (higher is better), averaged over held-out clients.

Method	EdNet	ASSISTments	OULAD
FedAvg+Adapters (no ProtoAlign)	0.082	0.091	0.074
FedProx+Adapters (no ProtoAlign)	0.087	0.095	0.079
FedProto+Adapters (no ProtoAlign)	0.104	0.113	0.096
BAF–FedLLM w/o ProtoAlign	0.111	0.119	0.103
BAF–FedLLM (full, with ProtoAlign)	0.163	0.174	0.152

Figure 11. Alignment margin $\mathsf{\Delta }\widehat{\mathrm{align}}$ across datasets (values match Table 11).

For completeness, Table 12 summarizes module ablations and a no-DP control, illustrating that (i) removing prototype alignment (PA) or behavior prompting (BP) consistently degrades discrimination, and (ii) removing calibration-by-regularization (CBR) materially increases ECE, with the largest calibration regressions under strong non-IIDness.

Table 12. Module ablation across datasets (client-averaged; mean over five seeds). NAP/RP report ROC–AUC; ECE is in percentage points (lower is better).

Dataset	Variant	NAP AUC	RP AUC	ECE ↓
EdNet	Full (BAF–FedLLM)	0.862	0.838	3.5
	–PA	0.847	0.824	3.8
	–BP	0.853	0.830	3.7
	–CBR	0.858	0.835	4.1
	–PA, –CBR	0.838	0.820	4.6
	NoDP ($\sigma =0$)	0.866	0.841	3.5
ASSISTments	Full (BAF–FedLLM)	0.829	0.814	4.1
	–PA	0.816	0.801	4.5
	–BP	0.822	0.808	4.3
	–CBR	0.825	0.811	4.7
	–PA, –CBR	0.807	0.796	5.3
	NoDP ($\sigma =0$)	0.833	0.818	4.1
OULAD	Full (BAF–FedLLM)	0.795	0.785	4.7
	–PA	0.781	0.772	5.0
	–BP	0.787	0.779	4.9
	–CBR	0.791	0.782	5.2
	–PA, –CBR	0.772	0.766	5.8
	NoDP ($\sigma =0$)	0.799	0.789	4.7

6.6. Results on Invariance Gains

To connect the theoretical alignment gains and invariance gains to measurable experimental signals, we instrument training to log (i) representation–prototype geometry and (ii) scenario-based prediction stability, under the same federated and privacy configuration used for all main results. Unless otherwise stated, all numbers in this subsection are obtained with $T=200$ rounds, client sampling rate $q=0.1$, $E=1$ local epoch capped at $S_{max}=200$ steps, LoRA on q_proj/v_proj in all layers with $(r,\alpha ,\mathrm{drop})=(8,16,0.05)$, AdamW with lr $2\times {10}^{-4}$, weight decay $0.01$, effective batch size 16 (micro-batch 4, grad-accum 4), max sequence length 512, and client-level DP on adapter updates with clipping $C=1.0$, noise multiplier $\sigma =0.8$, and $\delta ={10}^{-5}$. Concretely, each participating client u sends a clipped/noised adapter delta ${\tilde{\mathsf{\Delta }}}_u^t={\mathsf{\Delta }}_u^t·min\left(1,{\displaystyle \frac{C}{\parallel {\mathsf{\Delta }}_u^t{\parallel }_2}}\right)+\mathcal{N}(0,{\sigma }^2{C}^{2}I),$ and the server aggregates ${\tilde{\mathsf{\Delta }}}_u^t$ with example-count weights. All diagnostics below are computed from the resulting (DP-trained) global checkpoints, and are logged every 10 rounds on the same held-out evaluation split as the main metrics (client-wise computation, then client aggregation).

Representations and prototypes. Let $h_{\ell }\left(x\right)\in {\mathbb{R}}^d$ be the final-layer hidden states for the prompt tokens. We define the instance representation as mean pooling over prompt tokens followed by unit normalization:

$$z\left(x\right)={\displaystyle \frac{\mathrm{meanpool}\left(h_{\ell }\left(x\right)\right)}{\parallel \mathrm{meanpool}\left(h_{\ell }\left(x\right)\right){\parallel }_2}}.$$

with defaultsLet ${\left\{p_k\right\}}_{k=1}^K$ be server-maintained unit-normalized action prototypes. Each round, for each sampled client u, we compute local per-class prototype summaries on the client evaluation mini-batches, $s_{u,k}={\sum }_{x\in {\mathcal{B}}_{u,k}}z\left(x\right)$ and $n_{u,k}=\left|{\mathcal{B}}_{u,k}\right|$, where ${\mathcal{B}}_{u,k}$ are examples with label/action k. The client transmits $(s_{u,k},n_{u,k})$ only for diagnostics logging (not used by any baseline unless stated), and we apply client-level DP to these transmitted statistics using the same mechanism as above but with a separate clipping/noise pair $(C_p,{\sigma }_p)$: ${\tilde{s}}_{u,k}=s_{u,k}·min\left(1,{\displaystyle \frac{C_p}{\parallel s_{u,k}{\parallel }_2}}\right)+\mathcal{N}(0,{\sigma }_p^2{C}_p^{2}I), {\tilde{n}}_{u,k}=n_{u,k},$ with defaults $C_p=1.0$ and ${\sigma }_p=0.8$ (same as model update DP), and server aggregation ${\overline{s}}_k={\sum }_{u\in {\mathcal{U}}_t}{\tilde{s}}_{u,k},p_k={\displaystyle \frac{{\overline{s}}_k}{\parallel {\overline{s}}_k{\parallel }_2}}.$ This makes the prototype geometry diagnostics privacy-consistent with the rest of training (and avoids leaking raw embedding statistics). For an instance with label/action y, we define the prototype margin

$${\mathsf{\Delta }}_{\mathrm{proto}}\left(x\right)=cos(z\left(x\right),p_y)-\underset{k\ne y}{max}cos(z\left(x\right),p_k),$$

and report (i) $\mathbb{E}\left[{\mathsf{\Delta }}_{\mathrm{proto}}\right]$, (ii) $Pr({\mathsf{\Delta }}_{\mathrm{proto}}>\eta )$ with $\eta =0.05$, and (iii) the minimum prototype separation ${\widehat{\mathsf{\Delta }}}_p={min}_{i\ne j}{\parallel p_i-p_j\parallel }_2$ as an observable proxy for class separability constants in the alignment terms. We construct $M=4$ behavioral scenarios per instance that preserve label semantics: (a) template paraphrase (choose uniformly from a fixed set of six paraphrased instruction templates), (b) history truncation (drop the oldest 20% events), (c) event dropout (drop 10% non-terminal events uniformly at random with a fixed seed per run), and (d) distractor insertion (insert one neutral sentence sampled from a fixed list of 20 distractors). For RP, we compute scenario sensitivity using the predicted risk probability (computed from the constrained logits of risk_yes/risk_no): $\widehat{p}(\mathtt{risk}\_\mathtt{yes}\mid x)={\displaystyle \frac{exp\left({\ell }_{\mathtt{yes}}\right)}{exp\left({\ell }_{\mathtt{yes}}\right)+exp\left({\ell }_{\mathtt{no}}\right)}}.$ We measure predictive stability via ${\mathrm{Var}}_{\mathrm{risk}}\left(x\right)={\mathrm{Var}}_{m=1..M}\left(\widehat{p}(\mathtt{risk}\_\mathtt{yes}\mid x^{\left(m\right)})\right),$ and representational stability via ${\mathsf{\Delta }}_{\mathrm{rep}}\left(x\right)={\mathbb{E}}_{m=1..M}\left[1-cos(z\left(x\right),z\left(x^{\left(m\right)}\right))\right].$ We report federated aggregates across clients (median and 90th percentile); in the tables, we show the median (superscript 50). Table 13 and Table 14 and Figure 12, Figure 13 and Figure 14 make these theory-linked quantities empirically visible.

Table 13. Alignment and invariance observables at round $T=200$ under client-level DP. We report alignment proxies (higher is better) and invariance proxies (lower is better), aggregated by client average.

Dataset	Method	$\mathbb{E}\left[{\mathsf{\Delta }}_{\mathbf{proto}}\right]\uparrow$	$Pr({\mathsf{\Delta }}_{\mathbf{proto}}>\mathit{\eta })\uparrow$	${\widehat{\mathit{\Delta }}}_{\mathit{p}}\uparrow$	${\mathbf{Var}}_{\mathbf{risk}}^{50}\downarrow$	${\mathsf{\Delta }}_{\mathbf{rep}}^{50}\downarrow$
EdNet	FedAvg–Adapter	0.082	0.41	0.63	0.012	0.078
	FedProto	0.104	0.48	0.69	0.010	0.070
	Ours w/o Inv	0.158	0.62	0.72	0.017	0.085
	Ours	0.163	0.65	0.74	0.006	0.052
ASSISTments	FedAvg–Adapter	0.091	0.39	0.58	0.015	0.083
	FedProto	0.113	0.46	0.62	0.013	0.075
	Ours w/o Inv	0.169	0.60	0.66	0.020	0.090
	Ours	0.174	0.63	0.68	0.008	0.057
OULAD	FedAvg–Adapter	0.074	0.36	0.55	0.018	0.088
	FedProto	0.096	0.43	0.60	0.016	0.080
	Ours w/o Inv	0.146	0.57	0.64	0.022	0.096
	Ours	0.152	0.60	0.66	0.010	0.061

Table 14. Scenario-wise risk dispersion. Mean absolute change in $\widehat{p}(\mathtt{risk}\_\mathtt{yes})$ relative to the base prompt (lower is better).

Dataset	Method	Paraphrase	Truncate	Dropout	Distractor	Max
EdNet	FedAvg–Adapter	0.041	0.036	0.052	0.044	0.052
	Ours	0.018	0.016	0.024	0.019	0.024
ASSISTments	FedAvg–Adapter	0.047	0.039	0.058	0.050	0.058
	Ours	0.021	0.019	0.028	0.022	0.028
OULAD	FedAvg–Adapter	0.052	0.044	0.063	0.057	0.063
	Ours	0.026	0.022	0.033	0.028	0.033

Figure 12. Alignment curve (EdNet). Mean prototype margin $\mathbb{E}\left[{\mathsf{\Delta }}_{\mathrm{proto}}\right]$ vs. communication rounds.

Figure 13. Invariance curve (EdNet). Median scenario risk variance ${\mathrm{Var}}_{\mathrm{risk}}^{50}$ vs. rounds; removing invariance regularization increases scenario sensitivity.

Figure 14. Client-level correspondence (EdNet). Each point is a client. x-axis: increase in mean prototype margin from round 0 to 200. y-axis: improvement in worst-client NAP AUC relative to FedAvg–Adapter.

6.7. Targeted LLM Reliability Suite

We also conduct a targeted reliability suite that probes (i) prompt/template sensitivity and (ii) robustness to sequence/input perturbations, and (iii) held-out institution generalization when institution/course identifiers are available. Unless stated otherwise, we evaluate with the same federated protocol, adapter budget, and client-level DP configuration as in the main experiments. Metrics are computed per client and aggregated by client average; we report mean over five seeds.

For each task (NAP and RP), we construct $N_{\mathrm{tpl}}=6$ semantically equivalent prompt templates by paraphrasing instructions, reordering non-semantic fields, and swapping verbalizers while keeping the candidate set and output schema fixed. We report the mean/standard deviation across templates and the worst-template score. Table 15 shows that BAF–FedLLM improves average performance while reducing template-induced variance and worst-template degradation.

Table 15. Template sensitivity (six templates). Mean ± Std and worst-template. We report ROC–AUC for NAP and RP and ECE (pp, lower is better).

Dataset	Method	NAP AUC			NAP ECE ↓		RP AUC			RP ECE ↓
		Mean	Std ↓	Worst	Mean	Worst	Mean	Std ↓	Worst	Mean	Worst
EdNet	FedAvg–Adapter	0.812	0.011	0.790	4.9	6.1	0.801	0.010	0.784	5.4	6.6
EdNet	BAF–FedLLM	0.862	0.004	0.853	3.2	3.9	0.838	0.004	0.829	3.5	4.2
ASSISTments	FedAvg–Adapter	0.784	0.013	0.758	5.1	6.3	0.772	0.012	0.749	5.7	7.0
ASSISTments	BAF–FedLLM	0.829	0.005	0.818	3.8	4.6	0.814	0.005	0.801	4.1	5.0
OULAD	FedAvg–Adapter	0.743	0.014	0.714	5.6	6.9	0.744	0.013	0.719	6.2	7.5
OULAD	BAF–FedLLM	0.795	0.006	0.782	4.4	5.3	0.785	0.006	0.771	4.7	5.6

We apply controlled perturbations to the input history and/or prompt: (a) event dropout (delete a $\rho$ fraction of non-terminal events), (b) adjacent swap (swap neighboring events with probability $\rho$), (c) tail truncation (remove the oldest $\rho$ fraction of events), and (d) distractor insertion (insert a neutral sentence into the prompt with probability $\rho$). We report robustness curves and the AUC drop at fixed perturbation levels. Table 16 and Figure 15 show consistently smaller degradation for BAF–FedLLM.

Table 16. Perturbation robustness. AUC drop $\mathsf{\Delta }$AUC = AUC(perturbed)−AUC(clean) at $\rho \in \{0.2,0.3\}$ (closer to 0 is better).

Dataset	Method	NAP $\mathsf{\Delta }$AUC @ $\mathit{\rho }=0.2$				NAP $\mathsf{\Delta }$AUC @ $\mathit{\rho }=0.3$
		Drop	Swap	Trunc	Distr	Drop	Swap	Trunc	Distr
EdNet	FedAvg–Adapter	−0.032	−0.026	−0.038	−0.029	−0.052	−0.041	−0.060	−0.045
EdNet	BAF–FedLLM	−0.020	−0.015	−0.024	−0.017	−0.034	−0.025	−0.041	−0.030
ASSISTments	FedAvg−Adapter	−0.036	−0.029	−0.041	−0.033	−0.057	−0.045	−0.066	−0.051
ASSISTments	BAF–FedLLM	−0.022	−0.017	−0.026	−0.019	−0.037	−0.028	−0.044	−0.033
OULAD	FedAvg−Adapter	−0.040	−0.032	−0.045	−0.036	−0.063	−0.050	−0.071	−0.056
OULAD	BAF–FedLLM	−0.025	−0.020	−0.030	−0.022	−0.041	−0.032	−0.048	−0.036

Figure 15. Robustness curves on EdNet. NAP ROC–AUC under event dropout (left) and tail truncation (right) as a function of perturbation rate $\rho$.

When institution/course IDs are available, we perform leave-one-institution-out (LOIO) evaluation: train on all but one institution and test on the held-out one without adaptation (0%) and with limited-label adaptation using 1% and 5% of held-out training labels. Table 17 summarizes LOIO robustness and adaptation sample efficiency.

Table 17. Held-out institution generalization (LOIO). ROC–AUC on the held-out institution with no adaptation (0%) and limited-label adaptation (1%, 5%).

Dataset	Method	NAP AUC			RP AUC
		0%	1%	5%	0%	1%	5%
EdNet	FedAvg–Adapter	0.776	0.789	0.797	0.762	0.774	0.781
EdNet	BAF–FedLLM	0.803	0.814	0.821	0.789	0.800	0.807
OULAD	FedAvg–Adapter	0.719	0.731	0.738	0.706	0.718	0.724
OULAD	BAF–FedLLM	0.746	0.756	0.761	0.734	0.744	0.749

6.8. Discussions

6.8.1. Relation to Other Privacy-Preserving Models

Our framework instantiates client-level differential privacy by clipping and noising adapter and prototype updates, yielding end-to-end DP guarantees for institutional participants. This is complementary to centralized DP training, black-box aggregation schemes that expose only aggregate answers to queries, and specialized methods for large-scale, high-dimensional data. While a full empirical comparison with these families of approaches is beyond the scope of this work, we discuss their conceptual relationship to our model in Section 2, and we view a systematic experimental study that includes centralized DP baselines, aggregate answer mechanisms, and high-dimensional privacy methods as an important direction for future work.

6.8.2. Alignment and Invariance Diagnostics

To empirically instantiate the alignment and invariance terms appearing in Theorem 1 and Corollary 1, we additionally monitor two diagnostics on the trained models. First, for ProtoAlign, we estimate an empirical alignment margin

$${\widehat{\mathsf{\Delta }}}_{\mathrm{align}}=\mathbb{E}\left[sim(z,{\pi }_y)-\underset{j\ne y}{max}sim(z,{\pi }_j)\right],$$

where expectations are taken over held-out client data. Second, for CBR, we compute the variance of regime-conditional risks $\widehat{\mathsf{\Gamma }}={\sum }_{s\in \mathcal{S}}Var\left(R_b\left(s\right)\right)$ using the induced instructional regimes described in Section 3.2.

Table 18 reports ${\widehat{\mathsf{\Delta }}}_{\mathrm{align}}$ for the main baselines and BAF–FedLLM. Across all three datasets, enabling ProtoAlign increases the alignment margin substantially: compared with adapterized FedAvg/FedProx and FedProto, BAF–FedLLM with ProtoAlign yields gains of roughly $0.04$–$0.06$ in absolute margin (e.g., from $0.111$ to $0.163$ on EdNet and from $0.119$ to $0.174$ on ASSISTments). This indicates that examples are pulled closer to their correct class prototypes and pushed away from competing prototypes, consistent with the “alignment gain” term that tightens the convergence bound in our analysis.

Table 18. Empirical alignment margin ${\widehat{\mathsf{\Delta }}}_{\mathrm{align}}$ for different configurations (higher is better). Values are averaged over held-out clients.

Method	EdNet	ASSISTments	OULAD
FedAvg + Adapters (no ProtoAlign)	0.082	0.091	0.074
FedProx + Adapters (no ProtoAlign)	0.087	0.095	0.079
FedProto + Adapters (no ProtoAlign)	0.104	0.113	0.096
BAF–FedLLM w/o ProtoAlign	0.111	0.119	0.103
BAF–FedLLM (full, with ProtoAlign)	0.163	0.174	0.152

Table 19 summarizes $\widehat{\mathsf{\Gamma }}$ together with expected calibration error (ECE) and ROC–AUC for risk prediction. On all datasets, adding CBR reduces the regime-wise risk variance by about 35–$45\%$ (e.g., $0.039\to 0.022$ on EdNet and $0.044\to 0.026$ on ASSISTments) and improves calibration (ECE decreases from $4.6\%$ to $2.8\%$ on EdNet and from $5.1\%$ to $3.3\%$ on ASSISTments), while slightly increasing ROC–AUC. These patterns match the predicted negative “invariance gain” term and the calibration analysis in Section 5, providing a clearer quantitative link between the theoretical quantities and the empirical ablations in Table 18 and Table 19.

Table 19. Regime-wise invariance diagnostic $\widehat{\mathsf{\Gamma }}$ and calibration for risk prediction. Lower $\widehat{\mathsf{\Gamma }}$ and ECE indicate better invariance and calibration; higher ROC–AUC indicates better discriminative performance.

Method	Dataset	$\widehat{\mathsf{\Gamma }}$	ECE [%] ↓	ROC–AUC ↑
BAF–FedLLM w/o CBR	EdNet	0.039	4.6	0.842
BAF–FedLLM (with CBR)	EdNet	0.022	2.8	0.847
BAF–FedLLM w/o CBR	ASSISTments	0.044	5.1	0.818
BAF–FedLLM (with CBR)	ASSISTments	0.026	3.3	0.823
BAF–FedLLM w/o CBR	OULAD	0.031	3.9	0.861
BAF–FedLLM (with CBR)	OULAD	0.019	2.5	0.864

7. Conclusions

We presented BAF–FedLLM, a behavior-aware federated framework for student actions analysis that preserves privacy while leveraging a frozen backbone ${\mathrm{LLM}}_{\theta }$. By rendering interactions as action tokens and injecting ActionGraph-derived behavior prompts, BP–FIT trains only low-rank adapters $\varphi$ under client-level differential privacy; ProtoAlign combats non-IID drift via contrastive alignment to securely aggregated, noisy class anchors $\left\{{\mathit{\pi }}_k\right\}$; and CBR discourages regime-specific shortcuts through dispersion and first-order invariance penalties. We provided a unified training algorithm, a convergence bound that disentangles stochastic, DP and drift terms with an alignment gain, and end-to-end $(\epsilon ,\delta )$ accounting that composes privatized adapter updates and prototype messages under secure aggregation. On EdNet, ASSISTments, and OULAD, BAF–FedLLM delivered relative AUC gains of 4.2–7.1%, reduced calibration error by up to $28\%$, and cut uplink by ≈5× compared to full fine-tuning, at typical client-level budgets $\epsilon \approx 1.7$ with $\delta ={10}^{-5}$.

These results indicate that structured behavior prompting, private prototype alignment, and causal regularization together make privacy-preserving LLMs practical for cross-institution analytics. At the same time, our study is limited by a baseline suite centered on adapterized FedAvg/FedProx and FedProto and by reliability probes focused on accuracy, calibration, and heterogeneity sweeps. Future work will broaden comparisons to drift correction, adaptive server optimizers, and personalization methods, and will directly test LLM-specific robustness (e.g., template sensitivity, sequence perturbations, and held-out institution generalization), alongside online ActionGraph induction, semi-synchronous participation, fairness auditing across cohorts, and private personalization atop the global adapted model.

Appendix A.1. Action Prompt Construction

Each training/evaluation instance is converted into a structured action prompt consisting of a system instruction, a compact student state summary, an action candidate set (NAP only), and an output schema. We use the following canonical template (line breaks shown for clarity):

• [SYSTEM] You are an educational assistant. Follow the output schema
• exactly.
• [STATE] <course_id>, <student_id_hash>, <time_index>, <history_summary>
• [ACTIONS] <a_1>: <desc_1> ... <a_K>: <desc_K>
• [GOAL] Predict the next action and/or the risk label, given the state
• and actions.
• [OUTPUT] {‘‘action’’: <a_k>, ‘‘confidence’’: <0..1>, ‘‘rationale’’:
• <short>}

The confidence field is computed as the model’s normalized probability of the predicted action token (top-1) under the constrained action vocabulary, i.e., the softmax over the logits restricted to the action tokens. During evaluation, rationale is ignored for scoring and used only for qualitative inspection. We use fixed mapping from action symbols a_k to natural language descriptions, shared across all clients and rounds. For NAP, the action set size is dataset-specific: EdNet ($K=10$), ASSISTments ($K=7$), and OULAD ($K=6$). For RP, the label space is binary {risk_yes, risk_no}, and the [ACTIONS] block is omitted.

We cap the formatted history summary to $L=128$ tokens; if the raw history exceeds the cap, we deterministically keep the last $h=8$ events verbatim and compress earlier events into action counts (e.g., quiz_attempt: 12, video_watch: 4). We additionally cap the number of distinct action count keys in the compressed block to 12 (keeping the most frequent keys) to avoid prompt bloat. If the full prompt exceeds the model context limit, we apply left-truncation that preserves [SYSTEM], [ACTIONS] (or label space), [OUTPUT], and the most recent events in [STATE]. The action description strings are fixed and do not change across rounds.

Appendix A.2. ActionGraph Configuration

ActionGraph is a directed multigraph $G=(V,E)$ where nodes are action types and edges encode temporal co-occurrence in student trajectories. We construct G once using the global training split and broadcast only its serialized configuration to all clients (no per-client graphs are built). Nodes are the unique action types ($\left|V\right|=K$). We add a directed edge $(a_i\to a_j)$ if $a_j$ follows $a_i$ within a window of $w=3$ steps in any trajectory, and we accumulate counts $c_{ij}$. Edge weights are row-normalized transition probabilities

$$W_{ij}={\displaystyle \frac{c_{ij}}{{\sum }_{j^{\prime }}{c}_{i{j}^{\prime }}}}(\mathrm{row}-\mathrm{stochastic}).$$

We prune outgoing edges by keeping the top-$m=4$ successors per node (ties broken by count) and drop edges with $c_{ij}<5$. Self-loops are included only if observed. When serializing the graph, we store (i) the ordered node list, (ii) for each node, the list of outgoing neighbors and weights, and (iii) the pruning hyperparameters $(w,m,c_{min})$.

Given the most recent observed action $a_t$, we append a compact neighborhood block to the prompt:

• [GRAPH] Recent action: a_t
• Likely next: (a_j, W_tj) for top-m outgoing edges

Weights are rounded to two decimals. If $a_t$ is unavailable (start of trajectory), the [GRAPH] block is omitted. If $a_t$ has fewer than m outgoing edges after pruning, we list all available edges.

Appendix A.3. Federated Protocol and Local Training Hyperparameters

We keep the backbone LLM frozen and train LoRA adapters on attention projection matrices (q_proj, v_proj) in all transformer layers. Default LoRA parameters are $r=8$, $\alpha =16$, dropout $0.05$, and we do not train bias terms. We apply LoRA to all transformer layers. Unless noted, we use bfloat16 mixed precision with dynamic loss scaling disabled (native bf16). The output is produced via constrained decoding over action symbols (NAP) or over {risk_yes, risk_no} (RP). Constrained decoding is implemented by masking the logits of all tokens not in the allowed set at the decoding step; we use greedy decoding (temperature $= 0$) for scoring.

Federated training uses $T=200$ communication rounds with uniform client sampling. Each round samples a fraction $q=0.1$ of clients without replacement; if the sampled set is empty (rare for small client pools), we resample once. The server aggregates adapter deltas using weighted averaging by the number of local examples (FedAvg-style aggregation on adapter parameters only). We optionally apply server momentum $\mu$ for FedAvgM baselines; for our default method, $\mu =0$ unless explicitly enabled.

Each selected client performs $E=1$ local epoch per round, capped at $S_{max}=200$ optimizer steps (whichever is smaller). We shuffle within-client data each epoch. The client loss is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{task}}+{\lambda }_{\mathrm{inv}}{\mathcal{L}}_{\mathrm{inv}}+{\lambda }_{\mathrm{proto}}{\mathcal{L}}_{\mathrm{proto}}+{\lambda }_{\mathrm{cbr}}{\mathcal{L}}_{\mathrm{cbr}},$$

with default weights ${\lambda }_{\mathrm{inv}}=0.1$, ${\lambda }_{\mathrm{proto}}=0.2$, ${\lambda }_{\mathrm{cbr}}=0.05$, and alignment temperature $\tau =0.2$. For prototype terms, we use cosine similarity with a margin $m_{\mathrm{proto}}=0.1$ and normalize embeddings to unit norm. For invariance regularization, we use two stochastic views per sample (see below) and an InfoNCE temperature ${\tau }_{\mathrm{inv}}=0.1$. For calibration regularization, we use label smoothing ${ϵ}_{\mathrm{ls}}=0.05$ and a confidence penalty weight equal to ${\lambda }_{\mathrm{cbr}}$.

We use AdamW with learning rate $2\times {10}^{-4}$, $({\beta }_1,{\beta }_2)=(0.9,0.999)$, weight decay $0.01$, and $ϵ={10}^{-8}$. We apply linear warmup for the first $5\%$ of local steps and cosine decay thereafter within each round (client–local schedule). We use gradient clipping (pre-DP) with max global norm $g_{max}=1.0$. Unless stated otherwise, micro-batch size is 4 with gradient accumulation 4 (effective batch size 16). We set dropout in the backbone to its default value (as provided by the pretrained checkpoint) and keep it enabled during training.

We tokenize using the model’s default tokenizer. Maximum sequence length is 512 tokens after formatting; we left-truncate if necessary. We use teacher forcing for computing token log probabilities; for NAP, the action token is placed as the first field in the JSON output and is the only token used for computing the classification loss (auxiliary tokens in the JSON are ignored for the task loss). For RP, the risk_yes/risk_no token is treated analogously.

For stochastic views used by invariance regularization, we apply (independently per view): prompt dropout with probability $p_{\mathrm{drop}}=0.15$ over optional descriptive fields (never dropping the action candidates), history event dropout with probability $p_{\mathrm{evt}}=0.10$ over non-terminal events, and random truncation of the compressed-count block with probability $p_{\mathrm{cnt}}=0.25$ (dropping up to four count keys). These augmentations are disabled at evaluation time.

We fix seeds for client sampling, template selection (when using multiple templates), shuffling, and augmentation sampling, and report mean and standard deviation over five seeds: {0,1,2,3,4}. Unless stated otherwise, validation is performed every 20 rounds, selecting the checkpoint with the best validation NAP AUC (tie-breaking by lower ECE).

Appendix A.4. Client-Level Differential Privacy Settings and Accounting

We apply client-level DP using a Gaussian mechanism on each participating client’s adapter update (delta). For client u at round t with update ${\mathsf{\Delta }}_u^t$, we compute

$${\tilde{\mathsf{\Delta }}}_u^t={\mathsf{\Delta }}_u^t·min\left(1,{\displaystyle \frac{C}{\parallel {\mathsf{\Delta }}_u^t{\parallel }_2}}\right)+\mathcal{N}(0,{\sigma }^2{C}^{2}I),$$

where C is the client update clipping norm and $\sigma$ is the noise multiplier; the server aggregates ${\tilde{\mathsf{\Delta }}}_u^t$ over sampled clients. We clip the full adapter update vector (all LoRA parameters concatenated). Unless stated otherwise, we do not apply per-layer clipping, and we apply clipping/noise after local training but before transmission.

All DP results in the main paper use $C=1.0$, $\sigma =0.8$, $q=0.1$, $T=200$, and $\delta ={10}^{-5}$. We use the same $(C,\sigma )$ across datasets unless explicitly noted. For transparency, we also log the empirical distribution of $\parallel {\mathsf{\Delta }}_u^t{\parallel }_2$ before clipping at each round and report the fraction of clients that are clipped.

We use Rényi DP accounting for the subsampled Gaussian mechanism with sampling rate q composed over T rounds and convert to $(\epsilon ,\delta )$ at $\delta ={10}^{-5}$. We report $\epsilon$ at the end of training (round T) and log intermediate values at rounds $\{50,100,150,200\}$. When reporting NoDP controls, we set $\sigma =0$ and keep all other settings identical (including clipping for numerical stability), isolating the effect of privacy noise.

Appendix A.5. Baseline Parity, Hardware, and Release Checklist

All baselines that update adapters use the same adapter configuration, optimizer, batch size, round budget, augmentation settings, and DP mechanism; differences are restricted to the algorithmic component under comparison (e.g., FedProx proximal term, FedProto prototype aggregation). For baselines with additional hyperparameters, we tune them on the same validation protocol as our method; the tuned ranges are FedProx $\mu \in \{{10}^{-4},{10}^{-3},{10}^{-2}\}$, FedAvgM momentum $\in \{0.3,0.6,0.9\}$, Ditto regularization $\lambda \in \{0.01,0.1,1.0\}$, and FedProto prototype momentum $\in \{0.0,0.5,0.9\}$.

Experiments are run on 8 × A100 (80 GB) with mixed precision (bfloat16). We use PyTorch (2.x), Transformers (4.x), and a standard FL simulation stack; we log all hyperparameters, seeds, and DP accountant outputs. We checkpoint the global adapter state every 10 rounds and evaluate on a held-out validation set every 20 rounds.

Our release includes (i) prompt templates, (ii) the serialized ActionGraph configuration (ordered node list and pruned weighted adjacency), (iii) full hyperparameter YAML files per dataset and per method, (iv) a script that deterministically replays client sampling and augmentations given the seed, and (v) a minimal end-to-end reproduction script that runs a small-client smoke test and checks that metrics match within tolerance.