DR-Transformer: A Dual-Regularized Transformer Combining Sparse Attention and Supervised Contrastive Learning for Interpretable Stress Detection in Social Media Text

Abstract

Automatic detection of stress in social media text holds promise for supporting digital mental health, but most existing Transformer-based approaches are opaque and computationally demanding. This work presents DR-Transformer, a Dual-Regularized Transformer that combines two complementary mechanisms: (i) a group sparsity penalty ($L_{2,1}/L_2$ elastic net) applied to the query and key projection matrices of every attention head, which encourages whole-row sparsity, producing more concentrated and inspectable attention patterns; (ii) a supervised contrastive loss on the [CLS] projection, which organizes the latent space according to the stress label. The architecture is intentionally lightweight (six layers, eight heads, 256-dim embeddings; ∼9.5 M parameters) and runs entirely on consumer-grade hardware (NVIDIA GTX 1660, 6 GB). Experiments on the publicly available Dreaddit dataset (binary stress classification, 2838 train/715 test segments) compare DR-Transformer against Logistic Regression, BiLSTM, a Standard Transformer of identical architecture, and MentalBERT. Across five seeded runs, DR-Transformer (Full) reaches $F_1=0.876$ (bootstrap 95% CI $0.852$–$0.898$), outperforming the Standard Transformer ($F_1=0.842$; McNemar $p<0.001$ with Bonferroni correction) and performing comparably to the much larger MentalBERT ($F_1=0.879$; $p=0.421$). Sparse regularization increases the fraction of near-zero attention weights (below $0.01$) from $0.215$ to $0.682$, while the supervised contrastive loss improves the silhouette score of [CLS] embeddings from $0.312$ to $0.483$. Dual regularization thus combines accuracy, efficiency, and structurally induced attention concentration in a single model which can be trained without specialized infrastructure. We use the term “interpretable” throughout in this restricted, structural sense—to refer to concentrated and inspectable attention—rather than in the sense of established causal or mechanistic faithfulness; this is only partially and indirectly supported by our token deletion analysis.

1. Introduction

The proliferation of mental health discourse on platforms such as Reddit, Twitter, and Facebook has created a real-time data source for public health research [1]. Natural language processing (NLP) and deep learning have shown promise in automatically identifying signs of stress, depression, anxiety, and suicidal ideation from such text [2,3,4]. Stress in particular is a near-universal experience, the detection of which at scale could support targeted wellbeing interventions, content moderation, and epidemiological monitoring.

Despite this promise, two practical barriers limit potential use in clinical or content-moderation settings. First, the opacity of large Transformer models limits trust and accountability [5,6]; clinicians and content moderators need to understand why a system flags a post before acting on it. Second, many state-of-the-art models—ranging from MentalBERT [7] to MentaLLaMA [8]—are expensive to train and serve, restricting their use in resource-constrained settings.

We propose DR-Transformer, a Dual-Regularized Transformer that targets both barriers. Our contributions are as follows:

1.

A group-sparse attention regularizer ($L_{2,1}/L_2$ elastic net) on the query and key projection matrices that pushes whole input dimensions toward zero, yielding more concentrated and inspectable attention patterns—a structural property we distinguish from full faithfulness, which would require additional perturbation-based evaluation (Section 4.3).

2.

A supervised contrastive loss [9] on the [CLS] projection that organizes the latent space by class, improving generalization and visual separability.

3.

A lightweight design (six layers, eight heads, ∼9.5 M parameters) that we train and evaluate end-to-end on a single NVIDIA GTX 1660 (6 GB).

4.

An empirical study on Dreaddit [10] with five seeded runs, bootstrap confidence intervals, McNemar significance tests (reported both per seed and pooled), and ablations isolating each regularizer.

We note that group sparsity penalties and supervised contrastive losses are individually established techniques. The novelty of DR-Transformer lies in three specific aspects: (i) the application of an $L_{2,1}/L_2$ group elastic-net penalty directly to the query and key projection matrices ($W_Q$, $W_K$)—rather than to activations or classifier outputs—which induces row-level sparsity over input dimensions and propagates to token-level attention weights. (ii) The joint optimization of both regularizers within a single lightweight encoder trained from scratch without pretrained weights. (iii) The empirical demonstration, with rigorous statistical testing and ablations, that this combination achieves a competitive accuracy–efficiency–inspectability trade-off on the mental health stress detection task, a high-stakes domain where computational accessibility and model transparency are both critical.

Section 2 reviews related work. Section 3 describes the architecture and training. Section 4 reports the classification, interpretability, and efficiency results. Section 5 discusses implications and limitations, and Section 6 concludes.

2. Related Work

2.1. Mental Health Detection from Social Media

Early work used lexicon-based features and classical classifiers [2,3]. CLPsych shared tasks [11,12] and the Dreaddit benchmark [10] established standard datasets for distinguishing distressed users from control users. Recent deep learning approaches include time-aware attention for suicidal ideation detection [13], dual-context language models [14], and knowledge-aware attention [15]. Domain-adapted pretrained encoders—most notably MentalBERT [7]—substantially improve downstream performance by continued pretraining on mental health subreddits. The latest wave fine-tunes large language models for mental health analysis, e.g., MentaLLaMA [8], but their multi-billion-parameter scale is incompatible with consumer hardware.

More recently, Kyrou et al. [16] provided a comprehensive survey of deep learning approaches for stress detection, highlighting the ongoing challenge of balancing model capacity with interpretability and deployment feasibility. Soufleri and Ananiadou [17] proposed a multimodal framework that fuses synthesized visual representations with text for stress detection on Dreaddit, achieving strong accuracy but at significantly higher computational costs compared to text-only approaches. Zhuang et al. [18] introduced a BERT-fused model combining LDA topic features with contextual embeddings for psychological stress detection from social media, demonstrating the benefit of integrating structural topic information. Compared to these concurrent approaches, DR-Transformer distinguishes itself by simultaneously targeting attention concentration and computational efficiency within a single lightweight model trainable on consumer hardware—a dimension that remains underexplored in the recent literature.

2.2. Sparse and Efficient Attention

Efficient-attention methods aim to lower the $\mathcal{O}\left(N^2\right)$ cost of self-attention. Longformer [19] and BigBird [20] use structured sparsity (windowed plus global tokens). Reformer [21] replaces dot-product attention with locality-sensitive hashing. These approaches target computational sparsity. Our work instead induces sparsity on the projection matrices ($W_Q$, $W_K$) via a group $L_{2,1}$ penalty, targeting attention concentration for inspection. The two strategies are orthogonal and could be combined.

2.3. Contrastive Learning for Text

Self-supervised contrastive learning [22,23,24] produces high-quality sentence embeddings without labels. Supervised contrastive learning [9] extends the framework to labeled data, pulling same-class samples together and pushing different-class apart, and has been shown to outperform pure cross-entropy on text classification [25]. We adopt this objective on the [CLS] projection of our Transformer, providing a complementary signal to the sparse attention regularizer.

3. Materials and Methods

3.1. Overall Architecture

DR-Transformer is an encoder-only Transformer [26] that produces (i) a class logit and (ii) a normalized contrastive embedding from the same [CLS] representation. The pipeline consists of four stages: input text is tokenized with BERT WordPiece [27]; embeddings are added to learned positional encodings; a stack of $L=6$ Transformer layers processes the sequence, where each layer’s attention is regularized by a group elastic-net penalty on $W_Q$ and $W_K$; the final [CLS] embedding feeds a binary classifier and a 2-layer MLP projection used by the supervised contrastive loss. Figure 1 provides a schematic overview.

3.2. Input Representation

Let a post be tokenized into a sequence of WordPiece tokens. A special classification token [CLS]—the final hidden state of which serves as the aggregate sequence representation for classification—is prepended, and a separator token [SEP] is appended; sequences are padded or truncated to length $N=128$ (95th percentile of Dreaddit segment lengths). The input is:

$$X=\left[e_{\left[\mathrm{CLS}\right]},e_1,\dots ,e_N,e_{\left[\mathrm{SEP}\right]}\right]+P,$$

(1)

where $e_i\in {\mathbb{R}}^{d_{\mathrm{model}}}$ is a token embedding and P is a learned positional encoding matrix.

Token embeddings are randomly initialized from $\mathcal{N}(0,0.02)$ and trained from scratch jointly with all other model parameters. DR-Transformer does not use pretrained embeddings from BERT or any other external model; the WordPiece vocabulary (30,522 tokens) and tokenizer are reused from BERT for compatibility, but the embedding matrix is not initialized from BERT weights. This distinguishes DR-Transformer from fine-tuning approaches such as MentalBERT. The total parameter count of ∼9.48M includes the embedding matrix ($30,522\times 256\approx 7.8$ M), encoder layers (≈1.5 M), and classifier plus projection heads (≈0.18 M).

3.3. Transformer Encoder with Sparse Attention

For head h in layer l, queries, keys, and values are computed by separate (not fused) linear projections:

$$Q^{(l,h)}=H^{(l-1)}{W}_Q^{(l,h)},K^{(l,h)}=H^{(l-1)}{W}_K^{(l,h)},V^{(l,h)}=H^{(l-1)}{W}_V^{(l,h)},$$

(2)

with attention weights:

$$A^{(l,h)}=\mathrm{softmax}\left({\displaystyle \frac{Q^{(l,h)}{\left(K^{(l,h)}\right)}^{\top }}{\sqrt{d_k}}}\right).$$

(3)

We apply a group elastic-net regularizer to $W_Q$ and $W_K$. The $L_{2,1}$ norm

$${\parallel W\parallel }_{2,1}={\displaystyle \sum _{i=1}^{d_{\mathrm{model}}}}{∥W_{i,·}∥}_2$$

(4)

sums the $L_2$ norms of rows, encouraging entire rows to be pushed toward zero and effectively zeroing out specific input dimensions; this row-level sparsity propagates to the attention weights $A^{(l,h)}$, producing more concentrated token-level patterns. The Frobenius term acts as a standard weight decay. The per-head loss is

$${\mathcal{R}}_{\mathrm{sparse}}^{(l,h)}={\lambda }_1\left(\parallel W_Q^{(l,h)}{\parallel }_{2,1}+{\parallel W_K^{(l,h)}\parallel }_{2,1}\right)+{\lambda }_2\left(\parallel W_Q^{(l,h)}{\parallel }_F^2+{\parallel W_K^{(l,h)}\parallel }_F^2\right),$$

(5)

and the total sparse loss is ${\mathcal{L}}_{\mathrm{sparse}}={\sum }_{l,h}{\mathcal{R}}_{\mathrm{sparse}}^{(l,h)}$.

3.4. Supervised Contrastive Learning

Following Khosla et al. [9], we attach a 2-layer MLP projection head $g(·)$ to the [CLS] embedding, producing $L_2$-normalized vectors $z_i\in {\mathbb{R}}^{128}$. For a batch of $B=16$ samples with labels $y_i\in \{0,1\}$ and positive set $P\left(i\right)=\{j\ne i:y_j=y_i\}$, the supervised contrastive loss is:

$${\mathcal{L}}_{\mathrm{supcon}}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{\begin{array}{c}i=1\\ \left|P\right(i\left)\right|>0\end{array}}^B}{\displaystyle \frac{-1}{\left|P\right(i\left)\right|}}\sum _{p\in P\left(i\right)}log{\displaystyle \frac{exp(z_i·z_p/\tau )}{{\sum }_{j\ne i}exp(z_i·z_j/\tau )}},$$

(6)

where $\tau =0.07$ is a temperature. The sum in Equation (6) is restricted to anchors with at least one positive in the batch ($\left|P\right(i\left)\right|>0$); anchors with no positives are excluded following the standard implementation of Khosla et al. [9]. With batch size $B=16$ and the approximately balanced Dreaddit label distribution (53.5% stress/46.5% non-stress in training), on average, 7–8 positive samples are available per anchor, and the empty-positive case occurred in less than 2% of the mini-batches during training. Unlike a temporal contrastive loss that requires per-user chronology, this objective uses only batch-level class labels and therefore applies to any labeled dataset.

3.5. Total Objective

The training loss is:

$${\mathcal{L}}_{\mathrm{total}}=\underset{{\mathcal{L}}_{\mathrm{task}}}{\underbrace{{\displaystyle \frac{1}{B}}{\displaystyle \sum _{i=1}^B}\mathrm{CE}(y_i,{\widehat{y}}_i)}}+{\beta }_1{\mathcal{L}}_{\mathrm{sparse}}+{\beta }_2{\mathcal{L}}_{\mathrm{supcon}},$$

(7)

with ${\widehat{y}}_i$ the predicted class probability from a linear classifier on the [CLS] embedding, and ${\beta }_1,{\beta }_2\ge 0$ hyperparameters.

3.6. Training Procedure

• Inputs: dataset $\mathcal{D}={\left\{(x_i,y_i)\right\}}_{i=1}^M$; hyperparameters ${\lambda }_1,{\lambda }_2,{\beta }_1,{\beta }_2,\tau$; architecture parameters $L,H,d_{\mathrm{model}}$; number of epochs E; learning rate $\eta$.
• Output: trained model parameters $\mathsf{\Theta }$.

1.

Initialize the encoder, classifier head, projection head, and AdamW optimizer.

2.

For each epoch $e=1,\dots ,E$:

(a)

Shuffle $\mathcal{D}$.

(b)

For each mini-batch $\mathcal{B}\subset \mathcal{D}$:

(i)

Tokenize and embed sequences to obtain $X_{\mathcal{B}}$.

(ii)

Compute encoder output: $H^{\left(L\right)}\leftarrow \mathrm{Encoder}\left(X_{\mathcal{B}}\right)$.

(iii)

Extract [CLS] embeddings: $c_{\mathcal{B}}\leftarrow H^{\left(L\right)}[·,0,·]$.

(iv)

Compute contrastive projections: $z_{\mathcal{B}}\leftarrow \mathrm{normalize}\left(g\left(c_{\mathcal{B}}\right)\right)$.

(v)

Compute class logits: $\widehat{y}\leftarrow \mathrm{Classifier}\left(c_{\mathcal{B}}\right)$.

(vi)

Compute ${\mathcal{L}}_{\mathrm{task}}$, ${\mathcal{L}}_{\mathrm{sparse}}$, and ${\mathcal{L}}_{\mathrm{supcon}}$ per Equations (5)–(7).

(vii)

Form total loss: ${\mathcal{L}}_{\mathrm{total}}\leftarrow {\mathcal{L}}_{\mathrm{task}}+{\beta }_1{\mathcal{L}}_{\mathrm{sparse}}+{\beta }_2{\mathcal{L}}_{\mathrm{supcon}}$.

(viii)

Update parameters: $\mathsf{\Theta }\leftarrow \mathsf{\Theta }-\eta {\nabla }_{\mathsf{\Theta }}{\mathcal{L}}_{\mathrm{total}}$.

(c)

Evaluate on validation set; save checkpoint if $F_1$ improves.

3.

Return the checkpoint with the highest validation $F_1$.

Experiments are repeated 5 times with seeds $\{42,123,456,789,101,112\}$. A separate split seed (7) ensures identical train/val/test partitions across runs.

3.7. Dataset

We use the Dreaddit dataset [10], publicly downloadable from Columbia University (http://www.cs.columbia.edu/eturcan/data/dreaddit.zip (accessed on 5 July 2026)). Dreaddit comprises 190K Reddit posts from five subreddit categories (Abuse, Anxiety, Financial, PTSD, Social), with 3553 segments labeled for binary stress by 5 Amazon Mechanical Turk workers per segment (majority vote). The official split provides 2838 training and 715 test segments; we further split the training set 90/10 into train and validation (stratified by label, split seed $=7$), yielding 2554 train/284 validation/715 test segments. The training set contains 53.5% stress-labeled and 46.5% non-stress-labeled segments; the test set contains 54.0% stress-labeled and 46.0% non-stress-labeled segments.

3.8. Experimental Setup

3.8.1. Implementation and Hardware

The model is implemented in PyTorch 2.0.1 with the Hugging Face Transformers library v4.35 [28], CUDA 11.8. DR-Transformer experiments were run on a single consumer PC: Intel Core i7-10700 (8 cores @ 2.90 GHz), 32 GB DDR4 RAM, NVIDIA GeForce GTX 1660 6 GB GDDR5, 1 TB NVMe SSD; MentalBERT experiments were run on Google Colaboratory with a T4 GPU runtime.

Table 1. Hyperparameter configuration of DR-Transformer.

Parameter	Symbol	Value
Transformer layers	L	6
Attention heads	H	8
Embedding dimension	$d_{\mathrm{model}}$	256
Feed-forward dimension	$d_{\mathrm{ff}}$	1024
Maximum sequence length	N	128
Sparsity coefficient ($L_{2,1}$)	${\lambda }_1$	${10}^{-2}$
Sparsity coefficient ($L_2$)	${\lambda }_2$	${10}^{-3}$
Sparse-loss weight	${\beta }_1$	0.10
Contrastive-loss weight	${\beta }_2$	0.05
Contrastive temperature	$\tau$	0.07
Batch size	B	16
Peak learning rate	$\eta$	$2\times {10}^{-5}$
Warm-up steps	—	500
Total epochs	E	20
Run seeds	—	42, 123, 456, 789, 101,112
Split seed	—	7

lists key hyperparameters.

Hyperparameter selection used a grid search over the validation set covering ${\lambda }_1\in \{{10}^{-3},{10}^{-2},{10}^{-1}\}$, ${\lambda }_2\in \{{10}^{-4},{10}^{-3},{10}^{-2}\}$, ${\beta }_1\in \{0.05,0.10,0.20\}$, ${\beta }_2\in \{0.01,0.05,0.10\}$, and $\eta \in \{1\times {10}^{-5},2\times {10}^{-5},5\times {10}^{-5}\}$. To clarify a potential confusion noted by a reviewer: five hyperparameters with three values each would in principle yield $3^5=243$ combinations; however, the architectural temperature $\tau =0.07$ was fixed a priori based on the standard value used in Khosla et al. [9], reducing the free parameters to four, giving $3^4=81$ configurations. The configuration maximizing validation $F_1$ is reported in Table 1. The same tuning budget was applied to all neural baselines (BiLSTM and Standard Transformer). MentalBERT hyperparameters follow the original paper [7] (4 epochs, $\eta =2\times {10}^{-5}$).

Hardware

Experiments were conducted on two complementary platforms, reflecting the computational requirements of each model. DR-Transformer and all lightweight baselines (Logistic Regression, BiLSTM, Standard Transformer) were trained and evaluated on a local consumer machine: Intel Core i7-10700 (8 cores @ 2.90 GHz), 32 GB DDR4 RAM, NVIDIA GeForce GTX 1660 6 GB GDDR5, 1 TB NVMe SSD. No distributed computing is used for these models. MentalBERT (110M parameters) requires 11.2 GB peak GPU memory, which exceeds the 6 GB capacity of the GTX 1660; it was therefore fine-tuned and benchmarked on Google Colaboratory https://colab.research.google.com (accessed on 5 July 2026) using a T4 GPU runtime (15 GB VRAM)—a freely accessible cloud platform widely adopted in academic research. All efficiency measurements (training time, peak GPU memory, inference latency) were recorded on the respective platform for each model (full measurement protocol reported in Section 4.4). This two-platform setup directly illustrates the accessibility gap that motivates DR-Transformer: our model runs within a 6 GB consumer GPU, whereas MentalBERT requires either a high-memory GPU or a cloud platform.

3.8.2. Baselines

We compared DR-Transformer against the following models:

1.

Logistic Regression (BoW): TF-IDF weighted unigrams and bigrams, $C=1$.

2.

BiLSTM: 2 layers, 128 hidden units per direction, 200-dim learned embeddings.

3.

Standard Transformer: identical architecture to DR-Transformer but with ${\beta }_1={\beta }_2=0$ (no regularization beyond cross-entropy).

4.

MentalBERT [7]: mental/mental-bert-base-uncased fine-tuned for 4 epochs at $2\times {10}^{-5}$.

5.

DR-Transformer (Sparse Only): ablation with ${\beta }_2=0$.

6.

DR-Transformer (SupCon Only): ablation with ${\beta }_1=0$.

We do not include compact pretrained encoders such as DistilBERT or RoBERTa-base as baselines. Including a fine-tuned DistilBERT would be methodologically asymmetric, since such a model benefits from pretrained representations learned on billions of tokens, whereas DR-Transformer is trained entirely from scratch on Dreaddit alone. Comparing the two would conflate the effect of the proposed dual regularization with the effect of large-scale pretraining. We acknowledge this as a limitation (Section 5.4) and leave a controlled comparison—in which a compact pretrained encoder is also trained from scratch, or alternatively in which DR-Transformer is initialized from pretrained embeddings—for future work.

3.8.3. Evaluation Metrics and Statistical Testing

Classification: Binary $F_1$, accuracy, precision, recall, and AUC-ROC, reported as mean ± standard deviation over 5 seeded runs, with bootstrap 95% confidence intervals on $F_1$ (10,000 resamples on the pooled predictions across seeds; see below for interpretation).

Statistical significance: We report McNemar tests at two levels:

1.

Per-seed McNemar tests: for each of the five seeds, a McNemar test is computed independently on that seed’s test set predictions. The reported per seed p-value is the median across seeds after Bonferroni correction for 6 pairwise comparisons. This approach respects the independence assumption of the McNemar test.

2.

Pooled McNemar tests: predictions from all five seeds are pooled, effectively multiplying the sample size by five. This pooled test is reported as a secondary reference; because the same 715 test instances are reused across seeds, pooled predictions are not fully independent, and the pooled p-values should be interpreted with this caveat in mind.

• In practice, per seed and pooled results agree in direction and significance for all comparisons reported in

  Table 2. Classification performance on the Dreaddit test set (binary stress label). Values are mean ± standard deviation over 5 seeded runs; bracketed values are bootstrap 95% CIs for $F_1$. Bold = best per column. p refers to McNemar tests (pooled across seeds) against the Standard Transformer with Bonferroni correction; per seed median p-values agree in direction and significance for all rows.

  Model	${\mathit{F}}_1$ (95% CI)	Acc.	Prec.	Rec.	AUC	p
  Logistic Reg. (BoW)	$0.742$ ($0.718$–$0.765$)	$0.731$	$0.748$	$0.736$	$0.802$	<0.001
  BiLSTM	$0.795$ ($0.774$–$0.815$)	$0.788$	$0.800$	$0.790$	$0.855$	$0.009$
  Standard Transformer	$0.842$ ($0.815$–$0.867$)	$0.835$	$0.845$	$0.839$	$0.893$	ref.
  MentalBERT	$0.879$ ($0.862$–$0.894$)	$0.874$	$0.885$	$0.873$	$0.925$	$0.421$
  DR-Transformer (Sparse Only)	$0.858$ ($0.840$–$0.874$)	$0.852$	$0.862$	$0.854$	$0.908$	$0.003$
  DR-Transformer (SupCon Only)	$0.865$ ($0.848$–$0.881$)	$0.860$	$0.870$	$0.860$	$0.914$	$0.002$
  DR-Transformer (Full)	$\mathbf{0}.\mathbf{876}$ ($\mathbf{0}.\mathbf{852}$–$\mathbf{0}.\mathbf{898}$)	$\mathbf{0}.\mathbf{871}$	$\mathbf{0}.\mathbf{882}$	$\mathbf{0}.\mathbf{870}$	$\mathbf{0}.\mathbf{922}$	<0.001

  . The bootstrap 95% CIs are similarly computed from pooled predictions; they reflect stability across seeds rather than strictly the most frequent coverage and should be interpreted accordingly.

Interpretability: (i) attention sparsity—fraction of attention weights below $0.01$, averaged across all heads, layers, and test samples; (ii) silhouette score of [CLS] embeddings under cosine distance; (iii) token deletion confidence drop—see Section 4.3.

Efficiency: Training time, peak GPU memory (torch.cuda.max_memory_allocated at the end of the forward pass, batch size 16, sequence length 128, FP32 precision), inference latency over 200 single-sample forward passes after warm-up.

4. Results

4.1. Classification Performance

Table 2 reports the test set results. DR-Transformer (Full) achieves $F_1=0.876$ (bootstrap 95% CI $0.852$–$0.898$) and AUC-ROC of $0.922$, outperforming the Standard Transformer ($F_1=0.842$) by a statistically significant margin (McNemar $p<0.001$ with Bonferroni correction). The comparison with MentalBERT ($F_1=0.879$; $p=0.421$) does not reach statistical significance, indicating that DR-Transformer performs comparably to this much larger domain-adapted encoder while requiring roughly $12\times$ fewer parameters and less than half the training time (see Section 4.4 for the full efficiency comparison). Figure 2 illustrates these results graphically.

4.2. Training Dynamics

Figure 3 shows training and validation curves for DR-Transformer (Full) on a representative seed. Cross-entropy, sparse, and contrastive losses all decrease monotonically; the sparse and contrastive components remain small relative to the cross-entropy throughout training, indicating that regularization does not destabilize optimization.

4.3. Interpretability

Table 3. Interpretability metrics. Mean ± standard deviation over 5 seeded runs. Attention entropy (nats) is averaged over all heads, layers, and test samples; lower entropy indicates more concentrated distributions. Bold = best (proposed, DR-Transformer Full) row.

Variant	Attn. Sparsity	Silhouette	Attn. Entropy (Nats)
Standard Transformer (Vanilla)	$0.215\pm 0.023$	$0.312\pm 0.019$	$3.84\pm 0.12$
DR-Transformer (Sparse Only)	$0.593\pm 0.031$	$0.328\pm 0.021$	$2.71\pm 0.14$
DR-Transformer (SupCon Only)	$0.228\pm 0.025$	$0.451\pm 0.018$	$3.79\pm 0.11$
DR-Transformer (Full)	$\mathbf{0}.\mathbf{682}\pm \mathbf{0}.\mathbf{028}$	$\mathbf{0}.\mathbf{483}\pm \mathbf{0}.\mathbf{022}$	$\mathbf{2}.\mathbf{43}\pm \mathbf{0}.\mathbf{13}$

reports attention sparsity and silhouette score across variants. DR-Transformer (Full) achieves an attention sparsity of $0.682$ (vs. $0.215$ for the Standard Transformer), confirming that the $L_{2,1}/L_2$ penalty produces more concentrated attention. The supervised contrastive loss improves [CLS] embedding separability, with the silhouette score rising from $0.312$ (Standard Transformer) to $0.483$ (DR-Transformer Full). Figure 4 summarizes these interpretability metrics across all variants.

The attention entropy (Table 3) provides a complementary quantitative view: DR-Transformer (Full) reduces mean attention entropy from $3.84$ nats (Standard Transformer) to $2.43$ nats, corresponding to a substantially more peaked attention distribution. Attention concentration does not by itself establish causal faithfulness; we therefore complement these measures with a token deletion analysis below.

Token Deletion Analysis

To partially evaluate the causal relevance of the attended tokens, we conduct a token deletion analysis on the full test set. For each example, we identify the top-1 attended token under DR-Transformer (Full)—defined as the token receiving the highest [CLS] attention weight in the final layer, averaged across heads—and replace it with the [MASK] token. We then record the drop in prediction confidence (probability assigned to the original predicted class) relative to the unmasked input. The same procedure is applied to the Standard Transformer using its own top-1 attended token.

DR-Transformer (Full) shows a mean confidence drop of $\mathsf{\Delta }p=0.187\pm 0.093$ when its top-attended token is masked, compared to $\mathsf{\Delta }p=0.092\pm 0.071$ for the Standard Transformer (t-test $p<0.001$ on the 715 test examples). This indicates that the tokens receiving highest attention in DR-Transformer are more predictively relevant on average, providing partial support for the utility of the concentrated attention patterns—though a full comprehensiveness and sufficiency evaluation following established faithfulness protocols remains future work.

Qualitative attention maps for four representative test examples are shown in Figure 5, covering all prediction outcomes: true positive (TP), true negative (TN), false positive (FP), and false negative (FN).

4.4. Computational Efficiency

Table 4. Computational efficiency. Mean ± standard deviation over 5 runs. All models except MentalBERT ^† were measured on a single GTX 1660 (6 GB). Bold = proposed model (DR-Transformer).

Model	Train (min)	Peak GPU (GB)	Inference (ms/Sample)	# Params (M)
Logistic Regression	$1.2\pm 0.1$	$0.2\pm 0.05$	$0.8\pm 0.1$	—
BiLSTM	$18.5\pm 1.2$	$1.1\pm 0.1$	$5.2\pm 0.3$	$4.2$
Standard Transformer	$32.4\pm 1.8$	$3.6\pm 0.2$	$10.5\pm 0.5$	$9.48$
MentalBERT †	$67.8\pm 3.2$	$11.2\pm 0.4$	$28.3\pm 1.2$	$110.0$
DR-Transformer	$37.2\pm 2.1$	$4.8\pm 0.2$	$12.3\pm 0.6$	$9.48$

^† MentalBERT peak GPU usage (11.2 GB) exceeds the 6 GB capacity of the GTX 1660 and was therefore fine-tuned and benchmarked on Google Colaboratory using a T4 GPU runtime (15 GB VRAM). All other models were measured on the local GTX 1660. Measurement protocol for all models: torch.cuda.max_memory_allocated() at end of forward pass; batch size 16; sequence length 128; FP32 precision; no gradient accumulation.

reports computational cost. DR-Transformer adds a modest overhead over the Standard Transformer (the regularizers add no parameters, only a backward-pass term). MentalBERT is substantially larger and slower, requiring nearly $12\times$ more parameters, $3\times$ more training time, and $2.3\times$ more GPU memory. Figure 6 visualizes these efficiency comparisons.

5. Discussion

5.1. Principal Findings

Our experiments show that dual regularization—group-sparse attention combined with a supervised contrastive loss—produces a Transformer that is simultaneously competitive in accuracy, efficient, and structurally more transparent than a Standard Transformer of the same size. The ablations confirm that each regularizer contributes complementarily: Sparse Only increases attention concentration at a modest accuracy cost; SupCon Only improves accuracy and embedding separability but leaves attention diffuse; and the Full model retains the strengths of both. The Full model performs comparably to MentalBERT (a much larger domain-adapted encoder) on the binary stress task—without reaching statistical significance in either direction—while being trainable end-to-end on a 6 GB consumer GPU. This non-inferiority result is noteworthy given the $12\times$ parameter advantage and the absence of domain-specific pretraining in DR-Transformer.

5.2. Comparison with Existing Approaches

DR-Transformer trades the pretrained domain knowledge of MentalBERT for two architectural priors directly tied to the task: structurally concentrated attention and class-aware geometry. The result is a smaller, faster model that is also easier to inspect. Against efficient-attention work (Longformer [19], BigBird [20], Reformer [21]), our contribution lies in a different axis—structured sparsity on the projection matrices for attention concentration—which is orthogonal and could be combined with structured sparse attention to handle longer sequences. Compared to recent multimodal approaches such as Soufleri and Ananiadou [17], DR-Transformer achieves competitive text-only performance at a fraction of the computational cost, making it more accessible for resource-constrained deployment scenarios.

5.3. Ethical Considerations

Automated stress detection on social media raises ethical concerns that must be acknowledged before any deployment:

• False positives and stigmatization: Our model is far from perfect, and predictions must never trigger interventions unilaterally; a human-in-the-loop review is essential.
• Privacy: Passive monitoring raises consent issues; any deployment should align with platform terms of service and applicable data-protection law (e.g., GDPR).
• Bias: Dreaddit annotations are predominantly in the English language and skew demographically; fairness audits across language, age, and gender are required before generalization.
• Crisis routing: Stress is not equivalent to a mental health crisis. Predictions should not be used in place of validated clinical screening.

5.4. Limitations

1.

Dataset: Dreaddit is a relatively small dataset (3553 labeled segments) and English-language-only. Validation on multilingual or cross-platform data is needed before broad claims about generalizability can be made.

2.

Binary task: We focus on binary stress; multi-class or continuous severity prediction would be a natural extension.

3.

No temporal modeling: The model treats each segment independently; many practical applications involve longitudinal user data, which is outside the scope of this work and would require a different dataset (e.g., CLPsych under DUA).

4.

Limited interpretability evaluation: We report quantitative sparsity, attention entropy, token deletion confidence drop, and qualitative attention maps covering four prediction types, but do not conduct a clinical-expert evaluation or a full faithfulness evaluation (comprehensiveness/sufficiency tests); both are left for future work.

5.

Baseline scope: The current comparison does not include compact pretrained encoders such as DistilBERT or RoBERTa-base. As discussed in Section 3.8.2, including a fine-tuned pretrained encoder would be methodologically asymmetric given that DR-Transformer is trained from scratch. A controlled comparison with matching initialization conditions is left for future work. Consequently, the present experiments do not determine whether DR-Transformer is preferable to compact pretrained encoders such as DistilBERT or RoBERTa-base under realistic fine-tuning conditions; this question is left to the controlled comparison proposed above and in future work (Section 6).

5.5. Implications

The results support three broader claims for AI in digital health: (i) accessibility—high-quality models can be trained on consumer hardware, lowering the entry barrier; (ii) structural transparency—architectures with structurally induced attention concentration offer a complementary path to post hoc explanations (LIME/SHAP); while full faithfulness requires additional perturbation-based validation, the increased sparsity provides a useful inductive bias for inspection; (iii) methodological reuse—the dual-regularization recipe (structured sparsity for attention concentration + supervised contrast for class structure) is task-agnostic and could transfer to other text-classification problems in healthcare.

6. Conclusions

We presented DR-Transformer, a Dual-Regularized Transformer for stress detection in social media text. Group-sparse attention promotes more concentrated and inspectable token-level attention patterns; supervised contrastive learning produces class-aware latent geometry. On Dreaddit, the combined model reaches $F_1=0.876$ and performs comparably to MentalBERT while requiring $12\times$ fewer parameters and being trainable end-to-end on a single GTX 1660. Token deletion analysis provides partial evidence that the concentrated attention patterns are predictively relevant, though full faithfulness evaluation—including expert user studies—remains a topic for future work. These results suggest that dual regularization is a promising lightweight approach for mental health text classification, pending broader validation across datasets, languages, and demographic groups. Source code and trained checkpoints will be released upon acceptance (see Data Availability).

Future work will explore the following topics: (i) extension to multi-class severity prediction; (ii) combination with structured sparse attention mechanisms for long-document classification; (iii) cross-lingual transfer to non-English mental health datasets; (iv) prospective user studies with clinical experts; (v) integration of structured clinical lexicons (e.g., DSM-5) into the sparsity prior; and (vi) a controlled comparison with compact pretrained encoders initialized under equivalent conditions.