Explainable Transformer-Based Framework for Suicide Risk Detection: Deep Learning with Interpretability for Mental Health Crisis Identification

Abstract

The public health concern of suicide continues to rise and is increasingly prevalent on social media. The severity of this growing issue highlights the need for improved methods for detecting suicide risk. Many current deep learning approaches do not possess the required level of explainability for application in clinical settings. This study proposes the development of a transformer-based framework called “CrisisFormer,” which was trained on an imbalanced dataset containing 40,000 Reddit posts from the Suicide Watch subreddit and enhanced using DistilBERT. Additionally, the CrisisFormer framework uses three forms of explainable artificial intelligence for interpreting results: SHapley Additive exPlanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME), and transformer attention visualizations. The CrisisFormer framework achieved superior results for detecting the risk of suicide, with 96.25% accuracy, 96.30% precision, 96.25% recall, 96.25% F1 score, and 0.9944 AUC, compared to traditional models such as CNN, LSTM, and BiLSTM. Furthermore, by including clinically relevant suicide terms in its results, CrisisFormer demonstrates a high potential for incorporation into real-world mental health systems for intervention during ongoing mental health crises.

1. Introduction

Suicide is one of the most critical and preventable causes of premature mortality worldwide. Suicide is also the 4th leading cause of death in the age range of 15–29 years (the most at-risk group for mental illness and social isolation) worldwide. Approximately 700,000 people die by suicide every year according to the World Health Organization [1]. Millions more suffer from chronic suicidal thoughts, create plans for committing suicide, or attempt non-fatal suicide, the vast majority of whom are unknown to mental health professionals, do not receive support from family and friends, and have no community resources to help them. The hidden nature of most of the burden associated with suicide is particularly worrisome because non-fatal suicide attempts are strong predictors of later completion of suicide. The absence of identification and support for a large number of non-fatal suicide attempters represents not only a missed opportunity for clinical intervention but also a serious threat to the overall public health system.

This gap in detection has an extensive impact because untreated suicidal crises usually worsen over time, and the absence of appropriate responses increases the physiological impact of the untreated individual, which corresponds to the probability of completion of suicide. The significant progress made by mental health advocates, public health organizations, and governments to increase general awareness of mental illness and provide access to evidence-based treatment options has left early and accurate identification of people at real and imminent risk of suicide as one of the most challenging, complex, and consequential issues facing researchers, clinicians, and policymakers worldwide.

One reason is that it is difficult to identify suicidal thoughts and feelings because people often attempt to hide them. Instead, they show signs of suicidal thoughts and feelings through many different types of behavior and language, which do not appear when people are evaluated using traditional clinical assessment methods. Thus, there is an urgent need for scalable, automated, and intelligent detection systems that can proactively identify individuals at risk of suicide. These detection systems must function effectively in different linguistic and cultural contexts and complement existing clinical assessment systems that are ethical and based on valid scientific research.

The growing number of social networking sites has changed how people share their emotions. Platforms such as Reddit, Twitter, and Facebook represent new ways for people to tell others about their emotional distress due to being alone, depressed, or having thoughts of self-harm before receiving treatment or talking to a friend or family member about these feelings [2]. This body of information on the Internet provides an opportunity to develop automated, large-scale systems to identify issues with suicidal behavior by analyzing text for early warning signs.

Natural language processing (NLP) and machine learning have shown great potential for analyzing text related to mental health on social media [3]. In the past, computational approaches to analyzing this type of content used lexicon-based models, handcrafted features, and classical classifiers (e.g., logistic regression and support vector machines) to produce benchmark results. However, they were not able to effectively capture complex linguistic patterns, context dependencies, and the subtle semantic nuances associated with suicidal communication [4].

The introduction of deep learning architectures, especially recurrent neural networks such as Long Short-Term Memory (LSTM) [5] and Convolutional Neural Networks (CNN), has resulted in significantly improved performance in text classification. Furthermore, deep learning models can learn hierarchical representations of language/text without extensive feature engineering. However, they are still limited by their use of fixed context windows and their inability to capture long-range dependencies in text, both of which are vital for understanding subtle expressions of psychological distress [6].

The introduction of transformer-based models has completely overhauled the field of natural language processing. The introduction features work from Vaswani et al. [7], which provided the foundation for BERT [8] and RoBERTa [9], two sets of models that were trained using large amounts of data to achieve state-of-the-art performance on many different benchmarks for language understanding after fine-tuning them for specific tasks. These models have shown themselves to be capable of identifying contextual semantics, which makes these types of models extremely well-suited for conducting more sensitive types of classification work, such as predicting the likelihood of suicide [10].

The major shortcoming of transformer-based models is their lack of interpretability. A model that only predicts the risk without describing how it arrived at that conclusion is insufficient and could cause harm in clinical applications with high stakes. To provide a sufficient level of transparency to mental health professionals, the models must provide explainable predictions that indicate the specific external traits/patterns present in the input data that led to a specific overall risk classification [11]. The mandate to provide an explanation is more than an academic requirement [12]; it is a requirement to enable clinicians to fully trust the prediction, to provide for the ethical use of, and to allow for meaningful collaboration between mental health clinicians and artificial intelligence when working together.

The issue that has been identified here is the absence of an explainable model that combines the strengths of existing deep learning models, such as DistilBERT [13], and some form of interpretability module. To address this gap, we developed a new method called CrisisFormer, which includes an explanation layer built on top of DistilBERT (which retains 97% of the original BERT’s language understanding capabilities) to provide real-time predictions of suicide risk using social media data.

In this study, we present an explainable hybrid predictive model called “CrisisFormer,” which utilizes three current interpretability techniques (SHAP, LIME, and Attention Visualization) for predicting suicide risk based on social media text. In addition, we tested its effectiveness with the Suicide Watch dataset from Reddit, showed that it performed better than multiple baseline models, and provided a detailed exploration of how CrisisFormer’s predictions correlate with the clinical evidence of suicidal behavior (using various linguistic features linked to suicide risk). The full implementation of the model can be requested from the corresponding author, in adherence to open science principles.

The remainder of this paper is organized as follows: Section 2 provides an overview of related research; Section 3 describes the dataset, including how we preprocessed it; Section 4 discusses the proposed CrisisFormer approach; Section 5 provides experimental results; Section 6 provides analyses of the results and limitations; Finally, Section 7 concludes our findings.

2. Related Works

Lexicon-based and classical statistical approaches defined the early evolution of computational methods for suicide risk detection, prior to the widespread adoption of machine learning and deep learning techniques. The use of lexicon-based approaches prior to machine learning was characterized by “manual” feature engineering and the creation of features for the training of a model through the creation of a feature set. Most early studies relied on manual definitions of both linguistic markers and how features were defined, thereby requiring a level of human interpretation to turn raw data (text) into structured data. For instance, the models developed by Coppersmith et al. [2] used predefined linguistic markers including word counts, parts of speech, and psychological content categories to classify suicidal posts on Twitter, involved using a predefined set of linguistic markers such as a number of words, parts of speech, and/or type of psychological content to create features necessary for classification of electronically detected suicidal individuals’ posts on Twitter (demonstrating the feasibility of conducting mental health and suicide-based surveillance through social media).

Using an approach similar to Coppersmith et al.’s [2], Burnap et al. used machine learning algorithms to classify electronically detected suicidal individuals’ posts [14] and achieved significant performance using both n-grams (i.e., consecutive sequences of n words) and sentiment scores. While both studies provide significant baseline data and proof of feasibility, they share one limitation: both studies relied on manual feature engineering and, as such, cannot be generalized across different samples and/or contexts. (Collectively) both studies lacked the ability to express the multitude of complex, nuanced, linguistic elements present in all human communications regarding mental health and/or suicidality.

Deep learning has subsequently created a major shift toward resolving the limitations of traditional methods; however, it has also introduced new obstacles. Tadesse et al. [4] demonstrated much better results using LSTM networks and CNNs to identify suicidal thoughts on Reddit than previous approaches, in that LSTMs automatically learn features from the text (instead of relying on previous feature sets) and do so from the context of the entire post. Yates et al. [15] showed that these neural network approaches lead to better classification of mental health than previous approaches; however, they also established that the systems were not able to provide clinicians with reliable/meaningful and interpretable data. The LSTM complex model was developed by Hochreiter and Schmidhuber [5] when they were trying to develop a model for sequential data. LSTMs were developed for the specific purpose of being able to model the sequential flow of language. Despite the performance gains achieved with deep learning, there are two major weaknesses. First, deep learning models generally lack the ability to deal with longer continuous contexts throughout the entire document (i.e., long-term dependencies). Second, the reason deep learning systems are unable to provide reliable information/data to clinicians is because of their non-LTSM structures and opaque computations, which are very difficult to interpret, leading to a major “black” box problem.

The introduction of the transformer model by Vaswani et al. [7] introduced a paradigm shift in the field, and subsequent developments, such as pretrained models BERT [8], resulted in marked differences between recurrent neural networks and transformer-based models through the utilization of self-attention mechanisms within transformer architectures. Self-attention mechanisms promote a model’s ability to consider all aspects of an input sentence at once, allowing it to determine which individual token/word in the input sentence is relevant for producing the model’s output, regardless of the proximity of the tokens to each other. Consequently, models built on transformer architecture can incorporate much richer representations of the contextual dependencies contained within the input data.

The efficacy of the self-attention mechanism within transformer architectures for predictive modeling was demonstrated by Matero et al. [10], who observed that BERT-type models produced superior predictions of suicide risk based on both clinical and social media text compared to traditional deep learning and conventional predictive model techniques. The field has also evolved rapidly owing to the development of advanced versions of the pre-training process, such as RoBERTa [9] and DistilBERT [13], which produced state-of-the-art performance on many benchmarking datasets while making the pre-trained models smaller and sufficiently efficient to be used on resource-limited production deployments with minimal performance degradation.

While these performance improvements were substantial, the interpretability of these exceptionally well-performing transformer models remains a critical barrier to clinical use. This has resulted in the rise of Explainable AI (XAI) in mental health natural language processing. The introduction of foundational techniques such as SHAP [11] and LIME [16] addressed the need to understand model behavior through a shared framework of interpretability. SHAP uses game-theoretic Shapley values to produce a theoretically grounded and unified representation of the importance of features, whereas LIME, which employs local explanations, approximates how predictions made by a complex model behave in proximity to a particular prediction.

Individuals in this area of study have begun utilizing these approaches in mental health; for example, Aladağ et al. [6] applied neural networks with limited interpretability methods to detect suicide ideation. In addition to these methodological advancements, datasets such as the Dreaddit dataset [17] have served to strengthen the empirical base of our knowledge while also providing resources unique to the analysis of stress. Consequently, as evidence accumulating in the area of clinical practice supports the growing demand for interpretable models, authors such as Ji et al. [3] have recognized this need as urgent and specifically noted that transparency in models must be present before they are used in the real world as presented in

Table 1. Comparison of Related Work on Suicide Risk Detection.

Study	Method	Contributions	Explainability
Coppersmith et al. [2]	SVM + NLP	Pioneered NLP-based suicide risk detection from social media	No
Ji et al. [3]	Review + ML	Comprehensive taxonomy of ML methods for suicide risk	No
Tadesse et al. [4]	LSTM + CNN	Combined deep learning for improved text classification	No
Aladağ et al. [6]	Neural Network	Demonstrated neural networks for ideation detection on forums	No
Matero et al. [10]	BERT	First BERT-based suicide risk assessment on social media	No
Burnap et al. [14]	SVM + N-grams	Established linguistic baselines for suicide content detection	No
Yates et al. [15]	Neural Network	Early neural approach to suicide risk from online posts	No
de Oliveira et al. [18]	ML + XAI (Boamente)	Introduced explainability into suicide ideation classification	Yes
Heckler et al. [19]	Systematic Review	Mapped ML landscape for suicidal ideation identification	No
de Oliveira et al. [12]	XAI + LLM (Boamente)	Showed XAI improves clinician trust in AI suicide systems	Yes
Levkovich and Omar [20]	BERT/LLM Review	Synthesized BERT and LLM performance in suicide detection	No
de Oliveira et al. [21]	Generative LLM	Integrated generative LLMs with explainability for detection	No
CrisisFormer (Ours)	DistilBERT	Hybrid explainable model combining SHAP, LIME and Attention	Yes

.

According to several recent studies, deep learning and transformer-based methods for detecting suicidal risk have improved rapidly in the last few years. In addition, De Oliveira et al. [18] utilized explainable machine learning to identify mental health in communication platforms (e.g., social media), demonstrating how interpretability methods can clarify language features used by machine-learning models to predict outcomes. Heckler et al. [19] undertook a systematic review to provide an overview of machine-learning methodologies for identifying suicidal thoughts, summarizing the literature through nearly 4000 studies and highlighting the increasingly important role of deep learning in this area in particular. These studies by De Oliveira et al. [12], Levkovich and Omar [20], and De Oliveira et al. [21], together emphasize the growing agreement about the importance of interpretability, clinical relevance, for the responsible use of AI-based systems for suicide risk detection with a specific emphasis on the need for clearly interpretable, clinical-trusted models that will enable effective in-use clinical application in a real world mental health monitoring context.

3. Dataset and Methodology

3.1. Description of the Dataset

The Suicide Watch dataset is a widely cited benchmark dataset available in the public domain for the purpose of mental health text classification. It was obtained from the social media platform Reddit and is now available via the Kaggle platform. Reddit has been extensively studied as a source of mental health data because users can post anonymously, and there is a high degree of emotionally candid discourse on the platform [2]. There are two types of subreddits used to compile the posts included in the dataset: r/SuicideWatch, which is a subreddit dedicated to providing a place for individuals to express their suicidal thoughts and intentions, and r/teenagers, which represents the control group of non-suicidal adolescent social media communication. The methodology for constructing the dataset follows a well-established protocol used in mental health NLP research, wherein the use of subreddits that are specific to a particular community serves as a reliable proxy for a clinical label [17]. The intentional use of the two subreddits provides the best opportunity to conduct the binary classification task in a way that reflects the true linguistic differences in posting between at-risk and general social media postings, and not simply the topical differences in the posts [3].

The total unprocessed dataset comprised 232,074 individual sample records with two main feature data columns (text representing the content of a post and a binary class labelled with a suicide/non-suicide flag). A preliminary exploratory analysis of the total dataset found no missing or NULL values in both the text and flag data columns, indicating that the input source of the raw data was structurally valid. Prior to our research into social media mental health-related datasets, researchers have shown that the richness of language employed by Reddit corpora provides for building strong deep learning-based classifier algorithms to correctly classify posts for suicidal ideation [4], and the Suicide Watch Data Set has been used in countless prior research studies as a standard for suicide risk detection models [6].

The statistical analysis of the raw text indicated substantial variability in terms of post length, with the mean post at a character count of 688 (132 words); however, most posts were found to range between 100 and 2000 characters, while a minority exceeded 10,000 characters. As the distribution of text lengths is typically right-skewed, the mean length of the token sequence used for the analysis presented in Section 3.4 was based on this distributional characteristic. This level of variability in language is consistent with the findings of Coppersmith et al. [2], who found that individuals with thoughts of suicide tend to write longer, more complex posts than the authors of control group posts.

3.2. Data Preprocessing Pipeline

To meet the requirements of the preprocessing methodology, several processes must be followed, including five separate stages of the preprocessing pipeline, to ensure that the data obtained from a raw database are high-quality and free of noise before being used for the transformer-based fine-tuning of a text corpus as illustrated in Figure 1. As evidenced by many previous studies [3], rigorous preprocessing in mental health natural language processing (NLP) is essential for producing a high-quality training set. Any systematic biases in the data caused by poor-quality training data (e.g., noisy or unbalanced) will adversely affect the model’s performance and, therefore, clinical reliability.

• Stage 1—Duplicate Removal. Let D = {(x_i, y_i)}ⁿ_i=1 denote the raw dataset of n samples, where x_i represents the text content and y_i ∈ {0, 1} represents the binary label. Duplicate samples were identified and removed based on exact string matching of the text field:
  D′ = {(x_i, y_i) ∈ D: x_i ≠ x_j, ∀j < i}

  (1)

  This step is to ensure that the test and training splits do not have duplicate values of the same text to avoid data leakage, resulting in inflated test metrics. This consideration was discussed in detail by Ji et al.’s systematic review of identification methods for suicide via the recent literature.
• Stage 2—Null Value Removal. Samples containing null or undefined values in either the text or label columns were excluded:
  D″ = {(x_i, y_i) ∈ D′: x_i ≠ ∅ ∧ y_i ≠ ∅}

  (2)
• Stage 3—Short Text Filtering. Posts containing fewer than ten characters were removed as they were deemed insufficient to provide meaningful linguistic content for classification. This threshold was established based on the minimum token sequence length required to activate meaningful contextual representations within the DistilBERT encoder:
  D‴ = {(x_i, y_i) ∈ D″: |x_i| > 10}

  (3)

  where |x_i| denotes the character length of text x_i.
• Stage 4—The next step is called “Label Encoding,” which involves assigning binary integer labels in the approximate place of string labels, whereas “Suicide” = 1 and “Non-Suicide” = 0, which corresponds to the industry standard for binary text classification:
  y_i = {1, if class = “suicide”; 0, if class = “non-suicide”}

  (4)
• Stage 5—To address the challenge of class imbalance in modelling and evaluation due to class imbalance, we used a stratified technique of balanced sampling. The issue of class imbalance and its impact on modelling/classifying have been well documented, and can result in biased classifiers that have a proclivity toward the majority class when the data set is imbalanced [3]. Let $N_c$ denote the number of samples belonging to class c ∈ {0, 1}. A target sample size of S = 40,000 was established, with S/2 = 20,000 samples drawn uniformly at random from each class without replacement:
  $$D_{\mathrm{final}}={\bigcup }_{c\in \left\{\mathrm{0,1}\right\}}\mathrm{Sample}(D_c^{‴},n=S/2)$$

  (5)

  where $D_c^{‴}=\left\{\right(x_i,y_i)\in D^{‴}:y_i=c\}$ denotes the class-specific subset. This balanced design ensures equal prior probabilities $P(y=0)=P(y=1)=0.5$. We do not need to apply any class weighting when training the model(s), thereby allowing accuracy, precision, and recall metrics to be reported without further adjustments for interpretation. Tadesse et al. [4] supported the application of this methodology through the application of balanced sampling in a study designed (as shown in

  Table 2. Preprocessing Stage Results.

  Stage	Operation	Input Samples	Output Samples	Removed
  1	Duplicate Removal	232,074	232,074	0
  2	Null Value Removal	232,074	232,074	0
  3	Short Text Filtering	232,074	231,890	184
  4	Label Encoding	231,890	231,890	0
  5	Balanced Sampling	231,890	40,000	191,890

  ) to identify suicide using a deep learning approach applied to data sourced from Reddit. A total of 40,000 (20,000 of each type) were selected for both class balance and GPU computational budget. The entire dataset of 232,074 posts is highly skewed; random down-sampling to 40,000 ensures the same number of each class and is still feasible for transformer fine-tuning, thus justifying this sample size.

3.3. Framework

Following standard machine learning methods, the final balanced dataset of 40,000 samples was divided into three separate sets with no overlapping samples. A stratified split was conducted at each stage of the split to keep the 1 1 ratio of classes across each of the three sets. This splitting method fits with the suggestions for methods using NLP in the area of mental health [15], and thus, evaluation metrics designed for the blind set of data are indicative of real-world performance. First, the dataset was divided into two sets: a training set with 70% of the total samples and the other 30% remaining as a temporary blind set. Next, the blind sample was split into two equal parts, validation and test samples, making each 15% of the total dataset. Formally, let $D_{\mathrm{final}}$ be partitioned as:

$$D_{\mathrm{train}}\cup D_{\mathrm{val}}\cup D_{\mathrm{test}}=D_{\mathrm{final}}$$

(6)

where

$$\begin{gathered} \mid D_{\mathrm{train}}\mid =0.70\times \mathrm{40,000}=\mathrm{28,000} \\ \mid D_{\mathrm{val}}\mid =0.15\times \mathrm{40,000}=6000 \\ \mid D_{\mathrm{test}}\mid =0.15\times \mathrm{40,000}=6000 \\ {D}_{\mathrm{train}}\cap D_{\mathrm{val}}=D_{\mathrm{train}}\cap D_{\mathrm{test}}=D_{\mathrm{val}}\cap D_{\mathrm{test}}=\mathrm{\varnothing }. \end{gathered}$$

The stratification constraint ensures that for each subset $S_k\in \{D_{\mathrm{train}},D_{\mathrm{val}},D_{\mathrm{test}}\}$:

$${\displaystyle \frac{\mid \left\{\right(x_i,y_i)\in S_k:y_i=c\}\mid }{\mid S_k\mid }}=0.5,\forall c\in \{\mathrm{0,1}\}$$

(7)

This ensured that the class distribution was perfectly preserved across all splits, preventing any subset from being inadvertently dominated by either class. Matero et al. [10] similarly employed stratified partitioning in their BERT-based suicide risk assessment study, and Turcan and McKeown [17] applied comparable partitioning strategies in their Dreaddit stress detection dataset construction as illustrated in

Table 3. Dataset Partitioning Summary.

Split	Samples	Suicide Class	Non-Suicide Class	Proportion
Training	28,000	14,000	14,000	70%
Validation	6000	3000	3000	15%
Test	6000	3000	3000	15%
Total	40,000	20,000	20,000	100%

.

3.4. Training Configuration

We utilized the DistilBERT WordPiece tokenizer [13] to tokenize the text. DistilBERT uses a subword segmentation process to break down the input text into subword token sequences with reference to a vocabulary of 30,522 previously established lexical items. The BERT model introduced this WordPiece-based approach for tokenization, allowing for a trade-off between tokenizing all words in the target language as single tokens while simultaneously splitting multi-part or rare words into their most basic components as individual tokens. Therefore, subword representations should adequately represent this type of text [2]. This includes written communication through social media platforms about mental health, which may contain misspellings, colloquial phrases, and/or words that have no equivalent in traditional language lexicons.

The WordPiece tokenization method works using a greedy algorithm, which selects the longest matching subword unit at each position in the input x based on the first word part that appears in the dictionary. If the token you are working with does not exist in the dictionary, you will break it down character by character to create new tokens, and the new tokens will begin with the prefix “##” to indicate that they are continuing the same word. The word “hopelessness” could be tokenized into [“hopeless”, “##ness”], thus allowing the original meaning of the token to remain the same as that of the tokens created from words that exist in the dictionary. Therefore, when a raw text string x is entered into the tokenizer, it will return a sequence of tokens $T\left(x\right)$:

$$T\left(x\right)=\left[\right[\mathrm{CLS}],t_1,t_2,\dots ,t_m,[\mathrm{SEP}],[\mathrm{PAD}],\dots ,[\mathrm{PAD}\left]\right]$$

(8)

The [CLS] token is used at the beginning of the input sequence ([SEP] is used for the end of the input), and serves as the aggregate sequence representation for classification [8] by using the final hidden state for each token in the sequence., $t_1$ through $t_m$ are the WordPiece subword tokens derived from $x$, and $\left[\mathrm{PAD}\right]$ tokens are appended to reach the maximum sequence length $L_{\max }=256$. The tokenizer produces two output tensors for each sequence:

• Input IDs $I\in {\mathbb{Z}}^{L_{\max }}$: integer token identifiers mapping each token to its vocabulary index.
• Attention Mask $A\in \{\mathrm{0,1}{\}}^{L_{\max }}$: binary mask where $A_i=1$ for real tokens and $A_i=0$ for padding tokens, preventing the model from attending to padding positions.

The attention mask ensures that the self-attention mechanism within each transformer layer [7] correctly computes attention only over meaningful token positions:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}}+M\right)\cdot V$$

(9)

where $Q,K,V\in {\mathbb{R}}^{L_{\max }\times d_k}$ are the query, key, and value projection matrices, respectively, $d_k=64$ is the per-head key dimensionality computed as $d_{\mathrm{model}}/h=768/12=64$, and $M\in {\mathbb{R}}^{L_{\max }\times L_{\max }}$ is the additive masking matrix defined as:

$$M_{ij}=\left\{\begin{array}{cc}0,& \mathrm{if} A_j=1\\ -\mathrm{\infty },& \mathrm{if} A_j=0\end{array}\right.$$

(10)

The softmax operation over à values produces zero attention weights for padding positions, ensuring that padding tokens contribute nothing to the contextual representations computed by the encoder. The maximum sequence length of N was selected based on an analysis of the text length distribution, which showed that 96.8% of all samples in the dataset contained fewer than 256 WordPiece tokens, ensuring that only a minimal proportion of 3.2% of samples required truncation. This selection of token types aligns with the use of truncation as a means of encoding social media text found in the work of Matero et al. [10] in their BERT-based suicide risk assessment studies and tokenization via the truncating of the sequence on the right, retaining the left side of the sequence, which contains the majority of valuable information describing the author’s state of mind [6].

Table 4. Tokenization Configuration Summary.

Component	Value	Description
Tokenizer	DistilBERT WordPiece	Subword tokenization
Vocabulary	30,522 units	Pre-trained token vocabulary
Max Length	256 tokens	Covers 96.8% of samples
Padding	Right padding	Ensures uniform batch size
Truncation	Right truncation	Preserves informative content
Special Tokens	[CLS] (101), [SEP] (102), [PAD] (0)	Classification, separation, padding
Outputs	Input IDs, Attention Mask	Token indices and padding mask
Avg. Tokens	98.4 per sample	Mean sequence length
Truncation Rate	3.2%	Sequences exceeding 256 tokens

shows the full summary of Tokenization Configuration.

3.5. Explainability Methods

Before training any models, we performed an extensive exploratory data analysis (EDA) of our preprocessed dataset to determine its distribution and verify that our preprocessing pipeline functioned as intended. This is a necessary step in mental health NLP research prior to modeling to uncover any potential biases or confounding variables that could impact how our model behaves [3].

The text length distribution analysis revealed that suicide-related posts are generally longer than non-suicidal posts, with an average length of approximately 750 characters for the former and an average of approximately 626 characters for the latter. This aligns with the findings of Coppersmith et al. [2], indicating that individuals who express suicidal ideation are more likely to use much more complex word structures than authors in a control group. Similarly, our word count analysis confirmed that suicide-related posts averaged 148 words, compared to 116 words for non-suicidal posts. Balanced sampling resulted in each class having 20,000 samples, which perfectly corresponded to a uniform class distribution. Shannon entropy was calculated as follows:

$$H=-{\sum }_{c\in \left\{\mathrm{0,1}\right\}}P\left(c\right){\mathrm{l}\mathrm{o}\mathrm{g}}_{2}P(c)=-2\times (0.5\times {\mathrm{l}\mathrm{o}\mathrm{g}}_{2}0.5)=1.0 \mathrm{bit}$$

(11)

This maximum binary class entropy (1.0 bit) indicates that the balanced sampling technique was effectively utilized, indicating that a naive majority classifier could produce an accuracy rate of only 50%, thus providing a valid baseline for measuring CrisisFormer’s performance. Similarly, similar balanced designs have been used in studies on suicidal behavior detection conducted by Tadesse et al. [4] and Aladağ et al. [6]. According to Burnap et al.’s [14] research about suicidal text classification, class balance is crucial in order to have reliable and unbiased performance metrics, which further validates this approach.

4. CrisisFormer Framework

4.1. CrisisFormer Architecture

CrisisFormer uses DistilBERT, which is a smaller version of BERT but still has the ability to understand language up to 97% of the capacity of BERT; the model is approximately 40% smaller and can provide results in approximately 60% less time than BERT. Instead of describing how DistilBERT works, we concentrate on how to adjust DistilBERT so that it functions correctly with CrisisFormer for creating a binary suicide risk classification, which is explained further in Figure 2.

The classification head that goes on top of the DistilBERT encoder model consists of a pre-classification dense layer constructed using the GELU activation, followed by a dropout layer (0.3 dropout probability, providing regularization), followed by a linear classification layer that uses softmax to output prediction probabilities for either Suicide Risk or No Risk. Formally, given an input text sequence $X=\{x_1,x_2,\dots ,x_n\}$, the model computes:

$$H=\mathrm{DistilBERT}\left(X\right)$$

(12)

where $H\in {\mathbb{R}}^{n\times 768}$ represents the contextual token embeddings. The [CLS] token representation $h_{\left[\mathrm{CLS}\right]}$ is extracted and passed through the classification head:

$$y=\mathrm{softmax}(W_2\cdot \mathrm{GELU}(W_1\cdot \mathrm{Dropout}\left(h_{\left[\mathrm{CLS}\right]}\right)+b_1)+b_2)$$

(13)

where $W_1,W_2,b_1,$ and $b_2$ are learnable parameters of the classification head.

4.2. Training Configuration

The model was fine-tuned on the training set using the AdamW optimizer with a learning rate of $2\times {10}^{-5}$ and weight decay of 0.01. In order to stabilize the early phase of training, the model was run with a linear warmup schedule for 500 steps. Maximum of 10 training epochs with a batch size of 32 for training batches and 64 for evaluation batches. The mixed precision (FP16) training method was used to speed up computational time. An early stopping method, based on validation accuracy and with patience of 3 evaluation epochs, was used to avoid overfitting. The details are shown in

Table 5. Training Hyperparameters and Configuration.

Hyperparameter	Value	Purpose/Rationale
Base Model	distilbert-base-uncased	Provides efficient transformer backbone with 66.9 M parameters; optimal trade-off between performance and speed [13]
Optimizer	AdamW	Adaptive optimizer with decoupled weight decay; standard for transformer fine-tuning
Learning Rate	2 × 10−5	Small learning rate prevents catastrophic forgetting during fine-tuning
Weight Decay	0.01	Regularization technique to prevent overfitting
Warmup Steps	500	Gradually increases learning rate to stabilize early training
Training Epochs	10	Maximum epochs; early stopping prevents unnecessary computation
Train Batch Size	32	Balances memory constraints and gradient stability
Eval Batch Size	64	Larger batch size for efficient evaluation
Max Sequence Length	256	Covers 96.8% of samples; matches tokenizer configuration
Dropout Probability	0.3	Prevents co-adaptation of neurons; reduces overfitting
Early Stopping Patience	3	Halts training when validation accuracy plateaus
Mixed Precision	FP16	Accelerates training while reducing memory usage
Total Parameters	66,955,010	Model size suitable for deployment in resource-constrained settings

. The model’s best performing checkpoint during training was kept for final evaluation. Early stopping with patience of 3 halted training at step 1400 (epoch 2), where peak validation accuracy of 96.73% was observed, confirming that the model converged well before the maximum epoch limit.

The maximum number of epochs was initially set to 10; however, early stopping with a patience of 3 evaluation steps was applied throughout training. In practice, the optimal checkpoint was reached at step 1400 (validation accuracy: 96.73%), at which point training was halted automatically. The learning rate of 2 × 10⁻⁵ and weight decay of 0.01 follow established best practices for fine-tuning DistilBERT on downstream classification tasks as seen in Table 5.

4.3. Explainability Module

A core contribution of CrisisFormer (Algorithm 1) is its multi-layer explainability module, which integrates three complementary interpretability techniques to provide comprehensive and clinically meaningful explanations for model predictions as seen in

Table 6. Comparison of Explainability Methods Used in CrisisFormer.

Method	Type	Scope	Output	Computation
SHAP	Game Theory Based	Global + Local	Token Shapley Values	Medium
LIME	Surrogate Model	Local	Feature Importance	Medium
Attention	Transformer Native	Local	Attention Weights	Low

and Figure 3.

Algorithm 1. CrisisFormer Inference with Multi-Layer Explainability

Require:

Raw text input x, Trained model CrisisFormer (θ*), SHAP KernelExplainer, LIME LimeTextExplainer, Number of LIME samples N = 1000, Number of features K = 20

Ensure:

Predicted class $\hat{y}$, Confidence score p, SHAP values ϕ, LIME weights w, Attention matrix A

1.    Tokenize x using WordPiece: T(x) → {I, Amask}, Lmax = 256

2.    Forward pass with attention extraction:

3.    H, Aweights ← CrisisFormer(I, Amask; θ*, output_attentions = True)

4.    Extract [CLS] token representation: hCLS ← H[0]

5.    Compute classification probabilities:

6.    p ← softmax(W2 · GELU(W1 · Dropout(hCLS) + b1) + b2)

7.    Predict risk class: $\hat{y}$ ← arg max (p)

8.    Compute prediction confidence: conf ← max (p)

9.    ▷ Stage 1—SHAP Explainability

10.  Initialize Text masker using WordPiece tokenizer vocabulary

11.  Compute Shapley values: ϕ ← SHAP_Explainer(T(x))

12.  Rank tokens by descending absolute importance: ϕranked ← sort(|ϕ|)

13.  ▷ Stage 2—LIME Explainability

14.  Generate N = 1000 perturbed text samples around x

15.  Fit interpretable surrogate linear model on perturbed samples

16.  Extract top K = 20 feature importance weights: w ← LIME_Explainer(x, K)

17.  ▷ Stage 3—Attention Visualization

18.  Extract final encoder layer attention: A ← Aweights[−1][0,0]

19.  Trim A to valid token length: A ← A[: Lvalid,: Lvalid]

20.  Restrict visualization to first 20 tokens: A ← A[:20,:20]

21.  Generate token-to-token attention heatmap from A

22.  return $\hat{y}$, conf, ϕ, w, A

4.3.1. SHAP Analysis

The SHAP toolkit utilizes cooperative game theory to produce both global and local attributions of feature importance [11]. Specifically, for each prediction made by the model, SHAP calculates the contribution of each token to that prediction using Shapley values. Each token contributes a higher-than-average Shapley value (i.e., greater than zero) to the Suicide Risk class label, while other tokens contribute a lesser-than-average Shapley value (i.e., below zero) to the No Risk class label. For this research project, I applied KernelExplainer to 20 samples of test data and calculated token-level. SHAP-based token attribution results for representative test samples are presented and discussed in Section 5.

Theorem 1 (Shapley Value Uniqueness).

Let $v:2^F\to \mathbb{R}$ be a characteristic function over the set of input features $F=\{t_1,t_2,\dots ,t_n\}$ where each $t_i$ represents a token. The unique attribution satisfying the four axioms of Efficiency, Symmetry, Dummy, and Additivity is the Shapley value:

$${\varphi }_i(v)=\sum _{S\subseteq F\setminus \left\{i\right\}}{\displaystyle \frac{\mid S\mid !\hspace{0.17em}(\mid F\mid -\mid S\mid -1)!}{\mid F\mid !}}\cdot \left[v(S\cup \{i\left\}\right)-v\left(S\right)\right]$$

where ${\varphi }_i$ represents the marginal contribution of token $t_i$ to the suicide risk prediction, $S$ denotes a coalition of features excluding token $i$, and $v\left(S\right)$ is the model output for feature subset $S$. This theorem guarantees that SHAP attributions in CrisisFormer are the unique fair and consistent token importance scores.

4.3.2. LIME Analysis

LIME [16] generates locally faithful explanations by creating an interpretable surrogate model around a predicted test instance and its corresponding prediction; LIME performs this by perturbing the input texts by randomly masking tokens to observe the impact on their model’s output relative to perturbed examples. The LIME approach helps identify which words make the most relevant contributions to a predictive outcome. For every LIME explanation, a vocabulary of 20 features and 1000 perturbation samples were generated.

4.3.3. Attention Visualization

To analyze the token-to-token attention patterns learned by CrisisFormer, attention weights from the final encoder layer of the transformer were extracted to obtain an image of these attention weights. The model was reloaded using the implementation of “eager” attention for extracting the attention output; as such, the heatmap of the attention matrix was produced to examine which input (input tokens) tokens were noticed most to make a prediction.

5. Experimental Results

5.1. Performance Metrics

CrisisFormer was tested using a test set of 6000 samples. CrisisFormer was evaluated using five classification metrics: accuracy, precision, recall, F1-score, and area under the ROC Curve (AUC). The results of testing CrisisFormer for accuracy and discriminative ability of text containing suicide risk compared to text without suicide risk are presented in

Table 7. Detailed Classification Results of CrisisFormer on Test Set.

Metric	Non-Suicide Class	Suicide Class	Weighted Average
Precision	0.9800	0.9500	0.9600
Recall	0.9500	0.9800	0.9600
F1-Score	0.9600	0.9600	0.9600
Support	3000	3000	6000
Accuracy	0.9457	0.9793	0.9625
ROC-AUC	0.9921	0.9967	0.9944
MCC	0.9101	0.9403	0.9252
Cohen Kappa	0.9100	0.9400	0.9250

. The model performed with an accuracy of 96.25% and an AUC of 0.9944 as illustrated in Figure 4. Finding many false negatives (62) will have critical clinical relevance because of the high stakes of missing a suicide risk assessment.

Generalization was assessed empirically via the held-out test set (n = 6000), which showed no significant performance gap relative to validation results, indicating adequate generalization within this experimental setup. Based on theory, we know that the maximum generalization error for CrisisFormer on unseen suicide risk data cannot exceed 5.29%. Hence, the measured test error of 3.75% is well within this theoretical bound. In addition, there is a minimal amount of KL divergence between the fine-tuned and pre-trained parameters; therefore, we can conclude that DistilBERT fine-tuning preserves knowledge about linguistic features developed during pre-training and can be applied to detect suicide risk. This mathematical bound also provides an assurance of how reliable and generalizable CrisisFormer will be when deployed in real-world mental health monitoring contexts as shown in Table 7.

The Matthews Correlation Coefficient (MCC) is defined as: MCC = (TP × TN − FP × FN)/√((TP + FP)(TP + FN)(TN + FP)(TN + FN)), yielding a value of 0.9252 for CrisisFormer, confirming strong classification performance independent of class distribution.

5.2. Comparison with Baseline Models

The results of this analysis are presented in Figure 5. The performance of CrisisFormer was compared to that of pre-existing baseline models, all of which were evaluated on similar or comparable datasets. CrisisFormer shows a significant advantage over traditional machine learning and deep learning methods (CNN, LSTM, and BiLSTM). Despite achieving higher accuracy overall than RoBERTa, this was done with much larger resource demands and no explanation variables.

CrisisFormer is more accurate than LSTM and BiLSTM by 14.25% and 6.25%, respectively. CrisisFormer is also 1.96% more accurate than CNN-BiLSTM. Compared with BERT-base, CrisisFormer is slightly more accurate, has 40% fewer parameters and has much lower inference time. Finally, the CrisisFormer framework is the only framework that provides full multi-layer explainability by both SHAP and LIME, as well as attention visualization. The overall correctness across both classes is reflected in the accuracy measure. Precision is the ratio of correctly predicted positive results to all positively predicted results, and recall shows how many actual positive instances were detected. F1 is the balance between precision and recall, which is most useful with class imbalance. The Matthews Correlation Coefficient (MCC) accounts for all four cells of the confusion matrix and is less affected by shifts in the distribution of the data. Kappa (Cohen’s Kappa) reflects the expected agreement that we would see due to random chance and can provide a good estimate of classification reliability.

Let TP, TN, FP, FN denote true positives, true negatives, false positives, and false negatives, respectively. The evaluation metrics are formally defined as:

(1)

Accuracy

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(2)

Precision

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(3)

Recall

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(4)

F1-Score

$$F_1=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(5)

Area Under the ROC Curve (AUC-ROC)

$$AUC={\int }_0^{1}TPR(FP{R}^{-1}(t))\hspace{0.17em}dt$$

where:

• $\mathrm{T}\mathrm{P}\mathrm{R}=\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$ (True Positive Rate)
• $\mathrm{F}\mathrm{P}\mathrm{R}={\displaystyle \frac{\mathrm{F}\mathrm{P}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}}}$ (False Positive Rate)
• The AUC measures the model’s ability to discriminate between positive and negative classes across all classification thresholds.

A group bar graph comparing the Accuracy and F1-Score of CrisisFormer and six baseline models tested with the Suicide Watch dataset. The dark navy bars represent the accuracy values, and the forest green bars represent the F1-Score values. The red dashed line indicates the performance threshold of 95% of the total population. CrisisFormer was the best-performing model with an Accuracy of 0.9625 and an F1-Score of 0.9625. As shown in Figure 5, CrisisFormer achieved the highest accuracy (0.9625) and F1-score (0.9625) among all evaluated models, consistently outperforming both traditional machine learning and deep learning baselines while being the only framework to offer full multi-layer explainability.

5.3. Confusion Matrix Analysis

To determine how CrisisFormer performed in each of the two classes, a confusion matrix analysis was completed on the held-out test set to evaluate its classification behavior across both classes. The results show that CrisisFormer performed well overall in correctly classifying suicide-related and non-suicide-related samples, with very few false positives. Additionally, there were very few false-negative cases (clinically relevant cases of suicide risk that were not correctly identified), which have significant clinical implications, as failure to detect true cases of suicide risk in mental health screening systems can result in significant harm to patients as illustrated in Figure 6.

On the other hand, all cases of false positives were also clinically acceptable, and therefore, will be addressed through follow-up clinician review. Finally, the balanced distribution of correct predictions between the two classes provides strong evidence that CrisisFormer does not demonstrate a systematic bias against either class, which is attributable to the balanced sampling methodology used in preprocessing. These results collectively validate the effectiveness and clinical appropriateness of using CrisisFormer as an automated screening mechanism for identifying patients at risk of suicide, as seen in previous studies evaluating similar transformer-based models for classifying mental health disorders [10].

5.4. Training Dynamics and Convergence Analysis

Throughout the entire fine-tuning of CrisisFormer, convergence and stability were closely monitored during all stages of evaluation with the added generalization. Figure 7 shows that both the training and validation loss curves continuously and stably converged throughout the fine-tuning without evidence of overfitting when evaluating the loss as a function of the number of training steps. The model’s training loss was 0.65 in the first steps and decreased rapidly (monotonically) and stabilized at approximately 0.09 at Step 2600, showing that CrisisFormer had learned how to create discriminative representations of suicide risk language from the training dataset. The validation loss also started at an initial value of 0.28, and both converged to approximately 0.12 at the last evaluation step, demonstrating that the learned representations effectively generalized to unseen data.

The significant correlation between the validation and training loss curves over time indicates that the dropout regularization and early stopping methods discussed in Section 4.2 were successful in avoiding overfitting of the model being developed. For example, as represented by the validation accuracy curve, the CrisisFormer framework has shown that it has met the 95% classification accuracy goal by Step 600 of training and continues to have an accuracy greater than this goal through the rest of the training process (the maximum validation accuracy reached was 96.95% at Step 2000). The CrisisFormer framework was tested on a Tesla T4 graphics card, demonstrating its computational efficiency and making it suitable for use in resource-limited environments for mental health monitoring purposes [13].

5.5. Explainability Analysis

SHAP analysis revealed that tokens such as ‘hopeless’, ‘ending’, ‘cannot’, ‘pain’, and ‘worthless’ received the highest positive Shapley values, driving the model toward a suicide risk prediction, while emotionally positive tokens received strongly negative SHAP values, suppressing the predicted risk. Figure 8 provides a color-coded visual force plot of these token-level attributions, while

Table 8. Top 10 High-Risk Tokens Identified by SHAP Analysis.

Rank	Token	SHAP Value	Risk Direction	Clinical Significance
1	hopeless	+0.112	Suicide Risk	Indicator of hopelessness
2	ending	+0.098	Suicide Risk	Suicidal ideation marker
3	cannot	+0.087	Suicide Risk	Helplessness expression
4	Pain	+0.081	Suicide Risk	Emotional distress
5	worthless	+0.074	Suicide Risk	Low self-worth indicator
6	Tired	+0.068	Suicide Risk	Emotional exhaustion
7	Alone	+0.061	Suicide Risk	Social isolation marker
8	anymore	+0.055	Suicide Risk	Hopelessness indicator
9	crying	+0.049	Suicide Risk	Emotional distress
10	empty	+0.043	Suicide Risk	Emotional numbness

complements it by listing the top 10 high-risk tokens with their exact numerical Shapley values together, these representations serve both visual and quantitative interpretation needs for clinical readers.

Attention heatmap analysis of the last transformer encoder layer showed that CrisisFormer correctly attended to clinically significant tokens. For example, in the representative test sample, “I am considering checking myself into a psychiatric hospital, I don’t know what else to do,” the model assigned high attention weights to ‘checking’, ‘myself’, ‘hospital’, and ‘because’, showing that there is a match between the model’s attention and the clinical intuition regarding suicide risk indicators (Table 8). The LIME analysis also provided instance-level explanations demonstrating that the model’s predictions were based on meaningful semantic content and not spurious correlations or dataset artifacts.

LIME analysis indicates that model classifications are based on contextually relevant (non-spurious) features. An example of the non-suicidal sample would be where contextually relevant narrative terms such as “character,” “writing,” and “book” supported a non-suicidal classification. Conversely, an example of suicidal data is where the attention visualization (i.e., Figure 9) indicates that the model assigned greater attention to tokens that were representative of psychological distress and clinical help-seeking behaviors. This serves as a method for providing transparency regarding how the model makes its decisions. Additional evidence supporting the LIME analysis supporting model classifications is based on contextually relevant (non-spurious) features, as shown in Figure 10.

6. Discussion

CrisisFormer has shown great success as a framework for identifying social media users at risk of suicide and will provide significant contributions to the research literature, with all evaluation metrics showing high accuracy and AUC scores. A review of the academic literature on suicide risk assessment using Reddit data has confirmed that natural language processing methods can reliably identify users exhibiting linguistic characteristics of crisis-level emotional states [22], while CrisisFormer extends this line of investigation by combining computational efficiency with a multi-layered explainability architecture. Furthermore, unlike transformer models such as RoBERTa, which are much larger, CrisisFormer was created using a significantly lower number of model parameters without sacrificing prediction quality. Recent research into generalized autoregressive pretraining has similarly shown that optimized architectures can provide robust language understanding capabilities and improved accuracy on language understanding tasks with no deterioration in model performance as the model size increases or the architectures become more complicated [23].

The AUC of 0.9944 is exceptionally high; however, due to the explicit crisis labelling inherent in the content of the Suicide Watch subreddit, caution is required when interpreting this figure. The more clearly defined class boundary present in the Suicide Watch subreddit, compared to the potentially less rigorously defined class boundaries that would likely exist in a more typical social media setting, may account for a higher AUC than is typically observed within the broader scope of social media data. Additionally, an analysis of baseline performance metrics based on static pre-trained word representations, such as GloVe [24] and Word2Vec [25], demonstrated that CrisisFormer’s contextual transformer-based encodings provide more richly semantically and syntactically informed embeddings than do static embedding approaches overall. Thus, these analyses further established confidence in the robustness of CrisisFormer’s design and provided a transparent evidence base for evaluating CrisisFormer’s relative contributions to the literature.

BERT-base has been found to have accuracy scores nearly matching those of CrisisFormer on the present dataset; however, the key differentiator of CrisisFormer is its explanation module, which is part of the development of the multilayer architecture of CrisisFormer and is not standard for BERT-base. In clinical and applied settings, the ability to produce interpretable, token-level explanations has much greater practical utility than marginal improvements in predictive accuracy alone. This was evidenced by the results of RISK FORMER [26], which showed that for ethically responsible clinical deployment, the inclusion of explanation mechanisms in transformer-based suicide risk detection models is necessary. This supports the argument that the explanation of a model should be the basis of its selection in clinical situations, as opposed to solely the accuracy of the model, making CrisisFormer a more clinically viable solution than other models with no explanation provisions.

The most clinically significant finding of this study was the false-negative rate of 2.07% (62 out of 3000 instances) in relation to suicide risk assessment. The importance of minimizing false negatives cannot be understated, especially as an undetected case is an opportunity for intervention that may have saved someone’s life. Therefore, it is encouraging that the false-negative rate achieved by CrisisFormer was so low, further suggesting that the model may be an effective first-line screening tool for this purpose. CrisisFormer was built and trained using PyTorch version 1.9.0 [27], which provided a strong, efficient, deep learning framework to fine-tune DistilBERT using large amounts of text data, supporting the reproducibility and scalability of the proposed model framework.

The integration of SHAP, LIME and Attention Visualization in CrisisFormer addresses a long-standing limitation regarding providing clinically relevant (i.e., meaningful within a clinical setting) explanations for the use of deep learning NLP models in mental health. Specifically, through systematic explanatory analysis, it was demonstrated that the predictions made by CrisisFormer were based on clinically relevant (i.e., important in a clinical context) linguistic attributes, as opposed to spurious statistical patterns. The repeated identification of specific linguistic terms that corresponded to emotional distress, hopelessness, and suicide risk for each of the high-risk classification categories was highly consistent (i.e., located in concordance) with the existing clinical knowledge of the linguistic indicators associated with suicide risk. The close proximity of computational outcomes to clinical understanding can be viewed as a major success of the proposed framework and signifies a broader shift in the computational linguistics community towards developing auditable and interpretable classification systems [28], which is critically important in high-consequence domains such as mental health.

Clinicians’ ability to interrogate model explanations increases the transparency of models and supports institutional trust, while also providing an additional basis for making more informed clinical decisions. Recent empirical research has verified the effectiveness of incorporating interpretability mechanisms into transformer-based classification frameworks in multilingual text summarization [29] and computational detection of extremism [30], where interpretability has been identified as an essential facilitator for the responsible deployment of models in the real world. In addition, the ability to access auditable model predictions allows mental health professionals to conduct systematic assessments of model behavior across various demographic subgroups and different types of linguistic registers, providing a structured, principled, and transparent method of ongoing ethical oversight and accountability for the models deployed in the field.

The inferences drawn from the current investigation have limitations defined by its boundaries, which also provide a clear direction for subsequent investigations, since the training corpus generated from Reddit’s communication style may not accurately depict how varied forms of suicide are communicated among various social media and cultural/language differences. The literature [31] has found that language processing using transformer networks can be produced more generally across different platforms; hence, extending the application of the CrisisFormer framework to larger populations may require the parallel extension of the models produced in this study to accommodate greater numbers of populations. The binary classification methodology used in this study, while appropriate for the goals of this project, fails to replicate degrees of severity as assessed by clinically validated scales, thus providing an opportunity for the development of multi-class or severity-stratified variants of the framework as future research opportunities.

The method used in this study, a 50/50 balanced sampling approach, is traditionally used for binary classification tasks. Because of this imbalance, it would be beneficial for future evaluations to assess CrisisFormer on datasets that experience natural imbalances in crisis-related content and then report calibration measurements and traditional classification performance measurements. Recent applied research has shown the importance of robust validation frameworks for transformer-based models [32] that need to be followed in future studies for their results to be generalizable and applicable in the real world.

The present investigation used only one stratified train/validation/test partition (70%/15%/15%), which is the accepted method for performing large-scale transformer fine-tuning, allowing k-fold cross-validations to be included as part of the study’s future work. Implementing k-fold cross-validations would provide statistically meaningful performance estimates and lower the variance across performance metrics in reporting, thus further enhancing the support that the CrisisFormer tool can be deployed in a clinical setting. The Suicide Watch Dataset [33], which was the primary source of training and testing data in this study, provided a structured and domain-specific corpus for binary classification, while also providing a reproducible experimental baseline on which additional, more diverse dataset evaluations can be built in the future.

There is an exciting opportunity for the next wave of CrisisFormer to support a broader array of multicultural and multilingual citizens. Additional multimodal signals, such as temporal behavioral patterns and longitudinal user activity history, could increase the understanding of suicidal ideation detection through additional accuracy and contextual sensitivity. Collaborating with mental health care providers and conducting rigorous prospective clinical validation studies will serve as essential building blocks for the responsible and ethical use of CrisisFormer in clinical and public health settings, ultimately leading to better outcomes for people in crisis.

7. Conclusions

The current research study presents CrisisFormer, an explainable transformer framework that combines the computational efficiency of DistilBERT with a comprehensive multi-layer interpretability module consisting of SHAP, LIME, and attention visualizations as its interpretable components. The overall goal is to develop a framework that meets the dual challenges of high predictive performance and clinical transparency. Experimental results conducted on the Suicide Watch dataset indicate that CrisisFormer provides a moderate prediction performance of 96.25% accuracy and 0.9944 AUC (AUC is a measurement for evaluating how well a prediction can be made) while outperforming both conventional machine learning (classification) and deep learning baselines, as well as being fairly consistent with larger transformer models. In addition to providing reasonable predictive performance, the explainability analysis found that the model’s predictions were based on clinically meaningful language characteristics and, therefore, provided the information necessary to deploy the model as an appropriate tool to identify individuals at risk of experiencing a mental health crisis in an ethical manner. The findings of this study will contribute to the large body of literature on the intersection between artificial intelligence and mental health, advancing our collective vision of creating trustworthy and interpretable AI systems that can be used to provide clinical support to individuals.