Context-Aware Non-Functional Requirement Classification Using Functional Requirement Guidance and Domain-Specialized Representations

Abstract

Existing Non-Functional Requirement (NFR) classification approaches typically classify NFR statements independently, overlooking the semantic information provided by their associated Functional Requirements (FRs). Moreover, most existing models rely on general-purpose language representations and sentence-level evaluation protocols that may overestimate model generalization. This paper proposes a context-aware NFR classification framework that integrates FR context with the domain-specialized representations learned by the domain-specialized Stage-3 Re-Distill encoder. The proposed framework is evaluated under a strict project-level protocol designed to assess generalization across previously unseen software projects, together with a supplementary sentence-level 10-fold cross-validation experiment for comparison with existing studies. Experimental results demonstrate that the proposed framework achieves 0.86 Accuracy and 0.80 Macro-F1 under the project-level evaluation protocol while consistently outperforming generic pretrained transformer encoders. The experiments further reveal that the contribution of FR context is category-dependent, substantially improving Security and Usability classification while providing little or even negative benefit for Availability. Under sentence-level cross-validation, the same framework achieves a Macro-F1 score of 0.97, highlighting the strong influence of evaluation protocols on reported performance. These findings demonstrate that domain-specialized representations and FR context provide complementary benefits for NFR classification while highlighting the importance of realistic evaluation protocols for assessing model generalization.

1. Introduction

Requirements Engineering (RE) is a fundamental part of software development that focuses on the elicitation, analysis, documentation, validation, and management of software requirements throughout the software lifecycle [1,2]. Its main outcome is the Software Requirements Specification (SRS), which serves as the foundation for software design, implementation, testing, and maintenance. An SRS typically consists of two complementary requirement types: Functional Requirements (FRs) and Non-Functional Requirements (NFRs). FRs describe the functionality that a system must provide, whereas NFRs specify quality attributes and operational constraints.

Although FRs describe system functionality, NFRs determine how well these functions are delivered and therefore play a critical role in software architecture, design decisions, and overall software quality [3,4]. Consequently, accurately identifying NFRs during the early stages of software development is essential for reducing development cost, avoiding design conflicts, and improving system reliability [3,4].

Despite their importance, NFRs remain considerably more difficult to identify and classify than FRs. In contrast to FRs, which are generally stated explicitly by stakeholders, NFRs are often ambiguous, implicitly expressed, or embedded within functional descriptions [5,6]. Consequently, manually identifying quality concerns is both time-consuming and error-prone. Different stakeholders may use different vocabulary, sentence structures, or domain-specific terminology to describe the same quality concern [7]. These challenges become even more pronounced in large Software Requirements Specification (SRS) documents containing hundreds of natural language requirements.

This study focuses specifically on the classification of NFRs into four target categories: Performance, Security, Usability, and Availability. These categories are used as operational classification labels inherited from the previously constructed Re-Distill FR–NFR corpus [8]. The target categories were selected during corpus construction based on their representation across the PROMISE and PURE sources and their suitability for constructing verified FR–NFR associations. Although these labels are related to the ISO/IEC 25010 quality model [9], Availability is formally defined in the standard as a sub-characteristic of Reliability rather than as a top-level quality characteristic. It is nevertheless retained as a separate classification label in this study to preserve consistency with the original corpus taxonomy.

Over the past decade, numerous automated approaches have been proposed for NFR classification, evolving from traditional machine learning techniques to deep learning, Transformer-based models, and, more recently, Large Language Models (LLMs) [10,11,12]. Although these approaches have substantially improved the semantic understanding of software requirements, most existing methods continue to classify each NFR statement independently, assuming that the target NFR alone provides sufficient information for classification.

In practice, however, quality requirements rarely exist in isolation. Many NFRs derive part of their meaning from their associated FRs. For example, determining whether a requirement concerns performance, security, or usability often depends on understanding the functionality being described. Nevertheless, most existing NFR classification approaches do not explicitly incorporate this functional context and classify requirements using only the NFR statement. Consequently, such models may struggle to distinguish between quality attributes that are semantically similar but contextually different.

In our previous work [8], we introduced Re-Distill, a retrieval framework that models the semantic relationship between FRs and NFRs through multi-stage domain adaptation and knowledge distillation. The experimental results demonstrated that FRs contain valuable semantic information for retrieving their associated NFRs, suggesting that FRs may also provide useful contextual cues for downstream NFR classification. This finding motivates the central hypothesis of the present study: if FRs improve FR–NFR retrieval, can they also improve NFR classification when incorporated as contextual information?

Motivated by these observations, this paper proposes a context-aware NFR classification framework that integrates FR context with the domain-specialized semantic representations learned by Re-Distill. Unlike conventional approaches that classify NFR statements independently, the proposed framework utilizes the associated FRs to provide additional semantic context during classification.

Furthermore, unlike many previous studies that rely on sentence-level evaluation, this work adopts a strict project-level evaluation protocol to assess the generalization capability of the proposed framework under more realistic deployment conditions.

The proposed framework is evaluated through the following research questions:

RQ1. 

How effective is the proposed context-aware framework for NFR classification under a strict project-level evaluation protocol?

RQ2. 

Does using a domain-specialized encoder improve NFR classification performance compared with general-purpose language models?

RQ3. 

To what extent does Functional Requirement context influence the classification performance of different NFR categories?

To address these research questions, the main contributions of this work are summarized as follows:

• Proposing a context-aware NFR classification framework by explicitly incorporating FR context into the classification process, an aspect that has received limited attention in previous NFR classification research.
• Extending and reformulating the previously constructed FR–NFR retrieval dataset into a context-aware classification benchmark comprising 1915 labeled FR–NFR instances.
• Demonstrating that the contribution of FR context is category-dependent, providing substantial benefits for some NFR categories while offering limited or no benefit for others.
• Evaluating model generalization under both strict project-level splitting and supplementary cross-validation protocols, highlighting the influence of evaluation strategies on reported performance.

The remainder of the paper is organized as follows. Related studies are reviewed in Section 2, followed by the proposed methodology in Section 3. Experimental results are presented in Section 4 and interpreted in Section 5. Threats to validity are discussed in Section 6, and the paper concludes with conclusions and future research directions in Section 7.

2. Related Work

Early studies on automated NFRs classification relied primarily on traditional machine learning algorithms, including Naïve Bayes, Support Vector Machines, Logistic Regression, decision trees, and ensemble classifiers, together with manually engineered lexical and syntactic representations such as Bag-of-Words, TF–IDF, keyword features, and part-of-speech information [13,14,15]. For example, Canedo and Mendes [13] evaluated several conventional machine learning algorithms for software requirements classification, while Abad et al. [14] investigated the influence of preprocessing and feature design on classification performance. These approaches established the feasibility of automated NFR classification and laid the foundation for subsequent research on learning richer semantic representations.

To overcome the limitations of handcrafted feature engineering, subsequent studies adopted deep learning architectures such as Convolutional Neural Networks (CNNs), Bidirectional Long Short-Term Memory (BiLSTM) networks, and ensemble models integrating multiple neural architectures [16,17]. For example, Rahimi et al. [16] proposed a two-phase ensemble framework combining CNN, LSTM, GRU, and BiLSTM models, whereas Baskoro et al. [17] demonstrated the effectiveness of CNN-based models over conventional machine learning approaches for software requirements classification. These developments significantly improved contextual feature learning and motivated the adoption of more powerful language representation models.

The introduction of transformer-based language models significantly advanced automated NFR classification by enabling richer contextual representations than previous deep learning architectures. Recent studies have investigated several transformer-based paradigms, including conventional fine-tuning, transfer learning, prompt learning, domain-adaptive transformer models, and task reformulation strategies [10,11,18,19,20,21].

For example, Luo et al. [18] introduced PRCBERT, a prompt-learning framework for requirement classification, while Hey et al. [19] proposed NoRBERT, a transfer-learning framework specialized for software requirements. Subsequent studies further explored pretrained transformer representations for addressing limited-data scenarios [20] and developed transformer-based deep learning frameworks specifically tailored for NFR classification [21]. Fazelnia et al. [11] subsequently reformulated NFR classification as a Natural Language Inference (NLI) task to better exploit semantic relationships between requirement statements, whereas Ayed and Alrammal [10] benchmarked fine-tuned BERT models across the PROMISE and PURE datasets. Despite the diversity of these transformer-based formulations, they share the common objective of improving contextual representations for individual requirement statements, thereby advancing automated NFR classification.

More recently, Large Language Models (LLMs) have attracted increasing attention within requirements engineering for tasks such as requirements modeling and the generation of software-engineering artifacts [12,22]. For example, Ruan et al. [12] investigated the use of ChatGPT with GPT-3.5 to support requirements modeling in embedded systems, while Arora et al. [22] proposed a retrieval-augmented LLM approach for generating test scenarios from natural-language requirements. While these studies demonstrate the growing potential of LLMs within requirements engineering, their application to automated NFR classification remains relatively limited.

Overall, the literature reflects a clear evolution from traditional feature-engineering approaches to deep learning, transformer-based models, and, more recently, LLM-driven methods. Despite this rapid progress, three important research gaps remain. First, existing NFR classification approaches generally classify requirement statements independently, without explicitly incorporating their associated functional requirements as contextual information [10,11,18,19]. Second, although prior studies have explored transfer learning, prompt-based learning, and other task-specific adaptations for requirements classification, limited attention has been given to representations explicitly specialized for modeling semantic associations between FRs and NFRs [11,18,19]. Third, previous NFR classification studies commonly evaluate models using sentence-level hold-out or cross-validation strategies [10,11,18,23], leaving cross-project generalization to entirely unseen software projects insufficiently examined. Recent evidence further shows that requirement classifiers can experience substantial performance degradation when evaluated across datasets or domains [24].

To address these research gaps, the proposed framework combines domain-specialized representations learned through Re-Distill, FR context, and a strict project-level evaluation protocol to assess classification performance and cross-project generalization.

3. Methodology

This section presents the methodology adopted in this study. It begins by describing the source dataset and its reformulation into a supervised classification benchmark. It then presents the proposed context-aware input representation, the classification framework, and the training procedure. Finally, it introduces the evaluation protocols and performance metrics used throughout all experiments.

3.1. Source Dataset

The dataset used in this study is derived from the FR–NFR retrieval corpus introduced in our previous work, Re-Distill [8]. The corpus combines software projects collected from the PROMISE repository [25] and selected projects from the PURE repository [26]. Since the dataset construction, annotation procedure, and preprocessing pipeline have already been described in detail in the previous study, only the characteristics relevant to the present classification task are summarized here.

The corpus consists of 43 software projects collected from multiple application domains and includes 1162 FRs linked to 406 manually labeled NFRs. Each NFR belongs to one of four quality categories: Performance, Security, Availability, or Usability. These categories were retained to preserve the established FR–NFR mappings and classification taxonomy of the original corpus.

Table 1. Distribution of labeled NFR statements across the four target quality categories.

NFR Category	Count	Percentage (%)
Performance	178	43.84
Security	117	28.82
Usability	78	19.21
Availability	33	8.13
Total	406	100.00

summarizes the distribution of labeled NFR statements across the four target quality categories. The distribution is naturally imbalanced, reflecting the characteristics of real software requirement specifications, and is therefore preserved throughout all experiments. Performance is the largest category, followed by Security, Usability, and Availability.

The original project-level partitioning strategy established in Re-Distill was preserved throughout all experiments. Consequently, complete software projects remain isolated across the training, validation, and test subsets, preventing any project from appearing in more than one partition. This design reduces the risk of information leakage and provides a more realistic estimate of model generalization than sentence-level random splitting, which has been widely recognized as an important consideration in machine learning evaluation [27]. Representative application domains included in each subset are summarized in

Table 2. Distribution of software domains across the project-level training, validation, and test partitions.

Split	Example Domains
Training	Banking, Enterprise Systems, Scheduling, Public Web Portals, Database Systems, Development Tools
Validation	Medical Education, Resource Booking, Supply Chain, GIS Systems, Media Platforms
Testing	Real Estate, E-Commerce, CRM, Graphics Utilities, Embedded Systems

.

The independence of these subsets was previously validated using both Jaccard overlap and cosine similarity analyses [8]. The results showed negligible lexical overlap while preserving semantic diversity across software domains, thereby minimizing information leakage and providing a realistic benchmark for evaluating model generalization.

3.2. Classification Dataset Construction

Although the original dataset was developed to support FR–NFR retrieval in Re-Distill [8], the objective of the present study is supervised NFR classification. Therefore, the retrieval dataset was reformulated into a classification dataset in which each verified FR–NFR association represents an independent labeled training instance.

Within the original dataset, a single FR may be linked to multiple NFRs belonging to different quality categories. Consequently, the original representation cannot be directly used for supervised classification. Each verified FR–NFR association was therefore converted into an independent classification instance while preserving the original project-level partition.

To construct the classification dataset, all NFR associations were first separated into individual records. Each NFR was then matched with its corresponding quality category using the manually curated NFR category mapping developed during the dataset construction process. The matching procedure relied on both the project identifier and the preprocessed NFR text to ensure consistent alignment between the retrieval associations and their corresponding classification labels. Figure 1 illustrates the reformulation process. Each verified FR–NFR association becomes an independent supervised classification instance while preserving the original FR and its associated quality label.

Table 3. Statistics of the reformulated FR–NFR classification dataset.

Dataset	Instances
Training	1336
Validation	316
Testing	263
Total	1915

summarizes the resulting classification dataset. By preserving the original project-level partition established in Re-Distill, the reformulation produced 1336 training instances, 316 validation instances, and 263 testing instances, yielding a total of 1915 labeled FR–NFR classification instances that serve as the benchmark throughout this study.

3.3. Context-Aware Input Representation

The primary objective of this study is to investigate whether FR context improves NFR classification. To isolate the contribution of FR context, two input formulations were constructed from the same classification dataset:

• Context-Aware (FR + NFR): the model receives both the FR and its associated target NFR.
• NFR-Only: the model receives only the target NFR and serves as the baseline configuration.

The proposed formulation is referred to as context-aware because the FR serves solely as contextual information rather than an additional prediction target, providing complementary semantic cues for interpreting the target NFR.

In the context-aware formulation, the input explicitly distinguishes the two components using the field labels “Functional Requirement (FR)” and “Target NFR”, allowing the encoder to recognize their respective roles while preserving the original requirement text. In the NFR-only formulation, the FR component is omitted, and only the target NFR is provided. Both formulations employ the same task instruction and the same fixed set of category-specific examples to clarify the classification objective, following the general principle of instruction-based prompting in NLP [28].

To ensure a fair comparison, both formulations use exactly the same training, validation, and testing instances, as well as the same encoder, optimization strategy, hyperparameters, prompt template, category-specific examples, and evaluation protocol. Consequently, any observed performance differences can be attributed solely to the inclusion or exclusion of FR context.

Figure 2 illustrates the two input formulations evaluated in this study. The only difference between them is whether the associated FR is included alongside the target NFR.

3.4. Proposed Classification Framework

Figure 3 illustrates the overall architecture of the proposed context-aware NFR classification framework. The framework builds upon the Stage-3 Re-Distill encoder introduced in our previous work [8], reusing it as a domain-specialized backbone for downstream NFR classification. Stage-3 builds on the domain-specific representations learned in the earlier stages and further refines them using curriculum-guided distillation and teacher-guided hard-negative mining. This refinement helps the encoder capture the semantic relationship between FRs and NFRs, allowing it to better use FR context for NFR classification.

Within the proposed framework, the input sequence is first tokenized and embedded using the Re-Distill tokenizer before being processed by the transformer encoder to generate domain-specialized contextual representations. These representations are then processed through the feature fusion and classification components described below.

Given an input sequence X, the Stage-3 Re-Distill encoder produces contextual representations across all hidden layers. Instead of relying solely on the final hidden layer, the proposed framework extracts the contextual representation of the special [CLS] token from each of the last four transformer layers. These representations are then combined through mean pooling to obtain a single fused feature representation:

$$h_{\mathrm{fusion}}={\displaystyle \frac{1}{4}}{\displaystyle \sum _{i=L-3}^L}{\mathrm{CLS}}^{\left(i\right)},$$

(1)

where L denotes the final transformer layer, ${\mathrm{CLS}}^{\left(i\right)}$ represents the contextual embedding of the [CLS] token produced by layer i, and $h_{\mathrm{fusion}}$ denotes the final fused feature representation. By integrating representations from the last four transformer layers, the proposed framework follows the feature extraction strategy introduced in BERT, where combining multiple upper layers was shown to produce richer contextual representations than relying solely on the final layer [29]. This design is further supported by analyses demonstrating that different transformer layers capture complementary linguistic and semantic information [30,31].

The fused representation is subsequently passed through a dropout layer [32] to improve model generalization before being projected into a linear classification layer that produces four output logits corresponding to the target NFR categories:

$$z=W{h}_{\mathrm{fusion}}+b,$$

(2)

where W and b denote the learnable classifier parameters and z represents the output logits corresponding to the four target NFR categories.

During inference, the logits are converted into class probabilities using Softmax, and the category with the highest probability is selected using the Argmax operator as the final prediction.

3.5. Training and Implementation Details

All experiments were implemented in Python (version 3.13) using PyTorch (version 2.2.2), Hugging Face Transformers (version 4.39.3), Sentence-Transformers (version 2.6.1), and Scikit-learn (version 1.6). The same implementation settings were adopted for both the context-aware and NFR-only formulations to ensure a fair comparison. The main training and implementation settings are summarized in

Table 4. Training and implementation settings.

Parameter	Value
Optimizer	AdamW
Learning Rate	$2\times {10}^{-5}$
Weight Decay	0.01
Batch Size	16
Maximum Sequence Length	384
Dropout Rate	0.30
Loss Function	Focal Loss
Focal Loss $\gamma$	2.0
Maximum Epochs	14
Model Selection Metric	Validation Macro-F1
Random Seed	42
Training Platform	Google Colab Pro (GPU)

.

The hyperparameter configuration was determined through preliminary validation experiments and then kept fixed throughout all comparative experiments. The proposed framework was fine-tuned using the AdamW optimizer [33] with an initial learning rate of $2\times {10}^{-5}$. Since the model builds upon the Stage-3 Re-Distill encoder developed in our previous retrieval study, a relatively small learning rate was adopted to preserve the domain-specialized representations learned during retrieval training while adapting the encoder to the downstream NFR classification task.

A batch size of 16 was selected to balance optimization stability and GPU memory constraints. A maximum input length of 384 tokens was used throughout all experiments. The maximum sequence length was selected to accommodate both the FR context and the target NFR within a single input sequence while minimizing sequence truncation. A dropout rate of 0.3 and a weight decay coefficient of 0.01 were applied during training to reduce overfitting and improve model generalization.

Training was performed for a maximum of 14 epochs, with early model selection based on the highest validation Macro-F1 score. To address the natural class imbalance of the dataset, the proposed framework was optimized using Focal Loss [34] rather than the standard cross-entropy loss. Focal Loss emphasizes difficult and minority-class instances while down-weighting easily classified examples. It is defined as

$$\mathrm{FL}\left(p_t\right)=-{(1-p_t)}^{\gamma }log\left(p_t\right),$$

(3)

where $p_t$ denotes the predicted probability assigned to the correct class and $\gamma$ is the focusing parameter. Throughout all experiments, $\gamma$ was fixed to 2.0.

After each training epoch, model performance was evaluated on the validation set using Macro-F1 as the primary model-selection criterion. The checkpoint achieving the highest validation Macro-F1 score was retained for final evaluation on the held-out test set. The random seed was fixed to 42 for all experiments to improve the reproducibility of the reported results.

3.6. Evaluation Protocol

The proposed framework was evaluated using two complementary evaluation protocols designed to assess both realistic cross-project generalization and comparability with existing literature. This section first describes the evaluation protocols and then introduces the performance metrics used throughout all experiments.

3.6.1. Primary Evaluation (Project-Level Split)

The primary evaluation follows the strict project-level partitioning strategy introduced in Re-Distill [8]. Entire software projects remain isolated across the training, validation, and test partitions, ensuring that no project appears in more than one subset.

This protocol presents a substantially more challenging evaluation scenario than sentence-level splitting because software projects originating from different application domains often employ different terminologies, naming conventions, and requirement-writing styles. Consequently, the model must generalize beyond project-specific vocabulary and learn transferable semantic representations of software requirements.

This strict project-level evaluation protocol provides a more realistic assessment of model generalization to previously unseen software projects from different application domains. Unless otherwise stated, all reported experimental results are based on this primary evaluation protocol.

3.6.2. Supplementary Evaluation (Sentence-Level Cross-Validation)

To facilitate comparison with previous NFR classification studies, an additional stratified sentence-level 10-fold cross-validation experiment was conducted using the complete classification dataset [35].

Unlike the primary project-level evaluation, sentence-level cross-validation randomly partitions individual FR–NFR instances while preserving class distributions across folds. Consequently, requirements originating from the same software project may appear in both the training and testing subsets. Although this protocol provides a less stringent assessment of generalization to unseen projects, it represents the evaluation strategy most commonly adopted in the existing NFR classification literature.

The results obtained under this supplementary evaluation protocol are reported separately and are intended solely to facilitate comparison with previous studies. They do not replace the primary project-level evaluation adopted in this work.

3.6.3. Evaluation Metrics

The proposed framework was evaluated using three complementary performance metrics: Accuracy, Macro-F1, and Weighted-F1 [36]. In addition, per-class F1-scores were reported to analyze the effect of FR context on each target NFR category individually.

Accuracy measures the overall proportion of correctly classified instances and provides an overall indication of classification performance. It is computed as

$$\mathrm{Accuracy}={\displaystyle \frac{{\sum }_{i=1}^{C}T{P}_i}{N}},$$

(4)

where $T{P}_i$ denotes the number of correctly classified instances belonging to class i, C is the total number of classes, and N is the total number of evaluation instances.

Because the classification dataset is naturally imbalanced across the four NFR categories, Macro-F1 was selected as the primary evaluation metric. Unlike Accuracy, Macro-F1 assigns equal importance to every class regardless of its frequency and therefore provides a more reliable assessment of performance on minority categories such as Availability. Macro-F1 is computed as

$$\mathrm{Macro}-\mathrm{F}1={\displaystyle \frac{1}{C}}{\displaystyle \sum _{i=1}^C}F{1}_i,$$

(5)

where $F{1}_i$ denotes the F1-score of class i and C is the total number of classes.

Weighted-F1 was also reported as a complementary performance metric. Unlike Macro-F1, this metric computes the weighted average of class-specific F1-scores according to the number of instances belonging to each class:

$$\mathrm{Weighted}-\mathrm{F}1={\displaystyle \sum _{i=1}^C}{\displaystyle \frac{n_i}{N}}F{1}_i,$$

(6)

where $n_i$ denotes the number of instances belonging to class i, and N is the total number of evaluation instances.

Although Weighted-F1 provides a useful measure of overall performance, its class-frequency weighting gives greater influence to majority classes. Consequently, strong performance on frequently occurring categories may compensate for weaker performance on minority categories. For this reason, Macro-F1 was adopted as the primary criterion for model selection and comparison throughout this study, whereas Weighted-F1 was reported as a complementary measure of overall classification performance.

Finally, per-class F1-scores were reported to examine whether the contribution of FR context differs across individual NFR categories.

4. Results

This section presents the experimental results of the proposed framework. The primary project-level evaluation is first reported to answer the three research questions, followed by a supplementary sentence-level cross-validation experiment for comparison with previous studies.

4.1. Primary Evaluation

The primary evaluation addresses the three research questions of this study. It first evaluates the overall performance of the proposed framework, then examines the contribution of the domain-specialized encoder, and finally investigates the impact of incorporating FR context into NFR classification.

4.1.1. RQ1: Overall Performance of the Proposed Framework

To answer RQ1, the proposed context-aware framework was evaluated under the primary project-level evaluation protocol described in Section 3.6.1. This protocol serves as the primary evaluation setting because it assesses the ability of the proposed framework to generalize to previously unseen software projects originating from different application domains.

Table 5. Performance of the proposed context-aware framework under the primary project-level evaluation protocol.

NFR Category	Precision	Recall	F1-Score
Performance	0.94	0.82	0.87
Security	0.85	0.71	0.77
Availability	0.52	0.92	0.67
Usability	0.81	0.95	0.88
Macro Average	0.78	0.85	0.80
Weighted Average	0.87	0.85	0.85
Accuracy	–	–	0.86

summarizes the classification performance on the held-out test set.

As shown in Table 5, the proposed framework achieved an Accuracy of 0.86, a Macro-F1 score of 0.80, and a Weighted-F1 score of 0.85 under the primary project-level evaluation protocol. Since Macro-F1 was adopted as the primary model-selection and evaluation metric, these results indicate balanced performance across both majority and minority classes.

At the class level, the proposed framework achieved the highest F1-scores for Performance (0.87) and Usability (0.88), indicating that these quality attributes can be identified reliably even under strict project-level evaluation. Security achieved an F1-score of 0.77, reflecting the greater semantic similarity between security-related requirements and other quality attributes. Availability obtained an F1-score of 0.67 despite being the smallest category, demonstrating that the proposed framework remains effective even for minority classes.

To further examine the classification behavior across individual categories, the corresponding confusion matrix is presented in Figure 4. The majority of predictions are concentrated along the main diagonal, demonstrating consistent classification performance across the four target NFR categories. The majority of misclassifications originate from the Performance category, particularly toward Usability and, to a lesser extent, Availability. Nevertheless, comparatively few errors are observed for the remaining categories, indicating robust overall classification performance under the strict project-level evaluation protocol.

Overall, these results demonstrate that the proposed framework achieves robust NFR classification under the strict project-level evaluation protocol. The following section investigates the contribution of the domain-specialized encoder to this overall performance.

4.1.2. RQ2: Effect of the Domain-Specialized Encoder

This experiment investigates the contribution of domain-specific encoder specialization to the overall classification performance. To this end, the proposed Stage-3 Re-Distill encoder is compared with three widely used pre-trained language models serving as general-purpose baselines, namely BERT-base [29], RoBERTa-base [37], and MPNet (all-mpnet-base-v2) [38]. All remaining components of the proposed framework were kept identical, including the input formulation, classification architecture, last-four-layer fusion strategy, optimization settings, and evaluation protocol, thereby strictly isolating the effect of the underlying encoder representations.

Unlike the general-purpose encoders, Re-Distill was previously adapted to the software requirements domain through multi-stage domain adaptation and knowledge distillation, enabling it to learn software requirements-specific semantic representations before being fine-tuned for the downstream classification task [8].

Table 6. Performance comparison of the evaluated backbone encoders under the primary project-level evaluation protocol.

Encoder	Acc.	Macro-F1	Weighted-F1	Perf. F1	Sec. F1	Avail. F1	Usab. F1
BERT-base	0.63	0.59	0.68	0.66	0.29	0.56	0.86
RoBERTa-base	0.69	0.62	0.71	0.70	0.44	0.50	0.84
MPNet	0.71	0.64	0.73	0.75	0.43	0.55	0.82
Re-Distill	0.86	0.80	0.85	0.87	0.77	0.67	0.88

demonstrates that the domain-specialized Re-Distill encoder consistently outperformed all generic pre-trained encoders across every evaluation metric. Among the general-purpose encoders, MPNet achieved the strongest overall performance with a Macro-F1 score of 0.64, followed by RoBERTa (0.62) and BERT (0.59). In contrast, Re-Distill achieved a Macro-F1 score of 0.80, a Weighted-F1 score of 0.85, and an Accuracy of 0.86.

Compared with the strongest general-purpose encoder (MPNet), Re-Distill improved Macro-F1 by approximately 15 percentage points while increasing overall Accuracy from 0.71 to 0.86. These improvements demonstrate that encoder specialization contributes substantially more to classification performance than simply replacing one generic language model with another. Since MPNet serves as the underlying backbone of Re-Distill, this comparison directly measures the contribution of the proposed domain-specialization process rather than differences in transformer architecture.

The class-level analysis further shows that the largest improvement was obtained for the Security category. While the generic encoders achieved Security F1-scores ranging from 0.29 to 0.43, Re-Distill increased Security F1 to 0.77. Similar improvements were observed for Performance and Usability, whereas Availability remained the most challenging category because of its limited number of training instances.

The confusion matrices shown in Figure 5 further support these observations. While the generic encoders exhibit substantial confusion between several NFR categories, the domain-specialized Re-Distill encoder produces a noticeably clearer diagonal structure with fewer inter-class misclassifications, indicating more discriminative representations.

Overall, these results demonstrate that domain-specific specialization substantially enhances the representation capability of the underlying MPNet backbone, leading to consistently better classification performance under the strict project-level evaluation protocol.

4.1.3. RQ3: Effect of FR Context

To answer RQ3, an input ablation study was conducted to quantify the contribution of FR context to NFR classification. Two input formulations were evaluated under identical experimental conditions. The proposed context-aware formulation uses both the FRs and their associated NFRs as model input, whereas the baseline formulation uses only the target NFRs. All remaining components of the proposed framework, including the Stage-3 Re-Distill encoder, classification architecture, optimization strategy, and evaluation protocol, were kept unchanged to isolate the effect of FR context.

Table 7. Performance comparison between the context-aware (FR+NFR) and NFR-only input formulations under the primary project-level evaluation protocol.

Input Setting	Accuracy	Macro-F1	Weighted-F1	Perf.	Sec.	Avail.	Usab.
Context-Aware	0.86	0.80	0.85	0.87	0.77	0.67	0.88
NFR-Only	0.78	0.77	0.80	0.83	0.54	0.92	0.80

summarizes the resulting performance comparison.

As shown in Table 7, the two input formulations achieved similar overall Macro-F1 scores. The context-aware formulation achieved a Macro-F1 score of 0.80, compared with 0.77 for the NFR-only formulation. More importantly, incorporating FR context increased the overall Accuracy from 0.78 to 0.86 and the Weighted-F1 score from 0.80 to 0.85, suggesting that contextual information improves the robustness of the classifier under the primary project-level evaluation protocol.

A more informative pattern emerges from the class-level analysis. The contribution of FR context is clearly category-dependent rather than uniform across all quality attributes. Security and Usability benefit substantially from contextual information, with Security F1 increasing from 0.54 to 0.77 and Usability F1 from 0.80 to 0.88. These improvements suggest that the corresponding NFR statements often require knowledge of the associated system functionality to be interpreted correctly.

In contrast, Availability exhibits the opposite behavior. The NFR-only formulation achieved a substantially higher F1-score (0.92 versus 0.67), suggesting that Availability requirements frequently contain explicit lexical indicators such as availability, uptime, or service continuity. Consequently, these requirements can often be classified correctly from the NFR statement alone without requiring additional FR context. Performance exhibited only a modest difference between the two formulations (0.87 versus 0.83), indicating that this category is comparatively less dependent on additional FR context.

Figure 6 further illustrates the effect of FR context. The context-aware formulation particularly reduces confusion between Security and Usability requirements, whereas the NFR-only formulation produces noticeably more misclassifications between these categories.

Figure 7 summarizes these findings from a class-level perspective. While the influence of FR context is limited for Performance and even detrimental for Availability (which exhibit limited or negative gains from additional context), Security and Usability benefit substantially from incorporating the associated FRs, highlighting the category-dependent nature of contextual information.

Overall, these findings demonstrate that the contribution of FR context is category-dependent rather than universal. While some NFR categories can largely be inferred from the NFR statement alone, Security and Usability benefit substantially from the additional semantic context provided by the associated FR.

4.2. Supplementary Evaluation Using Sentence-Level Cross-Validation

Although the primary project-level evaluation serves as the principal benchmark adopted in this study, an additional sentence-level 10-fold cross-validation experiment was conducted to facilitate comparison with previous studies employing the conventional evaluation protocol.

Table 8. Performance of the proposed framework under the supplementary sentence-level 10-fold cross-validation protocol.

Evaluation Protocol	Acc.	Macro-F1	Weighted-F1	Perf. F1	Sec. F1	Avail. F1	Usab. F1
10-fold CV	0.97	0.97	0.97	0.97	0.96	0.95	0.98

summarizes the average classification performance obtained across the ten folds.

As shown in Table 8, the proposed framework achieved approximately 0.97 across Accuracy, Macro-F1, and Weighted-F1 under the sentence-level 10-fold cross-validation protocol. All four NFR categories also achieved consistently high class-level F1-scores exceeding 0.95.

Compared with the primary project-level evaluation, substantially higher performance was observed under the sentence-level protocol. Since the model architecture, training procedure, dataset, and hyperparameter configuration remained unchanged, the observed improvement is primarily attributable to the evaluation protocol rather than the underlying classification framework. The implications of this difference are discussed in Section 5.

The results presented in this section are intended solely to facilitate comparison with previous studies employing sentence-level evaluation protocols. Their implications for model generalization and the influence of evaluation methodology are discussed in the following sections.

4.3. Comparison with Previous Studies

Table 9. Comparison of the proposed framework with representative NFR classification studies.

Study	Evaluation Protocol	Core Architecture	Dataset	Macro-F1	Weighted-F1
Ayed & Alrammal (2025) [10]	Hold-out	Fine-tuned BERT	PROMISE	0.72	0.86
Ayed & Alrammal (2025) [10]	Hold-out	Fine-tuned BERT	PURE	0.60	0.93
Fazelnia et al. (2024) [11]	Sentence-level 10-fold CV	RoBERTa-based NLI	PROMISE	0.83	–
PRCBERT (2022) [18]	Sentence-level 10-fold CV	Prompt learning	PROMISE	–	0.96
Rahimi et al. (2021) [16]	Hold-out	Ensemble deep learning	PROMISE	–	0.88
Kaur & Kaur (2023) [23]	Sentence-level 10-fold CV	BERT–BiLSTM–CNN	PROMISE	–	0.94
This Work (Project-Level)	Strict project-level split	Re-Distill + context-aware classification	PROMISE + selected PURE projects	0.80	0.85
This Work (Sentence-Level CV)	Sentence-level 10-fold CV	Re-Distill + context-aware classification	PROMISE + selected PURE projects	0.97	0.97

positions the proposed framework relative to representative NFR classification studies employing different learning architectures, datasets, and evaluation protocols. Since these methodological differences limit direct numerical comparability, the reported results are presented as a contextual comparison rather than a strict ranking of model performance.

Results from both evaluation protocols adopted in this study are included. The strict project-level protocol represents the primary evaluation setting for assessing cross-project generalization, whereas the supplementary sentence-level 10-fold cross-validation results facilitate comparison with studies using conventional sentence-level evaluation.

As shown in Table 9, the proposed framework achieves competitive performance under both evaluation settings while retaining a unified multi-class classification formulation. The project-level results provide the primary evidence of generalization to unseen projects, whereas the sentence-level cross-validation results demonstrate performance under a protocol more directly comparable with much of the existing literature. The methodological implications of these differences are discussed in Section 5.

5. Discussion

The results presented in the previous section provide insights into the respective contributions of domain-specialized representation learning, FR context, and evaluation methodology to NFR classification performance. This section discusses these findings in relation to previous studies and highlights their implications for future requirements engineering research.

5.1. Domain-Specialized Representation Learning

The results presented in Section 4.1.2 demonstrate that domain-specialized representation learning substantially improves NFR classification performance. Although all evaluated models shared the same classification architecture and training configuration, the Stage-3 Re-Distill encoder consistently outperformed the generic pretrained encoders across all evaluation metrics. Notably, because Re-Distill shares the same MPNet backbone as one of the evaluated baselines, this comparison isolates the effect of domain specialization while controlling for the underlying transformer architecture. These findings suggest that domain specialization contributes more substantially to classification performance than replacing one generic pretrained encoder with another.

Unlike BERT, RoBERTa, and MPNet, the Re-Distill encoder was initialized from the MPNet backbone and subsequently specialized through the multi-stage representation learning framework proposed in our previous work [8]. This specialization process combined domain fine-tuning, curriculum distillation, and hard-negative distillation to learn software-specific terminology, requirement-writing patterns, and semantic relationships that are not explicitly captured by general-purpose language models. The complete description of the multi-stage specialization framework is provided in our previous work [8].

The results further suggest that the benefits of domain-specialized representation learning are particularly evident for semantically challenging NFR categories. The largest improvement was observed for the Security category, whose requirements often contain subtle semantic distinctions and domain-specific terminology.

Moreover, although Re-Distill was originally developed for FR–NFR retrieval [8], the present study demonstrates that the learned representations transfer effectively to downstream NFR classification. This finding suggests that retrieval-oriented representation learning provides a robust semantic foundation that can be effectively reused for related requirements engineering tasks.

5.2. Functional Context Is Category-Dependent

One of the principal findings of this study is that the contribution of FR context varies across NFR categories rather than benefiting them uniformly. Although incorporating FR context improves the overall robustness of the proposed framework, its effectiveness depends on the semantic characteristics of the target quality attribute. This observation suggests that FR context should be viewed as a category-dependent semantic cue rather than a universally beneficial source of information.

Security and Usability benefit most from the inclusion of FR context because their interpretation often depends on the functionality to which they are attached. Requirements describing authentication, authorization, privacy, or user interaction are frequently ambiguous when considered in isolation. The associated FR provides additional operational context that helps disambiguate these categories from other semantically similar quality attributes.

In contrast, Availability shows a different behavior. Availability requirements are commonly expressed using explicit operational constraints and domain-specific keywords such as availability, uptime, recovery, or service continuity. Consequently, the NFR statement itself often contains sufficient semantic information for correct classification, reducing the relative importance of the associated FR. Similar behavior was observed for the Performance category, although the effect was considerably smaller, suggesting that many performance requirements also contain explicit lexical indicators that remain informative without additional contextual information.

Taken together, these findings suggest that FR context functions primarily as a semantic disambiguation mechanism rather than a universal performance enhancer. Its contribution becomes most valuable when the meaning of an NFR depends strongly on the associated system functionality, whereas categories expressed through explicit lexical patterns may require little additional contextual information. This observation provides a more nuanced understanding of context-aware NFR classification and represents one of the main contributions of the present study.

The observed category-dependent behavior further suggests that future context-aware NFR classification systems may benefit from category-adaptive architectures, where contextual information is selectively incorporated only for categories that demonstrably require it.

5.3. Generalization and Evaluation Protocols

One of the primary objectives of this study was to evaluate NFR classification under conditions that more closely resemble real-world software engineering practice. For this reason, the proposed framework was primarily evaluated using a strict project-level protocol in which complete software projects were isolated across the training, validation, and test sets. Under this setting, the model was required to classify requirements originating from previously unseen projects and application domains rather than requirements sharing the vocabulary, writing style, and recurring terminology of projects observed during training.

Compared with conventional sentence-level partitioning, this project-level strategy presents a substantially more challenging learning scenario. Software projects frequently contain domain-specific terminology, naming conventions, and requirement-writing patterns that may unintentionally appear in both the training and evaluation sets when individual requirements are randomly partitioned. Consequently, successful performance under strict project-level evaluation provides stronger evidence that the model has learned transferable semantic representations rather than project-specific lexical patterns.

The supplementary sentence-level 10-fold cross-validation experiments further highlight the influence of the evaluation protocol. Although the model architecture, training procedure, optimization strategy, dataset, and hyperparameter configuration remained unchanged, the Macro-F1 score increased from approximately 0.80 under the project-level protocol to approximately 0.97 under sentence-level cross-validation. Since the data partitioning strategy was the only experimental factor that changed, these findings indicate that reported performance can vary substantially depending on the adopted data partitioning strategy.

This observation is consistent with the findings of Kamal and Islam [24], who investigated the robustness and cross-dataset generalization of recent requirement classification models. It also aligns with broader evidence showing that inappropriate data partitioning and information leakage can produce overly optimistic estimates of model performance in machine learning studies [27]. Similarly, the results of the present study show that even when the dataset, model architecture, training procedure, and hyperparameter configuration remained unchanged, changing only the evaluation protocol from sentence-level to strict project-level evaluation produced a considerable reduction in classification performance. Together, these findings emphasize that realistic evaluation settings are essential for assessing the true generalization capability of requirements classification models.

The comparison with previous studies should therefore be interpreted in light of both the adopted evaluation protocol and the underlying learning formulation rather than reported performance values alone.

Ayed and Alrammal [10] demonstrated that conventional fine-tuning of BERT achieves high overall performance on both the PROMISE and PURE datasets. However, the reported Macro-F1 scores remain noticeably lower than the corresponding Weighted-F1 values, indicating weaker performance on minority classes. By comparison, the proposed framework combines domain-specialized representations, FR context, and Focal Loss to achieve more balanced classification performance across all NFR categories.

Fazelnia et al. [11] reformulated NFR classification as a Natural Language Inference (NLI) task and reported a Macro-F1 score of 0.83 under sentence-level cross-validation. Under the same evaluation protocol, the proposed framework achieved approximately 0.97 Macro-F1 while preserving the original multi-class classification formulation. This suggests that competitive performance can be achieved without reformulating the task as an inference problem when domain-specialized representations and FR context are effectively incorporated.

Other representative studies, including PRCBERT [18], the hybrid BERT architecture proposed by Kaur and Kaur [23], and the ensemble deep learning framework of Rahimi et al. [16], improve NFR classification through prompt learning, architectural hybridization, or model ensembles. In contrast, the proposed framework demonstrates that substantial performance gains can also be achieved by improving representation quality and incorporating FR context without substantially increasing architectural complexity.

Overall, these findings demonstrate that evaluation methodology should be regarded as an integral component of NFR classification benchmarks, as changing only the evaluation protocol increased Macro-F1 by nearly 17 percentage points. Therefore, comparisons between studies should always be interpreted in the context of the adopted evaluation protocol rather than reported performance metrics alone.

5.4. Practical Implications for Requirements Engineering

The findings of this study provide several practical implications for the design, evaluation, and deployment of automated NFR classification systems.

First, the results demonstrate that the evaluation protocol should be selected according to the intended application. When the objective is to assess real-world deployment capability, strict project-level evaluation provides a more realistic estimate of model generalization because the model is required to classify requirements originating from previously unseen software projects. Conversely, sentence-level cross-validation remains valuable for benchmarking against existing studies, provided that the resulting performance is interpreted within the context of the adopted evaluation protocol.

Second, the experiments indicate that improving representation quality can be more beneficial than increasing architectural complexity. The proposed Re-Distill encoder consistently outperformed generic pretrained transformers while preserving the same underlying MPNet backbone. This suggests that domain specialization through representation learning can provide greater benefits than simply adopting increasingly complex model architectures.

Third, the input ablation experiments demonstrate that the usefulness of FR context depends on the target quality attribute. In particular, Security and Usability requirements benefited substantially from the additional contextual information, whereas Availability requirements were often classified successfully from the NFR statement alone. Consequently, future context-aware NFR classification systems may benefit from adapting the amount of contextual information according to the characteristics of individual NFR categories rather than treating all categories identically.

Fourth, the present findings emphasize that performance metrics should always be interpreted together with the corresponding evaluation protocol. Direct numerical comparisons between studies employing different datasets or evaluation strategies may lead to misleading conclusions regarding model capability. Reporting both project-level and sentence-level evaluation results provides a more comprehensive assessment of model performance and enables fairer comparison across future studies.

6. Threats to Validity

Although the proposed framework was evaluated using a strict project-level protocol designed to better reflect real-world deployment scenarios, several limitations should be acknowledged. First, the experiments were conducted using publicly available requirements datasets constructed from the PROMISE repository and selected projects from the PURE repository. Although these datasets cover multiple software domains, they are still considerably smaller than large-scale industrial requirements repositories. Additional validation using larger industrial datasets collected from different organizations would further strengthen the external validity of the proposed framework.

Second, this study focuses on four major NFR categories, namely Performance, Security, Availability, and Usability. While these categories represent some of the most widely studied software quality attributes in the requirements engineering literature, other important categories, such as Maintainability, Reliability, Portability, and Scalability, were not included. Future work may therefore investigate whether the observed benefits of domain-specialized representations and FR context generalize to a broader range of NFR categories.

Finally, although the adopted project-level evaluation protocol substantially reduces project leakage and provides a more realistic estimate of model generalization than conventional sentence-level partitioning, the proposed framework has been evaluated on a single benchmark derived from the PROMISE and PURE repositories. Evaluating the framework on additional requirement repositories and industrial datasets would further strengthen confidence in its robustness and practical applicability.

7. Conclusions and Future Work

This paper presented a context-aware framework for NFR classification that integrates FR context with the domain-specialized semantic representations learned by the Stage-3 Re-Distill encoder. The proposed framework was evaluated under a strict project-level protocol designed to assess generalization to previously unseen software projects.

The results demonstrate that domain-specialized representations substantially improve NFR classification performance and that the contribution of FR context is category-dependent rather than universal. While Security and Usability benefit particularly from contextual information, Availability can often be classified accurately from the NFR statement alone. In addition, the comparison between project-level evaluation and sentence-level cross-validation highlights the significant influence of the evaluation protocol on reported performance.

Overall, these findings demonstrate that FR context should be viewed as a category-dependent source of semantic information rather than a universally beneficial input, providing practical guidance for the design of future context-aware NFR classification systems.

Future work will investigate the integration of recent Large Language Models (LLMs) into context-aware NFR classification, extend the framework to additional NFR categories, and evaluate its effectiveness on larger industrial software requirements datasets to further assess its scalability and generalization.