Beyond the Black Box: Interpretable Multi-Trait Essay Scoring with Trait-Aware Transformer

Abstract

The rapid advancement of automated essay scoring (AES) has been constrained by a representation bottleneck, where monolithic models collapse diverse facets of writing constructs into a single, uninterpretable signal, undermining the pedagogical value of multi-dimensional rating traits. To address this limitation, the RoBERTa-based Trait-Aware Transformer (RoBERTa-TAT) is introduced. This architectural reframing replaces unified pooling with parallel, trait-specific attention streams, preserving and disentangling critical features such as conceptual depth and mechanical precision. Tested on the ASAP Dataset-7, RoBERTa-TAT attains a new state-of-the-art Quadratic Weighted Kappa (QWK) of 0.936, outperforming sequential baselines and conventional Transformer variants. Beyond gains in accuracy, this trait-specialized architecture recasts scoring from a black-box prediction into a transparent diagnostic tool, enabling actionable, fine-grained feedback at different rating traits. High-resolution inspection reveals that the model’s internal representations correlate with specific linguistic markers—such as discourse connectives for organization—suggesting a degree of structural alignment with expert judgment. By aligning high-capacity representation learning with the granular demands of formative assessment, RoBERTa-TAT provides a practical, interpretable blueprint for deploying accountable AI in education and broadening access to expert diagnostic insight.

1. Introduction

Writing is a foundational pillar of academic engagement and advanced cognitive progress, playing a crucial role in the integration of intricate ideas and the effective communication of knowledge [1,2,3]. In the era of large-scale standardized testing and rapidly expanding online education, the necessity for prompt, consistent, and unbiased assessment of student writing has surged dramatically. Traditional manual grading, however, remains a significant bottleneck; it is not only labor-intensive and costly but also vulnerable to human constraints such as rater fatigue, subjective biases, and the halo effect, where a general impression unduly affects the evaluation of specific characteristics [4,5,6,7,8,9].

Since Ellis Page pioneered Project Essay Grade (PEG) in the 1960s [10,11], the landscape of automated essay scoring (AES) has undergone a profound transformation, marked by a transition from simple surface feature statistics to complex neural network modeling [4,12,13,14,15,16]. Initial investigations focused on manually extracted linguistic features, such as word counts, average word length, sentence length, or frequency of spelling errors [10,17,18,19]. These features, while somewhat reflective of writing conventions, were often criticized for measuring proxy indicators of writing rather than its intrinsic quality. With the advent of enhanced computational capabilities and the proliferation of extensive pretrained language models [20,21,22,23,24,25,26], AES models now exhibit a remarkable alignment with human evaluators, often matching or exceeding human rater reliability on certain standardized tests.

Despite achieving robust performance benchmarks, contemporary AES models have struggled to gain traction among educators, largely due to an enduring reliance on holistic scoring [27,28,29]. Although administratively streamlined, such assessments lack the diagnostic granularity required to inform teaching. Therefore, students cannot understand their specific performance in areas like coherence, lexical sophistication, or syntactic sophistication. Such single holistic scoring lacks the diagnostic depth essential for formative assessment, failing to deliver the granular feedback essential for self-regulated learning and formative growth [30,31]. In the context of this study, an analytic trait is operationally defined as a specific, pedagogically distinct dimension of writing proficiency—such as Ideas, Organization, Style, or Conventions—that serves as an independent criterion within a formative assessment rubric. Disaggregating writing into these traits allows for granular evaluation rather than a single generalized score.

Multi-task learning, a principal mechanism for parameter sharing in NLP, has not remedied this deficit in AES because of task interference: a shared encoder asked to reconcile low-level grammatical checks with high-level semantic judgments tends to latch onto easily measured cues (for example, spelling or length) while under-representing nuanced semantic features. Compounding the problem, contemporary AES models remain diagnostically uninterpretable; their black-box outputs offer no transparent rationale, preventing the interpretable feedback necessary for effective pedagogical intervention [32,33,34].

To overcome these representational constraints, the RoBERTa-based Trait-Aware Transformer (RoBERTa-TAT) is proposed. Rather than relying on undifferentiated pooling, RoBERTa-TAT deploys a suite of parallel, autonomous attention modules, each dedicated to a specific analytic scoring trait. These modules independently evaluate token salience for their assigned trait. By embedding this task-specific specialization inside the model’s internal attention mechanisms rather than tacking it onto the output layer, RoBERTa-TAT yields trait-aware internal representations while preserving parameter efficiency.

The contributions of this study are three-fold:

• Architectural specialization to alleviate representation bottlenecks: The RoBERTa-TAT framework is presented, significantly advancing AES by replacing conventional unified pooling with parallel, trait-specific attention mechanisms. By allocating dedicated attention streams to each rating trait, feature interference is reduced, and distinct linguistic signals are preserved within a single end-to-end architecture.
• Empirical evidence of trait-aligned cognition: Through high-resolution attention visualizations, it is demonstrated that RoBERTa-TAT autonomously develops differentiated focus patterns that align closely with established pedagogical rubrics. The model prioritizes discourse markers and structural cues when assessing Organization and foregrounds thematic and topical keywords when evaluating Ideas, exhibiting strong concordance with expert rater criteria and indicating a meaningful alignment with pedagogical judgment.
• Paradigm shift toward transparent diagnostic assessment: By bridging high-capacity Transformer architectures with the granular requirements of multi-dimensional feedback, this study catalyzes a shift from black-box predictive scoring to transparent, actionable diagnosis. The proposed framework establishes a scalable blueprint for interpretable AI in education, democratizing access to expert-level diagnostic insights and providing a robust foundation for formative, teacher-in-the-loop assessment tools. This conceptual transition is empirically demonstrated in Figure 1, which visualizes the multi-trait attention maps for a representative response (Essay 938). For Conventions (score: 2), the attention highlights predominantly align with orthographic errors—such as the misuse of all ways instead of always—and missing punctuation. These visualizations surface the specific linguistic lapses that justify the model’s scoring. Conversely, for Ideas (score: 0), the sparsity of the highlighted regions visually substantiates the model’s determination that the response lacks meaningful thematic development. By transforming internal neural weights into visible diagnostic evidence, the framework provides actionable, dimension-specific feedback rather than mere numerical outputs.

Figure 1. Multi-dimensional diagnostic interpretability via trait-specific attention. Heatmap visualization for Essay ID 938 (scores: 0, 1, 1, 2). The visualization illustrates the attention weights learned by the RoBERTa-TAT architecture across four distinct analytical traits. Higher color intensity (darker highlighting) identifies critical linguistic features prioritized by the model for each specific trait. By segregating attention across Ideas, Organization, Style, and Conventions, the model provides a transparent evidentiary basis for its diagnostic scoring.

2. Related Work

2.1. Assessment Paradigms: Holistic vs. Analytic Scoring

Writing assessment is often framed by two contrasting paradigms—holistic and analytic—each presenting distinct trade-offs in reliability, validity and instructional utility [31,35,36].

Holistic scoring depends on a gestalt appraisal: a rater assigns a single, global mark that reflects overall proficiency. This broad perspective privileges higher-order abilities such as reasoning and creativity, can bolster learner confidence and, with systematic rater training, yield acceptable consistency [37]. It is also operationally efficient, which suits large-scale testing. Its principal weakness, however, is diagnostic opacity: one aggregate score conceals which subskills underpin performance and so offers limited guidance for targeted remediation [38]. Analytic scoring, by contrast, disaggregates writing into constituent dimensions (for example, organization, content and style) and allocates marks for each [31]. That granularity produces actionable, formative feedback, reveals uneven profiles of strength and weakness and supports fine-grained achievement judgments [39]. The approach is, however, more resource-intensive to design and apply.

Assessment design should therefore be driven by purpose: holistic instruments are economical and appropriate when the aim is to evaluate global competence and integrative skills, whereas analytic rubrics are essential when precise diagnosis and structured instructional growth are the objective [31]. Accordingly, the present study adopts a holistic and analytic scoring framework—chosen for its capacity to generate diagnostic, component-level profiles that directly inform targeted pedagogical interventions and allow empirical investigation of how sub-scales contribute to overall writing proficiency.

2.2. Approaches to AES: From Rule-Based Systems to Large Language Models

Early systems, epitomized by Ellis Page’s PEG [10], inferred essay quality from readily measurable textual proxies—word counts, sentence length and other surface statistics. Although insensitive to semantics, these cues correlated with human judgments by reflecting stylistic regularities of competent prose. Subsequent commercial and research systems such as E-rater [4] and IntelliMetric [40] broadened the feature set to discourse markers, error detection and lexical indices, and treated scoring as a supervised statistical problem, routinely using linear regression, naïve Bayes and support vector machines. While these methods produced dependable proxies for holistic ratings, they required intensive feature engineering and remained vulnerable to superficial gaming: writers could inflate scores by optimizing countable signals without improving substantive argumentation.

The rise of neural architectures shifted the field away from hand-crafted predictors toward learned, hierarchical representations. Convolutional networks captured local patterns [24], recurrent models—especially Long Short-Term Memory (LSTM [14]) and its bidirectional variants—traced longer-range coherence and thematic progression, and early attention mechanisms began to allow models to weight salient passages much as a reader would. These approaches reduced reliance on linguistic features and improved robustness across prompts and genres.

The introduction of the Transformer’s parallel self-attention recalibrated the understanding of context and scale. Large pretrained encoders—BERT, RoBERTa and XLNet [12,13,23,26,41]—when fine-tuned for scoring, produced substantial improvements on benchmark corpora. Subsequent work adapted these backbones through pairwise-ranking objectives, multi-task fine-tuning and hybrid architectures that fuse contextual encoders with graph representations, better capturing structures and inter-sentence relations and narrowing the gap with human raters.

Large language models such as GPT-4 and LLaMA have broadened the AES landscape. Their generative and inferential capacities enable zero- and few-shot scoring as well as richer, discourse-level feedback [22,42]. Careful prompt engineering, including context-aware few-shot exemplars and chain-of-thought elicitation, can improve alignment with human judgments. Furthermore, recent research highlights how AI-driven NLP frameworks can identify and disentangle domain-specific linguistic patterns through structured representation learning [43]. Integrating this concept with Transformer studies reinforces the rationale for the proposed trait-specific attention mechanisms. Despite these advances, most state-of-the-art systems remain oriented toward holistic scoring, offering limited capacity for multi-dimensional assessment or actionable diagnostic feedback.

2.3. The Interpretability Gap in Existing AES Models

The migration of AES from transparent, rule-based models to black-box deep learning models has created an interpretability gap that constrains both classroom adoption and high-stakes deployment. Contemporary AES models are typically optimized to maximize agreement with human raters, yet a solitary numeric grade without accompanying justification has limited pedagogical value and fails to engender confidence among teachers and learners.

Integrating deep neural architectures into AES has therefore posed a trade-off between predictive robustness and interpretability: black-box models deliver strong agreement but little transparency required for educational trust. Recent advances in interpretable artificial intelligence, however, are beginning to reconcile high-capacity architectures with human-aligned evaluative logic. Insights have been advanced by feature-attribution methods such as SHapley Additive exPlanations (SHAP [44]), which connect broad sets of linguistic indices to discrete rubric traits [34]. Similarly, approaches like QwenScore+ that combine rubric-aware chain-of-thought prompting with reinforcement learning from human feedback produce automated feedback that is more transparent, pedagogically coherent and higher in fidelity [45].

The discipline is shifting from mere score approximation toward diagnostically useful, transparent evaluation. Rationale-aware multi-trait models and mixture-of-experts architectures offer a viable route to reconcile holistic efficiency with analytic depth, yielding scores that are both reliable and interpretable against rubric descriptors. By reducing task interference and leveraging the contextual strengths of Transformer models, such models can become constructive partners in instruction—delivering scalable, objective and pedagogically relevant feedback.

To address these gaps mentioned above, the RoBERTa-TAT framework is introduced, which decomposes model judgments into pedagogically informed analytic traits and yields interpretable, fine-grained scores with targeted, evidence-based feedback.

3. Materials and Methods

3.1. Dataset

For model training and validation, this research leverages the ASAP Dataset-7 (Kaggle, San Francisco, CA, USA, available online: https://www.kaggle.com/competitions/asap-aes/data, accessed on 1 March 2026), which contains 1730 essays sourced from a 7th-grade writing prompt. ASAP Dataset-7 was specifically selected because it is the unique prompt within the entire ASAP corpus that provides comprehensive, multi-dimensional analytic trait scores (Ideas, Organization, Style, and Conventions) annotated by experts, alongside a holistic score. This granular labeling is a strict prerequisite for validating the trait-aware architecture.

To evaluate the annotation quality and establish a human performance baseline, the inter-rater agreement between the two expert raters was calculated based on their raw, individual scores (see Table 1). The rater–rater Quadratic Weighted Kappa (QWK) is 0.721 for the holistic score. Notably, the adjacent agreement (scores within a single-point margin) across all analytic traits is exceptionally high, ranging from 0.975 to 0.997. This near-perfect adjacent agreement indicates that major rating divergences are exceedingly rare, robustly substantiating the consistency of the rubric application.

Table 1. Inter-rater agreement between two human expert raters on ASAP Dataset-7.

Rating Traits	Rater–Rater QWK	Exact Agreement	Adjacent Agreement (±1)
Ideas	0.695	0.625	0.975
Organization	0.576	0.611	0.984
Style	0.544	0.653	0.997
Conventions	0.567	0.618	0.990
Holistic	0.721	0.292	0.607

To ensure rating reliability, each essay was evaluated by two human experts, and the sum of these scores was adopted for analysis. The descriptive statistics in Table 2 reflect the raw scores assigned by individual raters (0–3 per analytic trait). In this scoring framework, higher scores indicate superior writing quality. A score of 0 represents an ungradable response or a severe deficiency in the evaluated construct, whereas a score of 3 denotes mastery and sophisticated application of the specific rubric criteria. Intermediate scores (1 and 2) reflect incremental, rubric-defined levels of partial proficiency. Holistic scores (aggregate of the four traits) exhibit a mean of 8.03 with a standard deviation of 2.29, reflecting considerable variability across essays (Figure 2). Holistic scores approximate a quasi-normal distribution with a peak between 8 and 9 and spanning the full 0–12 range. The pronounced right skew at the highest score indicates a caliber of exceptional essays necessitating further nuanced differentiation.

Table 2. Descriptive statistics of individual rater essay scores for ASAP Dataset-7.

Rating Traits	Mean	SD	Min	Max
Holistic	8.03	2.29	1	12
Ideas	1.84	0.80	0	3
Organization	2.03	0.64	0	3
Style	2.00	0.55	0	3
Conventions	2.17	0.61	0	3

Figure 2. Distribution of individual rater holistic scores for ASAP Dataset-7.

By contrast, the individual trait scores show more constrained distributions (means between 1.84 and 2.17 with smaller standard deviations), indicative of consistent rubric application and coherent alignment in the assessment of different constructs of writing quality (Figure 3). Organization and Conventions display relatively balanced distributions, primarily concentrated around a score of 2. Conversely, Ideas and Style manifest greater variability, suggesting that while students commonly excel in fundamental structural and mechanical aspects, the attainment of superior conceptual depth and stylistic expression remains pivotal in distinguishing writing excellence. These properties make ASAP Dataset-7 a particularly well-suited benchmark for research on multi-dimensional, trait-oriented AES.

Figure 3. Distribution of individual rater trait scores (Ideas, Organization, Style, and Conventions) for ASAP Dataset-7.

The inter-trait correlation matrix presented in Figure 4 reveals the intricate relationships between holistic and analytic scores. All analytic traits correlate strongly with the holistic score (r ranging from $0.82$ to $0.93$), with Organization showing the greatest convergence (r = 0.93), indicating that structural coherence is the dominant signal in this corpus. By contrast, the substantially weaker association between Ideas and Conventions ($r=0.53$) signals a partial decoupling of high-level cognitive content from low-level linguistic mechanics. This heterogeneous correlation structure highlights the limits of monolithic scoring schemes and motivates the RoBERTa-TAT architecture, which is designed to disentangle and subsequently integrate these distinct rating traits.

Figure 4. Correlation matrix of holistic and analytic scores for ASAP Dataset-7.

3.2. RoBERTa-TAT Model Architecture

The RoBERTa-based architecture was specifically selected due to its robust dynamic masking pretraining, which empirically provides a highly stable and efficient baseline for AES tasks compared to standard BERT. Furthermore, it is emphasized that the proposed trait-aware attention mechanism is inherently architecture-agnostic. These parallel, trait-specific attention streams can be seamlessly generalized and integrated into other Transformer-based encoders, such as DeBERTa, without requiring fundamental structural overhauls.

RoBERTa-TAT introduces a trait-specific attention pooling layer that departs from conventional global pooling. Rather than producing a single summary vector, RoBERTa-TAT deploys parallel, learnable attention modules that each generate a dedicated representation for an individual trait (see Figure 5). By enabling modules to attend selectively to trait-relevant signals, the layer preserves fine-grained information often lost in standard pooling schemes. These intermediate, trait-wise representations are optimized prior to final scoring, yielding gains in both predictive accuracy and interpretability. The architecture remains parameter-efficient, imposing only marginal overhead on the underlying Transformer. Beyond improved metrics, RoBERTa-TAT reveals interpretable patterns that align with human scoring criteria, enhancing transparency and the diagnostic value of multi-dimensional writing assessment. When applied to the ASAP Dataset-7, RoBERTa-TAT not only meets but frequently surpasses the performance of existing models, offering enhanced clarity and understanding.

Figure 5. Architecture of the RoBERTa-TAT model.

Core encoder and tokenization: A RoBERTa-based backbone (12 layers, 768-dimensional hidden states) is adopted as a shared encoder. Input essays are tokenized and passed through the Transformer to yield contextualized hidden states; the output of the final layer is denoted as $H_{final}\in {\mathbb{R}}^{B\times L\times H}$, where B, L, and H represent batch size, sequence length and hidden dimension. This shared representation provides the high-dimensional linguistic substrate from which trait-specific signals are extracted.

Trait-specific attention pooling: In contrast to standard models that rely on a single [CLS] token or global mean pooling, RoBERTa-TAT employs five parallel, learnable attention modules, one per rubric: Holistic, Ideas, Organization, Style and Conventions. For each trait i, its specific pooled representation $P_i\in {\mathbb{R}}^{B\times H}$ is obtained by: (1) Computing attention logits: $a_i=Linea{r}_i(tanh\left(H_{final}\right))$, where $Linea{r}_i$ maps ${\mathbb{R}}^H$ to ${\mathbb{R}}^1$. (2) Normalizing across the token dimension: ${\alpha }_i=\mathrm{softmax}(a_i,\dim =1)$ where softmax is applied across the sequence length dimension (L). ${\alpha }_i\in {\mathbb{R}}^{B\times L\times 1}$ represents the learned importance of each token for score i. (3) Forming a weighted sum $P_i=\sum ({\alpha }_i\ast H_{final})$. By maintaining independent parameters for each $Linea{r}_i$, the model can attend to distinct textual cues—e.g., the Conventions head emphasizing syntactic and punctuation markers, while the Ideas head highlights semantic anchors and narrative content.

Prediction heads and efficiency: Finally, five independent prediction heads, each consisting of a single linear layer ($Linear\_pred\_i$: ${\mathbb{R}}^H\to {\mathbb{R}}^1$), map the corresponding trait-specific pooled representation $P_i$ to the final predicted score for that dimension. The entire architecture—the pretrained RoBERTa encoder together with the trait attention and prediction layers—is trained end-to-end. The additional attention and prediction parameters are lightweight, constituting a negligible fraction of the total model size (<0.1%), preserving parameter efficiency while enabling trait-resolved representations and improved interpretability.

3.3. Baseline Models

To unravel the distinct benefits of the proposed model from traditional pooling methods, a structured comparison was performed against three principled baselines, constituting a hierarchical ablation that validates each design choice:

• RoBERTa-Pool: Using the identical RoBERTa-based encoder, this baseline utilizes a shared encoder to extract a single pooled representation via global mean pooling. This single vector is then passed through five separate, parallel linear regression heads to simultaneously predict the holistic and the four trait scores. It tests the null hypothesis that all tokens contribute equally to downstream judgments, thereby quantifying the benefit of replacing static spatial averaging with a learned, dynamic aggregation.
• RoBERTa-[CLS]: This baseline probes the representational power of the dedicated [CLS] token—commonly employed for classification—to capture multifaceted aspects of writing quality. The comparison examines whether explicit, trait-directed attention better consolidates distributed information than the implicit aggregation embodied by the [CLS] vector, which is subsequently passed through five parallel regression heads identically to the Pool baseline.
• RoBERTa-SharedAttention: Architecturally identical to RoBERTa-TAT except that a single attention module is shared across all rating traits, this model serves as a diagnostic control. It clarifies whether essay scoring requires a unified notion of token importance or, as hypothesized, distinct trait-specific attention maps to reach optimal precision.

Together, this hierarchy of models forms a comprehensive suite of ablation studies, systematically disentangling the layers of TAT. It provides empirical evidence indicating that the enhanced performance is directly attributable to the multi-dimensional attention strategy rather than simply increasing model capacity or leveraging backbone strength.

3.4. Evaluation Metrics

To assess the correspondence between RoBERTa-TAT predictions and gold-standard human evaluations, model performance is examined through three evaluation metrics:

Quadratic Weighted Kappa (QWK): Unlike traditional unweighted measures, QWK takes into account the severity of discrepancies between predicted and human-assigned scores, applying a quadratic penalty to outliers. The metric is defined as:

$$\kappa =1-{\displaystyle \frac{{\sum }_{i,j}{w}_{ij}{O}_{ij}}{{\sum }_{i,j}{w}_{ij}{E}_{ij}}}$$

(1)

where $w_{ij}$ denotes the weight matrix ${(i-j)}^2/{(N-1)}^2$, and $O_{ij}$ and $E_{ij}$ represent the observed and expected matrices of scores, respectively.

Exact and adjacent agreement (EA and AA): EA evaluates the frequency of perfect matches between machine-generated and human scores, acting as a stringent measure of the model’s discriminative accuracy. AA broadens the scope by considering predictions within a single-point margin of the true score.

3.5. Experimental Setup

A 5-fold cross-validation strategy was adopted, utilizing a 60%/20%/20% split for training, validation, and testing sets. All experiments were conducted over 5 independent runs with fixed random seeds. The optimal model checkpoint for testing was rigorously selected based on the highest holistic QWK score achieved on the validation set. Statistical significance of the performance gains over baselines was confirmed using paired t-tests ($p<0.05$).

All models were trained with a composite, unweighted loss defined as the sum of mean-squared errors (MSE) between predicted and reference scores for the five rating traits:

$$Los{s}_{total}=\sum \left[MSE(pre{d}_i,tru{e}_i)\right]\mathrm{for}i\in \left\{\mathrm{Holistic},\mathrm{Ideas},\mathrm{Org},\mathrm{Style},\mathrm{Conv}\right\}$$

(2)

While essay scores are inherently ordinal, unweighted MSE is widely adopted in regression-based AES to provide stable, continuous gradient updates across multiple prediction heads. It is acknowledged that exploring ordinal-aware alternatives remains a highly promising direction for future research. During evaluation, continuous regression predictions were mathematically rounded to the nearest integer prior to computing the discrete QWK metric.

All experiments used a RoBERTa-based initialization and were optimized end-to-end with AdamW (peak learning rate $2\times {10}^{-5}$, see Table 3). A linear warm-up was applied over the first 10% of training steps, and models were trained for 10 epochs with a batch size of 16 and a maximum input length of 512 tokens, thereby preserving the full essay context while ensuring stable convergence.

Table 3. Hyperparameters used for training RoBERTa-TAT and baseline models.

Hyperparameters	Value
Base Model	RoBERTa-base
Optimizer	AdamW
Peak Learning Rate	$2\times {10}^{-5}$
Batch Size	16
Max Sequence Length	512
Warm-up Ratio	0.1
Number of Epochs	10

4. Results

4.1. Comparisons with RoBERTa-Based Baselines

RoBERTa-TAT and the ablation suite were evaluated on the ASAP Dataset-7, aggregating performance across holistic and analytic ratings (see Table 4). RoBERTa-TAT consistently surpassed all RoBERTa-based baselines, establishing a new benchmark for this prompt: a holistic QWK of 0.936 versus 0.909 for RoBERTa-SharedAttention and 0.887 for RoBERTa-[CLS]. The marked increase in agreement with human raters indicates that the trait-aware architecture effectively relieves the representational bottleneck inherent to unified scoring models, yielding more faithful alignment with expert judgment.

Table 4. Comparative performance of RoBERTa-TAT and ablation baselines on ASAP Dataset-7. Reported metrics are QWK (Quadratic Weighted Kappa), EA (exact agreement) and AA (adjacent agreement).

	Holistic			Ideas			Organization			Style			Conventions
Models	QWK	EA	AA	QWK	EA	AA	QWK	EA	AA	QWK	EA	AA	QWK	EA	AA
RoBERTa-Pool	0.885	0.277	0.643	0.882	0.666	0.946	0.841	0.697	0.955	0.852	0.736	0.990	0.816	0.691	0.959
RoBERTa-[CLS]	0.887	0.290	0.650	0.891	0.675	0.959	0.842	0.678	0.965	0.859	0.755	0.987	0.835	0.713	0.965
RoBERTa-SharedAttention	0.909	0.309	0.723	0.908	0.726	0.965	0.885	0.755	0.975	0.902	0.822	0.994	0.863	0.771	0.965
RoBERTa-TAT	0.936	0.309	0.774	0.929	0.732	0.987	0.919	0.793	0.987	0.921	0.847	0.997	0.888	0.787	0.978

The performance improvements of RoBERTa-TAT extend across analytic rating traits. For Ideas and Organization, RoBERTa-TAT attained QWK values of 0.929 and 0.919, respectively, reflecting a refined sensitivity to conceptual depth and discourse structure. For Style and Conventions the model demonstrated exceptional consistency: AA reached 0.997 for Style and 0.978 for Conventions, indicating that predictions almost invariably fell within a one-point margin of human judgments.

Comparisons with the ablation baselines underline the importance of trait-specific mechanisms. Although RoBERTa-SharedAttention exhibited reasonable performance, it failed to match RoBERTa-TAT’s exact agreement levels—most notably in Organization (0.793 versus 0.755) and Style (0.847 versus 0.822)—providing evidence that disentangled attention maps improve alignment with rubric criteria. The RoBERTa-Pool baseline performed least well across metrics (e.g., Holistic QWK = 0.885), implying that simple pooling discards the fine-grained information required for nuanced scoring.

A detailed inspection of the confusion matrices in Figure 6 reveals the model’s consistent performance across different score ranges, further validating its diagnostic reliability. In the mid-range scores (e.g., scores 3–5 for analytic traits), where the training distribution is most dense, RoBERTa-TAT exhibits exceptional precision with very high EA. For instance, in the Style trait, the model correctly identifies 137 samples at score 4. Regarding boundary scores (e.g., 0–1 or 6), while data scarcity in these long-tail regions typically poses a challenge, the model maintains high robustness. The errors at these boundaries are primarily adjacent (off by one), with very few instances of drastic misclassification. For example, in the Ideas trait, true scores of 6 are almost exclusively predicted as either 5 or 6, demonstrating that even when the model misses the exact boundary, its diagnostic output remains within a pedagogically acceptable margin. This stability at the extremes mitigates concerns regarding score-range bias and reinforces the model’s fairness for both high- and low-achieving students.

Figure 6. Confusion matrices for RoBERTa-TAT predictions vs. true scores on ASAP Dataset-7 for analytic traits. Note: The axes range from 0 to 6 because they represent the sum of the two independent raters’ scores.

4.2. Ablation Study: Dissecting the Contributions of Trait Awareness

To isolate the specific mechanisms driving the performance gains, a comprehensive ablation study was conducted, progressively stripping the specialized attention layers from the architecture. As visualized in Figure 7, the transition from static pooling to specialized attention layers yields incremental yet significant gains in evaluative accuracy across all rating traits.

Figure 7. Performance comparison of RoBERTa-TAT and baseline models across dimensions. Here, TAT denotes RoBERTa-TAT, SharedAtt denotes RoBERTa-SharedAttention, [CLS] denotes RoBERTa-[CLS], and Pool denotes RoBERTa-Pool.

A fundamental divergence in performance emerges when comparing learned pooling mechanisms against traditional heuristic-based methods. The RoBERTa-Pool baseline, relying on simple mean pooling, consistently serves as the lower performance bound, yielding a Holistic QWK of only 0.885. In contrast, architectures utilizing learned task-relevant representations—such as RoBERTa-[CLS] and RoBERTa-SharedAttention—demonstrate superior alignment with human expert consensus. For Organization, RoBERTa-SharedAttention achieves a QWK of 0.885, substantially eclipsing the 0.841 recorded by the mean-pooling variant. These results suggest that the high-dimensional semantic features necessary for assessing essay quality are non-uniformly distributed, requiring learned mechanisms to prioritize salient textual segments over mere structural averages.

The definitive validation of the core hypothesis is found in the performance leap between RoBERTa-SharedAttention and RoBERTa-TAT. While shared attention provides modest gains over [CLS]-based pooling, it cannot match the discriminative resolution of trait-specific attention. As detailed in Table 4, the introduction of independent, trait-specific attention parameters leads to a marked improvement in QWK scores, specifically gaining +0.034 for Organization and +0.027 for Holistic scores over the shared attention baseline. This gap reflects a pedagogical reality: the linguistic indicators of Ideas (conceptual richness) are materially different from those of Conventions (grammatical and surface correctness). Permitting the model to develop distinct attentional foci for each trait relieves the representational bottleneck inherent to unified models.

Empirical corroboration appears in the structure of the error patterns: pronounced diagonal clustering in the confusion matrices and very high adjacent agreement (Style = 0.997; Organization = 0.987) demonstrate that specialized attention is the primary factor enabling the model to emulate expert raters’ nuanced judgments. Together, these results position RoBERTa-TAT as a robust, scalable architecture for high-stakes automated pedagogical assessment.

4.3. Performance Benchmarking Against Established Models

To contextualize the efficacy, the architecture was compared against state-of-the-art AES systems at the holistic scale (see Table 5). Early seminal work, exemplified by LSTM-CNN-ATT [14], combined long short-term memory and convolutional modules with an attention layer to capture both local semantic patterns and broader contextual dependencies. Subsequent studies shifted focus to pure convolutional architectures [24], comparing a traditional AES approach using support vector machines paired with Coh-Metrix linguistic features against a contemporary convolutional neural network model to evaluate their scoring performance relative to human raters. The arrival of large pretrained language models prompted new strategies: BERT+SST+DAT [12] enriched representations by integrating semantically aware signals with domain adaptation techniques, while R²BERT [26] introduced a novel architecture that optimizes scoring precision through a dynamic weighting of MSE and batch-wise ListNet losses. Contemporary frontiers now emphasize multi-scale feature synthesis and knowledge-informed aggregation; notably, DualBERT-Trans-CNN [13] utilized convolutional transition networks for cross-granularity fusion, and KWM-B [23] applied a hierarchical weighting mechanism from token to document to distill the latent semantic core.

Table 5. Comparison with well-established AES models on ASAP Dataset-7.

Models	QWK
Classical Deep Architectures
CNNs [24]	0.740
LSTM-CNN-ATT [14]	0.747
Transformer-Based Encoders
BERT+SST+DAT [12]	0.840
R2BERT [26]	0.839
DualBERT-Trans-CNN [13]	0.832
KWM-B [23]	0.834
Contemporary Generative LLMs
GPT-3.5 [46]	0.727
GPT-4 [47]	0.680–0.750
RoBERTa-TAT (Proposed)	0.936

The proposed RoBERTa-TAT model yields a clear and consistent advantage. Classic deep architectures such as LSTM-CNN-ATT (QWK = 0.747, Ref. [14]) and CNNs (QWK = 0.740, Ref. [24]) fall well short of the Transformer-based approach, underscoring the limitations of recurrent and convolutional kernels for modeling long-range semantic dependencies in high-stakes essays. Against contemporary Transformer-enhanced models, RoBERTa-TAT remains superior, improving upon BERT+SST+DAT (QWK = 0.840, Ref. [12]) and KWM-B (QWK = 0.834, Ref. [23]) by 9.6% and 10.2%, respectively, (p < 0.001 for both). The robustness of RoBERTa-TAT across benchmarks suggests that explicit trait awareness confers more than a marginal uplift. It advances the fidelity of multi-dimensional AES and better approximates the nuanced judgments of human raters.

Given the rapid evolution of Transformer architectures, the emergence of contemporary generative LLMs, such as LLaMA variants and other state-of-the-art (SOTA) generative architectures, must be addressed. Although powerful zero- and few-shot generation capabilities are exhibited by these models, lower performance on strict ordinal tasks is frequently observed when they are compared to supervised, encoder-based systems. For instance, a peak Quadratic Weighted Kappa (QWK) of 0.727 was reported by Xiao et al. [46] following the resource-intensive fine-tuning of GPT-3.5. Furthermore, few-shot QWKs for GPT-4 were observed by Kim and Jo [47] at 0.750 (Ideas), 0.680 (Organization), 0.733 (Style), and 0.679 (Conventions), which consistently trail the performance of task-specific fine-tuned encoders. Because generative LLMs are inherently optimized for next-token probability rather than ordinal regression, score volatility and poorer alignment with human raters across multi-dimensional rubrics are produced when they are directly applied without extensive parameter fine-tuning. Consequently, the comparison scope in this study was restricted to supervised, encoder-based models to ensure a fair evaluation of parameter-efficient, discriminative architectures.

4.4. Interpretability and Attribution Analysis

To gain insights into the decision-making process of the proposed model and explore how it evaluates different rating traits, the token-level relevance for each specific trait was analyzed. While attention heatmaps are helpful for building intuition about the model’s inner workings, relying solely on the qualitative visualization of attention weights is insufficient to claim that the model directly mirrors rubric-driven expert judgment. Consequently, an independent, gradient-based attribution method was introduced as a lightweight check to quantitatively validate these qualitative observations.

4.4.1. Qualitative Visualization via Attention Weights

First, the learned attention weights from the trait-specific attention pooling layers were visualized. In these heatmaps (see Appendix A for representative examples), darker highlighting indicates higher attention weights assigned by the corresponding trait head. A visual inspection confirms the model successfully attends to distinct, trait-relevant textual cues:

Ideas: Attention is distributed across key content words and thematic elements that reflect the core narrative (e.g., patience, calm, reward in Essay 10 (Appendix A.1); Hunting, hobbies, wood in Essay 115 (Appendix A.3)).

Organization: The attention mechanism for Organization tends to highlight discourse markers, transitional words, and conjunctions that guide the structural flow of ideas. For instance, in Essay 10, structural connectors like when, also, For, and But are emphasized. In Essay 115, the model focuses on sequence and transition words such as When, After, But, and This.

Style: The attention for Style frequently attends to adverbs, intensifiers, and descriptive vocabulary that contribute to the tone and syntactic variety of the essay. In Essay 10, words like extremely, kinda, and really receive high attention. In Essay 115, descriptive terms like favorite and entertaining are highlighted.

Conventions: The attention weights for this trait consistently focus on grammatical errors, misspellings, and punctuation issues. For example, in Essay 10, the model isolates misspelled or non-standard words such as wateing and inpatient, as well as words missing apostrophes, such as dont. Similarly, in Essay 115, heavy attention is paid to typos like boreing and definetly, and in Essay 328, the misspelling doller is correctly highlighted.

Moving from unified representations to trait-specific attention shifts scoring from an uninterpretable black box to interpretable, actionable feedback. Unlike holistic baselines (e.g., RoBERTa-Pool, which yielded the weakest metrics), RoBERTa-TAT returns independent trait scores alongside attention heatmaps, enabling targeted guidance. By learning parameters independently for each trait, the model reduces cross-trait bias and offers a transparent and practical foundation for AI-assisted, trait-based feedback in educational settings.

4.4.2. Validation via Gradient-Based Attribution

To provide a more robust validation, an independent check was conducted using a gradient-based attribution method. Integrated gradients [48] were computed using the Captum library. This approach estimates each token’s contribution to the predicted score by accumulating gradients along an interpolation path—a mechanism fundamentally independent of the learned attention pooling weights.

This analysis was applied across four representative essays, and the resulting gradient attributions (Appendix B) were compared with the trait-specific attention weights from the qualitative analysis. The comparison reveals strong alignment between the two methods: both approaches consistently target topical nouns for Ideas, transitional words for Organization, adverbs for Style, and localized morphological anomalies for Conventions. This convergence provides independent evidence that the feature attributions of RoBERTa-TAT are grounded in robust, trait-relevant linguistic patterns consistent with human evaluation criteria.

For Ideas, both methods consistently distribute weight across key content words, topical nouns, and thematic elements that construct the core narrative or central argument of the text (e.g., patience and calm in Essay 10 (Appendix B.1); Hunting and hobbies in Essay 115 (Appendix B.3)).

For Organization, both methods reliably highlight discourse markers and transitional words (e.g., But, When, After).

For Style, both focus heavily on vocabulary choices, specifically targeting adverbs and intensifiers (e.g., extremely, really).

For Conventions, both precisely pinpoint localized morphological anomalies, misspellings, and punctuation errors (e.g., wateing, boreing, dont).

5. Discussion

5.1. Pedagogical Alignment and the Mitigation of Cognitive Bias

A principal contribution of the proposed architecture is its capacity to recast AES from a black-box predictive engine into an interpretable diagnostic tool. As demonstrated by the qualitative attention heatmaps and the corroborating gradient-based attributions in Section 4.4, the decision-making process of the model is deeply grounded in rubric-relevant linguistic patterns.

By learning parameters independently for each trait, the architecture successfully mitigates cross-trait bias. In manual holistic grading, this cognitive interference is widely recognized as the halo effect, where an evaluator’s judgment of one dimension disproportionately biases the evaluation of another [49]. The demonstrable alignment between the internal representations of the model and established pedagogical rubrics strongly supports the growing imperative for Explainable Artificial Intelligence (XAI) in writing assessment [34,45,50,51]. Providing actionable, rubric-aligned rationales rather than solitary predictive scores is critical for establishing algorithmic fairness and fostering trust among educators.

5.2. Operational Efficiency and Green AI in Assessment

Beyond predictive improvements, RoBERTa-TAT delivers practical advantages for educational deployment. Automating multi-dimensional grading substantially reduces labor, cost, and rater fatigue. The trait-specific attention modules and independent heads add negligible overhead—under 0.1% of the base RoBERTa parameter count—illustrating strong parameter efficiency consistent with recent PEFT paradigms [52]. This economy permits rapid processing on conventional institutional hardware and markedly reduces the computational, memory, and carbon costs associated with very large foundation models, aligning with Green AI principles [53,54].

5.3. Limitations and the Horizon of Generative Models

A primary limitation of the current study is the absence of direct empirical comparisons with recent SOTA large language models, such as LLaMA variants or other contemporary generative architectures. The decision to restrict the comparative scope to conventional Transformer-based encoders is justified by several methodological and practical considerations. First, the core objective of this research was focused on resolving the representation bottleneck through parameter-efficient, trait-specific attention mechanisms rather than through the massive scaling of model parameters. Second, the full-parameter fine-tuning of multi-billion-parameter SOTA models for multi-dimensional ordinal regression requires prohibitive computational resources. Such resource requirements are in conflict with Green AI principles [53,54], which are prioritized in this study to ensure that the model can be deployed efficiently in resource-constrained educational settings.

It has been demonstrated by recent studies that remarkable zero-shot and few-shot scoring capabilities are offered by generative LLMs, with rich natural-language feedback often being generated alongside scores [55,56]. However, empirical comparisons between discriminative architectures and generative LLMs are rendered methodologically complex, as fundamentally different paradigms are operated upon. Significant volatility in automated essay scoring tasks is still exhibited by LLMs, with performance and reliability fluctuating considerably across different test prompts and student demographics [56]. Furthermore, generative models are inherently probabilistic, and non-deterministic outputs, commonly referred to as hallucinations, are prone to be generated [57]. In high-stakes educational testing environments, critical reliability and fairness risks are posed by the generation of plausible but factually unsupported text or unfaithful feedback. In contrast, parameter efficiency, deterministic reliability, and exact token-level interpretability are prioritized by the approach presented herein. Future research should be directed toward exploring hybrid frameworks where the robust, rubric-aligned feature attribution of trait-aware encoders is leveraged to guide and constrain the natural language generation capabilities of modern LLMs, thereby uniting predictive stability with generative richness.

6. Conclusions

This research introduces RoBERTa-TAT, a novel architecture that significantly advances multi-dimensional AES by overcoming the representation bottleneck inherent in unified models. Rather than relying on a single shared representation, RoBERTa-TAT deploys parallel, trait-specific attention streams—tailored to Ideas, Organization, Style and Conventions—that disentangle distinct linguistic signals and preserve fine-grained diagnostic information. Experiments on the ASAP Dataset-7 demonstrate that RoBERTa-TAT establishes a new state-of-the-art benchmark with a holistic QWK of 0.936, significantly outperforming both established sequential models and strong Transformer-based baselines. Crucially, the architecture yields attention maps that align with rubric-relevant cues, converting previously uninterpretable predictions into interpretable evidence and reducing cross-trait interference. By returning independent, actionable trait scores, RoBERTa-TAT supports targeted, pedagogically meaningful feedback, narrowing the gap between high-stakes scoring accuracy and advancing more transparent AI-assisted writing assessment.

Appendix A.1

Trait-specific attention visualization for Essay ID 10 (scores: Ideas = 1, Organization = 2, Style = 3, Conventions = 4). The heatmap displays attention weights learned by RoBERTa-TAT for each rating trait, with darker highlighting indicating higher attention.

Appendix A.2

Trait-specific attention visualization for Essay ID 93 (scores: Ideas = 1, Organization = 2, Style = 3, Conventions = 3).

Appendix A.3

Trait-specific attention visualization for Essay ID 115 (scores: Ideas = 0, Organization = 4, Style = 4, Conventions = 6).

Appendix A.4

Trait-specific attention visualization for Essay ID 328 (scores: Ideas = 2, Organization = 2, Style = 1, Conventions = 2).

Appendix B.1

Trait-specific gradient attribution visualization for Essay ID 10. Darker highlighting indicates a greater gradient-based attribution score, signifying that the token exerts a stronger influence on the model’s predicted score for the corresponding trait.

Appendix B.2

Trait-specific gradient attribution visualization for Essay ID 93.

Appendix B.3

Trait-specific gradient attribution visualization for Essay ID 115.

Appendix B.4

Trait-specific gradient attribution visualization for Essay ID 328.