Enhancing Cross-Prompt Essay Trait Scoring via External Knowledge Similarity Transfer

Abstract

Cross-prompt automated essay scoring presents a significant challenge due to substantial differences in samples across prompts, and recent research has concentrated on evaluating distinct essay traits beyond the overall score. The primary approaches aim to enhance the effectiveness of AES in cross-prompt scenarios by improving shared representation or facilitating the transfer of common knowledge between source and target prompts. However, the existing studies only concentrate on the transfer of shared features within essay representation, neglecting the importance of external knowledge, and measuring the degree of commonality across samples remains challenging. Indeed, higher similarity of external knowledge also results in a better shared representation of the essay. Based on this motivation, in this paper, we introduce an extra-essay knowledge similarity transfer to assess sample commonality. Additionally, there is insufficient focus on the intrinsic meaning of the traits being evaluated and their varied impact on the model. Therefore, we incorporate extra-essay knowledge representation to enhance understanding of the essay under evaluation and the target of the task. Experimental results demonstrate that our approach outperforms other baseline models on the ASAP++ datasets, confirming the effectiveness of our method.

1. Introduction

Automated essay scoring (AES) aims to automatically assess an essay based on a provided prompt that presents the writer with the topic and often includes specific guidelines or instructions. The AES system has the potential to generate objective and equitable results swiftly and efficiently, addressing the issue of resource consumption and strong subjectivity in manual evaluation [1,2,3]. Initial studies in AES focused on providing a holistic evaluation by assigning an overall score to the essay [4,5,6,7,8] and gradually shifted to trait scoring from different perspectives [9,10] as the functionality expanded and achieved the objective of delivering multidimensional analysis to improve reliability. Currently, the research focus has changed from prompt-specific [6] to more challenging cross-prompt scenarios [11,12,13,14,15,16], as depicted in Figure 1.

Cross-prompt AES involves the evaluation of previously unseen target prompts using a training model from the source prompt. In this paper, we focus on the cross-prompt essay trait scoring (CPETS) where both the overall score and detailed trait scores in the target domain are provided, as illustrated in Figure 1b. Existing CPETS methods improve effectiveness by improving shared representation or facilitating the transfer of common knowledge between source and target prompts through content-related features [17,18] or external knowledge [1,19]. Ridley et al. [17] proposed a model for CPETS, which utilizes part-of-speech (POS) embeddings as input for a neural AES model and concatenates features such as readability over essay representation. Sun et al. [18] fused syntactic embedding with POS embedding to extend the essay representation, leveraging common linguistic knowledge to enhance the essay representation of the target domain. Do et al. [19] enhanced essay representation through incorporating prompt information and alleviated data insufficiency through extracted topic coherence features. They also endeavored to improve the model by quantifying relations between traits. While the studies above improved essay-shared representation by integrating various features, they devoted less attention to knowledge beyond the essay content, such as genre features and the intrinsic meanings of traits. Moreover, they did not focus on quantifying knowledge transfer across prompts. Knowledge transfer provides methodological support for cross-domain research, and knowledge flows across domains through techniques such as transfer learning and domain adaptation [20]. Chen and Li [1] concentrated on the challenge of knowledge transfer across distinct prompts, arguing that similar prompts can gain more shared knowledge.

As shown in

Table 1. Examples of prompts and partial essay texts.

Prompt ID (Genre): Prompt Text
Prompt 1 (argumentative): Write a letter to your local newspaper in which you state your opinion on the effects computers have on people. Persuade the readers to agree with you.
Prompt 2 (argumentative): Write a persuasive essay to a newspaper reflecting your vies on censorship in libraries.
Prompt 3 (narrative): Tell a true story in which laughter was one element or part.
Prompt 1 example: Censorship is found on television, in libraries, in schools, and even in homes. Do you think that censorship should be used? Censorship can be used to block one’s personal thoughts and opinions to promote conformism. I disagree with the idea of Censorship. Not everyone thinks people should be able to have their own images and visions; however, that is what makes our world so individualized.
Prompt 2 example: Final exams, omnipresent in schools, are divisive. Do they truly benefit students? I argue against their overemphasis. They can overshadow holistic learning, promoting stress over understanding. Yet, it’s this very pressure that ignites the drive for mastery, shaping individual academic journeys.
Prompt 3 example: Laughter is a really good thing to have in any relationship you are in whether its a friend ship, a love relation ship, or work relation ship. Either way you need laughter so you don’t get so up tight that you are no fun to be around. Every now and then some body around you in your life needs some laughter to cheer them up.

, Prompt 1 and Prompt 2 were both argumentative essays written to require a discussion of a situation related to the academics of the students, and Prompt 3 was a narrative essay. As can be seen in the essays for Prompt 1 and Prompt 2, both essays are organized as question-and-answer writing with a high degree of vocabulary similarity. However, there were fewer commonalities between Prompt 3 and Prompts 1 and 2 at the organizational level of the essays, as well as greater differences in vocabulary use and language expression. It can be seen that essays with similar prompts or similar genres are more likely to contain similar information, even though the different essays come from different writers. Overall, the rational use of external information can enhance the ability of the model to extract shared features between source and target prompts and significantly improve the effectiveness of the model in scoring cross-prompt essays, further illustrating the effectiveness of solving the problem from the perspective of external information. Despite the study’s success, its focus was limited to the dissimilarities between prompts. The researchers did not explore the transferability of other dimensions of external knowledge between the source and target prompts. In addition, the effective measurement of the degree of commonality between knowledge and the efficient transfer of knowledge are crucial.

To resolve the aforementioned problems, we present cross-prompt trait scoring based on extra-essay knowledge transfer (ExtCTS). For the first problem, inadequate attention to knowledge beyond the content of the essay, we introduce extra-essay knowledge representation. This enhancement augments the understanding in the model of both the prompt under evaluation and the intrinsic meaning of the assessed trait by integrating the essay prompt and the trait description. Description refers to the explanation of a specific essay trait in a natural language text. For instance, a trait description may illustrate the evaluation of the trait “narrativity”, defined as “a measure of the coherence and cohesion of the response to the prompt”. Genre description, prompt description, and trait description details are provided in [21]. To tackle the problem of insufficient sources for knowledge transfer and the complexity of assessing commonality, we propose extra-essay similarity transfer, which involves evaluating the similarity of different external knowledge in source and target domains, setting similarity factors to determine transferability, and applying it to essay representation. The main contributions of our work are summarized as follows:

• We propose extra-essay knowledge representation, which consists of an essay prompt encoder and a trait description encoder to integrate knowledge beyond the content of the essay.
• We introduce extra-essay similarity transfer, which quantifies the similarity between various prompts by considering knowledge similarity factors beyond essay content, thereby enhancing essay representation.
• Experimental results demonstrate that our approach achieved state-of-the-art performance, validating the effectiveness of the proposed methodology.

The paper is organized as follows. In Section 2, we first present the literature review. We describe our method and experiments in Section 3 and Section 4. Finally, in Section 5, we provide a conclusion, future work, and the limitations of the paper.

2. Related Work

In this section, we briefly review relevant studies concerning cross-prompt holistic scoring and cross-prompt essay trait scoring.

2.1. Cross-Prompt Holistic Scoring

Presently, AES is mostly performed as supervised learning, demanding substantial amounts of labeled data. However, the process of labeling data is time-consuming and labor-intensive, resulting in a shortage of training data for identical prompts. Phandi et al. [11] initiated research on cross-prompt holistic scoring to address the challenges. Phandi et al. [11] and Dong and Zhang [22] test the impacts of traditional AES and neural AES in cross-prompt settings on the ASAP [available online: https://www.kaggle.com/c/asap-aes (accessed on 1 May 2025)] dataset, respectively, establishing the foundational framework for subsequent research.

Subsequent research has focused on refining frameworks [23,24,25] and improving methods of cross-prompt knowledge transfer [26,27,28], utilizing multi-task learning [29], or transfer learning [30,31] in natural language processing. To tackle the challenge of extracting highly transferable scoring knowledge from multiple prompt sources, Li et al. [28] integrated prompt information and extracted diverse knowledge essential to the target domain through the fusion of data from all source prompts. Cao et al. [26] proposed a domain-adaptive framework to address the issue of neglecting domain differences in prior studies, optimize the utilization of data under various prompts, and enhance the domain-adaptive capabilities of models. Jiang et al. [27] developed disentangled representation learning to tackle the challenge of cross-domain generalization in essay representation. They further devised a prompt-aware neural AES model to extract the combined representation of prompt-invariant and prompt-specific features. These studies have enhanced the representation and transfer of knowledge across prompts. However, they have not explored knowledge transfer across traits.

2.2. Cross-Prompt Essay Trait Scoring

Cross-prompt essay trait scoring that evaluates not only the essay as a whole but also the individual trait was first proposed by Ridley et al. [12]. They proposed a CTS model that evaluates multiple attributes using a multi-task learning framework. The model utilizes POS embeddings as input and concatenates features like readability in essay representation. Furthermore, they improved representation by incorporating inter-trait relationships. Sun et al. [18] fused syntactic features and contrastive learning to enhance model performance by augmenting trait representation and improving model stability. Although both studies introduced linguistic knowledge, such as lexicon and syntax, at the input layer, they fell short in enhancing the utilization of external knowledge of essay across prompts. Do et al. [19] employed prompt information and topic coherence features to enhance essay comprehension of evaluation topics. They also established a loss function for trait relations by utilizing the correlation between different traits to improve the representation of the essay. However, they did not focus on quantifying knowledge transfer across prompts. Chen and Li [1] concentrated on the challenge of knowledge transfer across distinct prompts, arguing that the similarity in essay representations reflects the extent of transferable features. Consequently, they proposed that prompt-mapping captures the consistent representation of the target and source prompts. Zhang et al. [16] proposed a PDA framework, which effectively aligns the features of source and target prompts by using two alignment strategies, such as prompt-level alignment and category-level alignment, and further improves the performance of the model through classifier consistency constraint. Nevertheless, they only focus on the transfer of shared features within essay representation, neglecting the transfer of external knowledge. We believe that evaluating the degree of commonality of different dimensions of knowledge between source and target prompt samples and quantifying transferable knowledge are crucial directions for addressing cross-prompt trait essay scoring.

3. Methodology

In this paper, we focus on cross-prompt essay trait scoring. When given a dataset of n essay prompts, one is selected from the dataset as the target prompt ${\mathcal{P}}_T$ and the remaining prompts are used as the source prompt set ${\mathcal{P}}_E=\left\{{\mathcal{P}}_1,{\mathcal{P}}_2,\dots ,{\mathcal{P}}_i,\dots ,{\mathcal{P}}_{n-1}\right\}$; the goal of cross-prompt traits scoring is to train a model using the source prompt ${\mathcal{P}}_E$ and test it on the target prompt ${\mathcal{P}}_T$. The ith prompt ${\mathcal{P}}_i$ contains $N_i$ examples of essays with annotations ${\left\{x_{ij},{\mathcal{Y}}_{ij}\right\}}_{j=1}^{N_i}$, where $x_{ij}$ contains the text of the jth essay $e_{ij}$ and the external information of the jth essay ${\mathcal{K}}_{ij}$, ${\mathcal{Y}}_{ij}=\left\{y_{ij}^1,y_{ij}^2,\dots ,y_{ij}^K\right\}$ is the set of scores to different traits for the jth essay, and K is the number of traits to be scored. The external information ${\mathcal{K}}_{ij}$ consists of the prompt text $p_{ij}$ for the ith prompt, the genre text $g_{ij}$ for the ith prompt, the trait $t{r}_{ij}$ to be evaluated for the ith prompt, and its corresponding trait description $t{d}_{ij}$, etc.

Existing cross-prompt essay trait scoring studies have focused on improving shared representations or facilitating the transfer of common knowledge between source and target prompts to improve the effectiveness of AES in cross-prompt scenarios. In this paper, we introduce the transfer of external essay knowledge similarity to assess sample commonality, which to some extent reflects symmetry considerations because when external knowledge has higher similarity, the shared representation of essays is also better, which implies that there is a potential symmetrical correlation between essays with different prompts in terms of external knowledge dimensions, reflecting the dynamic balancing mechanism in the process of knowledge transfer, and through which the grasp of essay commonality can be enhanced. Meanwhile, previous studies have paid insufficient attention to the intrinsic meanings of the assessed features and their varying impacts on the model. In contrast, this paper introduces additional essay knowledge representation, which enables the model to have a more in-depth understanding of the essays to be assessed and the task objectives. This approach also helps to achieve a more balanced and comprehension of the features across essays with different prompts, thereby establishing a kind of symmetry in feature understanding.

The proposed ExtCTS model is shown in Figure 2. Figure 3 is the sub-module of ExtCTS. It contains three components: extra-essay knowledge representation (EKR), multilevel extra-essay knowledge similarity transfer (EST), and a trait-specific encoder. EKR represents external knowledge to enhance the model to understand the prompt under evaluation and the trait being assessed. EST quantifies the degree of commonality among various external knowledge between the source and target domains. The factor from EST is then applied to the trait-specific encoder, enhancing the ability of the model to transfer knowledge from diverse prompts to obtain superior essay representation.

3.1. Extra-Essay Knowledge Representation

In order to enhance the ability of the model to extract shared features of the target prompts through external information, this section incorporates the trait description information under the framework of multi-trait scoring with fused prompt information. The following describes the EKR used in this paper to extract the external information representation, including the trait description encoder (TDE) and the essay prompt encoder (EPE).

3.1.1. Trait Description Encoder

Trait description is a piece of natural language that details and describes an evaluation trait. The TDE enhances the understanding of evaluation traits by fusing trait description information. For the traits $\left\{t{r}_1,t{r}_2,\dots ,t{r}_i,\dots ,t{r}_K\right\}$ included in the training set, K is the number of traits, $t{r}_i$ is the ith target trait, and the trait descriptions are $\left\{t{d}_1,t{d}_2,\dots ,t{d}_i,\dots ,t{d}_K\right\}$. The ith trait description $t{d}_i$ has the word sequence $\left\{t{w}_{i1},t{w}_{i2},\dots ,t{w}_{in}\right\}$, n is the number of words, and obtains the corresponding word embedding vector as $\left\{{\mathbf{tx}}_{i1},{\mathbf{tx}}_{i2},\dots ,{\mathbf{tx}}_{in}\right\}$. All trait descriptions are sequentially input to the word encoder (WE), and the model structure is shown in Figure 3a. The word encoder captures sentence-level representations of the input sequence using a CNN that incorporates an attention mechanism. Then the sentence-level trait description representation $\mathbf{ts}=\left\{{\mathbf{ts}}_1,{\mathbf{ts}}_2,\dots ,{\mathbf{ts}}_K\right\}$ is obtained by computing the Equations (1)–(4), and the formulas are shown below:

$$\begin{array}{cc}\hfill {\mathbf{c}}_{ik}& =f\left({\mathbf{W}}_c·\left[{\mathbf{tx}}_{ik}:{\mathbf{tx}}_{ik+h_w-1}\right]+{\mathbf{b}}_c\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathbf{z}}_{ik}& =tanh\left({\mathbf{W}}_z·{\mathbf{c}}_{ik}+{\mathbf{b}}_z\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\alpha }_{ik}& ={\displaystyle \frac{exp\left({\mathbf{w}}_{\alpha }·{\mathbf{z}}_{ik}\right)}{\sum exp\left({\mathbf{w}}_{\alpha }·{\mathbf{z}}_{ij}\right)}},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathbf{ts}}_i& =\sum {\alpha }_{ik}{\mathbf{c}}_{ik},\hfill \end{array}$$

(4)

where ${\mathbf{W}}_c$ and ${\mathbf{b}}_c$ are the CNN trainable weight matrix and bias, respectively, $h_w$ is the window size of the CNN, ${\mathbf{W}}_c$ is used to capture the features of all the possible consecutive $h_w$ words in the sentence, $f\left(·\right)$ is the nonlinear activation function, and ${\mathbf{W}}_z$, ${\mathbf{w}}_{\alpha }$, and ${\mathbf{b}}_z$ are weight matrices, trainable weight vectors, and bias for the attention mechanism. The sentence-level trait description representation $\mathbf{ts}$ is computed from the contextual features ${\mathbf{c}}_i$ of the CNN output. Attention vectors ${\mathbf{z}}_i$ and attention weights ${\alpha }_i$ are computed by Equation (2) and Equation (3), respectively.

Next, the inter-trait relations are further captured through the multi-head self-attention layer and LSTM with an attention mechanism, thus obtaining the final trait description representation. The sentence-level trait description representation $\mathbf{ts}$ is input to the multi-head self-attention layer and the formulas are shown below:

$$\begin{array}{cc}\hfill MSA(\mathbf{q},\mathbf{k},\mathbf{v})& =\left({\mathbf{H}}_1\oplus \dots \oplus {\mathbf{H}}_i\oplus \dots \oplus {\mathbf{H}}_{n^{\prime }}\right){\mathbf{W}}_o,\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathbf{H}}_i& =Attention\left(\mathbf{q}{{\mathbf{W}}_q}_i,\mathbf{k}{{\mathbf{W}}_k}_i,\mathbf{v}{{\mathbf{W}}_v}_i\right),\hfill \end{array}$$

(6)

where ${{\mathbf{W}}_q}_i$, ${{\mathbf{W}}_k}_i$, ${{\mathbf{W}}_v}_i$ are the parameter matrices of the ith head $\mathbf{q}=\mathbf{k}=\mathbf{v}=\mathbf{ts}$, Attention is the scaled dot product attention, ⊕ is concatenation, ${\mathbf{H}}_i$ is the ith head used in the multi-head self-attention layer, $n^{\prime }$ is the number of heads in the multi-head self-attention layer, and ${\mathbf{W}}_o$ is the parameter matrix of the multi-head self-attention layer. Then, an LSTM is used to obtain the trait description representation $\mathbf{d}=\left\{{\mathbf{d}}_1,{\mathbf{d}}_2,\dots ,{\mathbf{d}}_K\right\}$, which is computed as shown in Equation (7):

$${\mathbf{d}}_u=LSTM\left({\mathbf{o}}_{u-1},{\mathbf{o}}_u\right),$$

(7)

where $\mathbf{o}$ is the output of the multi-head self-attention layer and ${\mathbf{d}}_u$ denotes the hidden state vector at the uth time.

3.1.2. Essay Prompt Encoder

In order to enhance the understanding of the model on the evaluation prompt and obtain more consistent evaluation results with the prompt information, this section uses a Word+POS Encoder that fuses word sequences and POS sequences to represent the essay prompt information, and the model structure is shown in Figure 3b. For the training set containing essay prompt texts $\left\{p_1,p_2,\dots ,p_i,\dots ,p_J\right\}$, J is the total number of prompts, the ith essay prompt text $p_i$ has the word sequences $\left\{p{w}_{i1},p{w}_{i2},\dots ,p{w}_{im}\right\}$, and obtains the corresponding word embeddings $\left\{{\mathbf{px}}_{i1},{\mathbf{px}}_{i2},\dots ,{\mathbf{px}}_{im}\right\}$ and the POS embeddings $\left\{{\mathbf{pp}}_{i1},{\mathbf{pp}}_{i2},\dots ,{\mathbf{pp}}_{im}\right\}$. First, the word embeddings and POS embeddings are summarized $\left\{{\mathbf{px}}_{i1}+{\mathbf{pp}}_{i1},{\mathbf{px}}_{i2}+{\mathbf{pp}}_{i2},\dots ,{\mathbf{px}}_{im}+{\mathbf{pp}}_{im}\right\}$. Second, the summed embedding vectors are sequentially input to the CNN with attention mechanism to capture the sentence-level prompt features using the Equations (1)–(4). Then, the sentence-level prompt features are input to the multi-head self-attention layer and LSTM computed using Equations (5)–(7) to obtain the essay prompt representation $\mathbf{h}=\left\{{\mathbf{h}}_1,{\mathbf{h}}_2,\dots ,{\mathbf{h}}_J\right\}$. Finally, the attention mechanism is computed for the essay prompt representation using Equations (8)–(10) to obtain the essay prompt contextual representation ${\mathbf{h}}^{\prime }$:

$$\begin{array}{cc}\hfill {\mathbf{z}}_i^{\prime }& =tanh\left({\mathbf{W}}_{z^{\prime }}·{\mathbf{h}}_i+{\mathbf{b}}_{z^{\prime }}\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\alpha }_i^{\prime }& ={\displaystyle \frac{exp\left({\mathbf{w}}_{{\alpha }^{\prime }}·{\mathbf{z}}_i^{\prime }\right)}{\sum exp\left({\mathbf{w}}_{{\alpha }^{\prime }}·{\mathbf{z}}_j^{\prime }\right)}},\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill {\mathbf{h}}^{\prime }& =\sum {\alpha }_i^{\prime }{\mathbf{h}}_i,\hfill \end{array}$$

(10)

where ${\mathbf{W}}_{z^{\prime }}$, ${\mathbf{w}}_{{\alpha }^{\prime }}$ and ${\mathbf{b}}_{z^{\prime }}$ are the weight matrices, trainable weight vectors, and bias of the attention mechanism. The essay prompt contextual representation ${\mathbf{h}}^{\prime }$ is computed from the LSTM output of the essay prompt representation $\mathbf{h}$. The attention vector ${\mathbf{z}}^{\prime }$ and the attention weight ${\alpha }^{\prime }$ are computed by Equation (8) and Equation (9), respectively.

3.2. Multi-Level Extra-Essay Knowledge Similarity Transfer

Essays with a high degree of external knowledge similarity are more likely to help the model learn better-quality shared features. Based on this motivation, we design multilevel similarity transfer to measure the degree of similarity between external information of the source prompts and the target prompt. First, the features of external knowledge are extracted using a pre-trained language model. Given a dataset of $N_p$ prompts, there is the set of external knowledge $\mathcal{K}=\left\{{\mathcal{K}}_{genre},{\mathcal{K}}_{prompt},{\mathcal{K}}_{trait}\right\}$, where ${\mathcal{K}}_{genre}={\left\{g_i\right\}}_{i=1}^{N_g}$ is the set of $N_g$ genre descriptions, ${\mathcal{K}}_{prompt}={\left\{p_i\right\}}_{i=1}^{N_p}$ is the set of $N_p$ essay prompts, ${\mathcal{K}}_{trait}={\left\{t{d}_i\right\}}_{i=1}^{N_t}$ is the set of $N_{td}$ trait descriptions. Given a source prompt training data $e^S$ with external knowledge $g^S$, $p^S$, and a target prompt with external knowledge $g^T$, $p^T$, the representation of the external knowledge of the essay is extracted using BERT with the following formula:

$$\mathbf{H}=BERT\left(\left[\mathtt{C}\mathtt{L}\mathtt{S}\right]\oplus c\oplus \left[\mathtt{S}\mathtt{E}\mathtt{P}\right]\right),$$

(11)

where c is the input of BERT, ⊕ is concatenation operation. Then the cosine similarity of the external knowledge between the source prompt ${\mathbf{H}}^S$ and the target prompt ${\mathbf{H}}^T$ is calculated to obtain the similarity transfer factor; the formula is as follows:

$$cosine\left({\mathbf{H}}^S,{\mathbf{H}}^T\right)={\displaystyle \frac{{\mathbf{H}}^S·{\mathbf{H}}^T}{\parallel {\mathbf{H}}^S\parallel \parallel {\mathbf{H}}^T\parallel }}.$$

(12)

In this paper, we consider two similarity factors, $f_{sim}=\left\{f_p,f_g\right\}$, where $f_p$ is the similarity factor for essay prompt and $f_g$ is the similarity factor for genre description. Finally, the features at different levels of trait-specific encoder are weighted using the $f_{sim}$ as shown in Figure 2, so as to enable the source prompt features extracted by the neural network to transfer directionally towards the target prompt and to enhance the ability of the model to generalize to the target prompt.

3.3. Trait-Specific Encoder

We employ a hierarchical structure [4] and basic framework from [19]. The model uses POS encoder to obtain sentence-level representation as shown in Figure 3c and trait-specific encoder for each trait to obtain document-level representation as illustrated in Figure 2. For a sentence containing n words $\left\{w_1,w_2,\dots ,w_n\right\}$, the POS embedding $\left\{{\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_n\right\}$ is first obtained, and then an attention layer is used to obtain the sentence-level representation $\mathbf{s}$. The formulas are as follows:

$${\mathbf{c}}_i=CNN\left(\left[{\mathbf{p}}_i:{\mathbf{p}}_{i+h_w-1}\right]\right),i=1,2,\dots ,n,$$

(13)

$$\mathbf{s}=Attention\left(\left[{\mathbf{c}}_1:{\mathbf{c}}_n\right]\right),$$

(14)

where $h_w$ is the kernel size of CNN.

Then, sentence-level essay representation is extracted using a text encoder as depicted in Figure 3d and further incorporate prompt representation. In addition, to enhance awareness of the target prompt, we calculate the document-level essay representation. The final representation of the model is ${\mathbf{t}}_i$, which is weighted by $f_{sim}$, and the score is obtained using the following equation:

$${\widehat{y}}_i=\sigma \left({{\mathbf{W}}_y}_i·\left(f_{sim}·{\mathbf{t}}_i\right)+{{\mathbf{b}}_y}_i\right),$$

(15)

where $\sigma$ is the linear layer of sigmoid activation, ${{\mathbf{W}}_y}_i$ and ${{\mathbf{b}}_y}_i$ are the trainable weight matrices and bias terms, and ${\widehat{y}}_i$ is the score predicted for the ith trait; ${\mathbf{t}}_i$ contains the output of trait-specific encoder and all external knowledge representations obtained from TDE and EPE.

3.4. Loss Function

Loss function includes score loss ${\mathcal{L}}_{score}$ and inter-trait similarity loss based on trait description ${\mathcal{L}}_{sim}$, where ${\mathcal{L}}_{score}$ uses the same strategy as [12].

$$\mathcal{L}=\eta ·{\mathcal{L}}_{score}+\left(1-\eta \right)·{\mathcal{L}}_{sim},$$

(16)

$${\mathcal{L}}_{score}={\displaystyle \frac{1}{N}}{\displaystyle \frac{1}{K}}\sum _{i=1}^N\sum _{j=1}^K{\left({\widehat{\mathbf{y}}}_{ij}-{\mathbf{y}}_{ij}\right)}^2,$$

(17)

$${\mathcal{L}}_{sim}={\displaystyle \frac{1}{C}}\sum _{i=2}^K\sum _{j=i+1}^{K}TDS\left({\widehat{\mathbf{y}}}_i,{\widehat{\mathbf{y}}}_j,{\mathbf{H}}_{td}^{\left(i\right)},{\mathbf{H}}_{td}^{\left(j\right)}\right),$$

(18)

$$TDS=\left\{\begin{array}{cc}1-cos\left({\widehat{\mathbf{y}}}_i,{\widehat{\mathbf{y}}}_j\right)\hfill & ,\mathrm{if}cos\left({\mathbf{H}}_{td}^{\left(i\right)},{\mathbf{H}}_{td}^{\left(j\right)}\right)\ge \delta \hfill \\ 0\hfill & ,\mathrm{otherwise}\hfill \end{array}\right.,$$

(19)

where $\eta$ is a manual hyperparameter; N is the number of essays, and K is the number of traits; cos is the cosine similarity, $\delta$ is the similarity threshold, and C is the number of times the similarity of the trait description is greater than the threshold; ${\widehat{\mathbf{y}}}_i=\left[{\widehat{\mathbf{y}}}_{1i},{\widehat{\mathbf{y}}}_{2i},\dots ,{\widehat{\mathbf{y}}}_{Ni}\right]$ is the prediction of the ith trait of N training data; ${\mathbf{H}}_{td}^{\left(i\right)}$ is the trait description for the ith trait.

4. Experiments

4.1. Datasets and Settings

We conducted experiments on the ASAP++ dataset [21], initially introduced by Ridley et al. [17] for CPETS, which was labeled with fine-grained trait scores on the ASAP [available online: https://www.kaggle.com/c/asap-aes (accessed on 1 May 2025)] dataset. The statistical results are presented in

Table 2. ASAP++ dataset statistics for cross-prompt scoring.

Prompt	No. Essay	Max Len	Avg Len	Traits	Genre	Range
1	1783	950	350	OA, Cont, Org, WC, SF, Conv	ARG	[2, 12]
2	1800	1173	350	OA, Cont, Org, WC, SF, Conv	ARG	[1, 6]
3	1726	415	150	OA, Cont, PA, Lang, Nar	RES	[0, 3]
4	1772	395	150	OA, Cont, PA, Lang, Nar	RES	[0, 3]
5	1805	474	150	OA, Cont, PA, Lang, Nar	RES	[0, 4]
6	1800	502	150	OA, Cont, PA, Lang, Nar	RES	[0, 4]
7	1569	732	250	OA, Cont, Org, Conv	NAR	[0, 30]
8	723	1146	650	OA, Cont, Org, WC, SF, Conv	NAR	[0, 60]

ARG: Argumentative; RES: Source-Dependent Response; NAR: Narrative; OA: Overall; Cont: Content; Org: Organization; WC: Word Choice; SF: Sentence Fluency; Conv: Conventions; PA: Prompt Adherence; Lang: Language; Nar: Narrativity.

. Compared with overall scoring, trait scoring mainly evaluates the quality of essays in terms of language quality (vocabulary, syntax, etc.), content quality (strength of argument), and thinking level (organization). Evaluating the quality of essays from multiple dimensions can reflect writing level more comprehensively and promote the comprehensive improvement of the writing ability of students.

Additionally, other resources like genre descriptions, prompt descriptions, and trait descriptions derived from expert annotations are detailed in [21]. Prompt descriptions are shown in the

Table 3. Description of the trait of the ASAP++ dataset [21].

Trait: description
Genre: Argumentative, Narrative
Content: The amount of relevant text present in the essay.
Organization: The way the essay is structured.
Word Choice: The choice and aptness of the vocabulary used in the essay.
Sentence Fluency: The quality of the sentences in the essay.
Conventions: Overall writing conventions to be followed, like spelling, punctuations, etc.
Style: How well the essay is written consists of two major parts, namely sentence fluency and word choice.
Voice: The quality and accuracy of voice.
Genre: Source-Dependent Response
Content: The amount of relevant text present in the essay.
Prompt Adherence: A measure of how the writer sticks to the question asked in the prompt.
Language: The quality of the grammar and spelling in the response.
Narrativity: A measure of the coherence and cohesion of the response to the prompt.

. Genre descriptions are shown in the

Table 4. Description of the genre of the ASAP++ dataset [21].

Genre: description
Argumentative: These are essays where the prompt is one in which the writer has to convince the reader about their stance for or against a topic (for example, free speech in public colleges).
Source-Dependent Response: These essays are responses to a source text, where the writer responds to a question about the text (for instance, describing the writer’s opinion about an incident that happened to him in the text).
Narrative: These are essays where the prompt requires us to describe or narrate a story.

. For evaluation, we employed quadratic weighted kappa (QWK) [available online: https://www.kaggle.com/c/asap-aes#evaluation. (accessed on 1 May 2025)], a widely used metric in existing AES research [2]. QWK assesses the consistency between the AES system results and the golden score and assigns a value between $\left[0,1\right]$. QWK can be calculated by Equations (20) and (21):

$${\mathbf{W}}_{i,j}={\displaystyle \frac{{(i-j)}^2}{{(N-1)}^2}},$$

(20)

$$\mathbf{QWK}=1-{\displaystyle \frac{{\sum }_{i,j}{\mathbf{W}}_{i,j}{\mathbf{O}}_{i,j}}{{\sum }_{i,j}{\mathbf{W}}_{i,j}{\mathbf{Z}}_{i,j}}},$$

(21)

where ${\mathbf{O}}_{i,j}$ denotes the number of essays that receive a rating i by the human rater and a rating j by the AES system. The expected rating matrix $\mathbf{Z}$ is histogram vectors of the golden rating and AES system rating and normalized so that the sum of its elements equals the sum of its elements in $\mathbf{O}$; N is the range of score for each essay.

We use the same data settings as Ridley et al. [12], who first proposed the CPETS task, with the data in the target domain coming from a prompt that does not appear in the training set and the other prompts serving as the source domain. We use the same data settings as Ridley et al. [12]. We employ the handcrafted features proposed by Do et al. [19], encompassing length-based, readability, text complexity, text variation, sentiment, and topic coherence features. We employ the handcrafted features proposed by Do et al. [19]. We utilize word 50-dimension embedding as input from [32]. We build POS 50-dimension embedding and train all models for 50 epochs with a batch size of 32. The output size of the CNN layer and the LSTM layer are both configured at 100. The window size for the CNN layer is set to 5, and we employ two heads for the multi-head self-attention. Each model is executed five times with the seeds $\{12,22,32,42,52\}$, following Ridley et al. [12], and the average QWK is reported. We utilize the RMSProp optimizer, setting the initial learning rate to 0.001 and the dropout rate to 0.5. $\delta$ is set to 0.7 and $\eta$ is set to 0.7. Our experiments are conducted using an NVIDIA GeForce GTX 1080 Ti GPU.

4.2. Comparisons

We compare our method with the existing CPETS models:

• Hi att [4]: This model was first applied to CPETS by Ridley et al. [12]. The model utilizes a hierarchical structure that uses a convolutional neural network with attention and a recurrent neural network with attention to model word-level and sentence-level representation.
• AES aug [33]: The model builds on [6] by simultaneously evaluating multiple traits through multi-task learning.
• PAES [17]: This model employs POS embedding as input, employing a hierarchical structure with distinct models dedicated to each trait.
• CTS-w/o att [12]: This model removes the trait-level attention layer compared to CTS.
• CTS [12]: This model is based on [17], which designs shared encoder modeling word-level representation, builds submodules for each trait, and enhances the representation by exploiting the relationships between different traits via trait-level attention.
• FEATS [34]: This model is based on the handcrafted features produced by Ridley et al. [17] and Uto et al. [35], with the addition of POS bigram, prompt adherence features, top-N words features, and pronoun features.
• PMAES [1]: This model improves representation by facilitating knowledge transfer through prompt-mapping, thus acquiring more shared knowledge from similar prompts.
• RDCTS [18]: This model integrates POS embedding and syntactic embedding as inputs, leveraging contrastive learning to address data insufficiency.
• PLAES [36]: This model captures shared knowledge between different source prompts through meta-learning and utilizes level-aware learning to improve the ability of model to distinguish between different levels of essay.
• ProTACT [19]: This model utilizes prompt information and prompt coherence features to enhance the understanding of essays within the evaluation prompt.

4.3. Main Results

This section describes the overall effectiveness of the model proposed in this paper on a cross-prompt multi-trait task. The experimental results are summarized in terms of prompts and traits, and the average results for each prompt are shown in

Table 5. Average results for different traits.

Model	Prompts								AVG QWK ↑
	1	2	3	4	5	6	7	8
Hi att †	0.315	0.478	0.317	0.478	0.375	0.357	0.205	0.265	0.349
AES aug †	0.330	0.518	0.299	0.477	0.341	0.399	0.162	0.200	0.341
PAES †	0.605	0.522	0.575	0.606	0.634	0.545	0.356	0.447	0.536
CTS w/o att	0.619	0.539	0.585	0.616	0.616	0.544	0.363	0.461	0.543
CTS	0.623	0.540	0.592	0.623	0.613	0.548	0.384	0.504	0.553
PMAES	0.656	0.553	0.598	0.606	0.626	0.572	0.386	0.530	0.566
RDCTS	0.651	0.553	0.608	0.623	0.651	0.580	0.375	0.529	0.571
PLAES	0.648	0.563	0.604	0.623	0.634	0.593	0.403	0.533	0.575
ProTACT	0.647	0.587	0.623	0.632	0.674	0.584	0.446	0.541	0.592
ExtCTS (${\mathrm{BERT}}_{base}$)	0.665	0.583	0.646	0.656	0.675	0.611	0.421	0.562	0.602
ExtCTS (${\mathrm{BERT}}_{large}$)	0.670	0.596	0.648	0.654	0.674	0.603	0.422	0.543	0.601

The best measures are in bold; ^†: results from [12]. The upperarrow indicates that a higher value is better.

, while the results for each trait are shown in

Table 6. Average results for different traits.

Model	Traits									AVG QWK ↑
	OA	Cont	Org	WC	SF	Conv	PA	Lang	Nar
Hi att †	0.453	0.348	0.243	0.416	0.428	0.244	0.309	0.293	0.379	0.346
AES aug †	0.402	0.342	0.256	0.402	0.432	0.239	0.331	0.313	0.377	0.344
PAES †	0.657	0.539	0.414	0.531	0.536	0.357	0.570	0.531	0.605	0.527
CTS w/o att	0.659	0.541	0.424	0.558	0.544	0.387	0.561	0.539	0.605	0.535
CTS	0.670	0.555	0.458	0.557	0.545	0.412	0.565	0.536	0.608	0.545
FEATS	0.682	0.592	0.478	0.459	0.452	0.439	0.617	0.556	0.637	0.546
PMAES	0.671	0.567	0.481	0.584	0.582	0.421	0.584	0.545	0.614	0.561
RDCTS	0.673	0.561	0.480	0.591	0.576	0.426	0.609	0.560	0.634	0.568
PLAES	0.673	0.574	0.491	0.579	0.580	0.447	0.601	0.554	0.631	0.570
ProTACT	0.674	0.596	0.518	0.599	0.585	0.450	0.619	0.596	0.639	0.586
ExtCTS (${\mathrm{BERT}}_{base}$)	0.681	0.612	0.521	0.617	0.604	0.451	0.635	0.607	0.654	0.598
ExtCTS (${\mathrm{BERT}}_{large}$)	0.683	0.608	0.511	0.619	0.610	0.458	0.633	0.606	0.651	0.598

OA: Overall; Cont: Content; Org: Organization; WC: Word Choice; SF: Sentence Fluency; Conv: Conventions; PA: Prompt Adherence; Lang: Language; Nar: Narrativity. The best measures are in bold; ^†: results from [12]. The upperarrow indicates that a higher value is better.

. The table contains two scenarios proposed in this paper, ExtCTS using BERT(base) and ExtCTS using BERT(large) to extract external information for calculating the similarity transfer factor. From Table 5 and Table 6, it can be seen that the ExtCTS model obtains the best results in terms of average QWK values for all prompts and traits. In particular, the ExtCTS model using BERT(base) to extract external information has average QWK values of $0.602$ and $0.598$ for all prompts and traits, respectively, which are 1% and 1.2% higher compared to the best baseline model. The ExtCTS model using BERT (large) to extract external information has average QWK values of $0.601$ and $0.598$ for all prompts and traits, respectively, which are 0.9% and 1.2% higher than the best baseline model.The above results show that ExtCTS is effective.

Comparing the results for each prompt, the two ExtCTS models were only slightly lower than the ProTACT model for Prompt 7, and the other results were superior to all baseline models. Compared with the best baseline model, ProTACT, the two ExtCTS gain about 2% improvement on prompts 1, 3, 4, and 6. Comparing the two different external information extraction schemes, ExtCTS using BERT(base) works better on Prompts 4, 5, 6, and 8, and ExtCTS using BERT(large) works better on Prompts 1 to 3. The above results show that the proposed method can enhance the effectiveness of each prompt set, and both external information feature extraction using BERT(base) and BERT(large) are effective.

Comparing the results for each trait, the best results for ExtCTS outperform the baselines on all traits, with only ExtCTS using BERT(large) being slightly lower than the best baseline on Organization. The two pre-trained models have similar results in the attribute dimension. Comparing ExtCTS with the best baseline, ExtCTS gains 2.5%, 2%, 1.6%, and 1.6% in Sentence Fluency, Word Choice, Content, and Prompt Adherence, respectively. The above results show that our method is able to obtain great enhancement in each trait. ExtCTS has more improvement in the traits of Sentence Fluency and Word Choice which are related to language expression, indicating that our method is able to better obtain the shared features from the source prompt that are related to language expression. ExtCTS has more enhancement in the trait of Content, and Prompt Adherence also gained good improvement, indicating the effectiveness of the fusion of external knowledge. However, the two traits of Organization and Conventions are not very sensitive to external knowledge. Organization focuses on internal text structure, such as paragraphing and levels of argumentation, which can be assessed directly through the analysis of contextual relationships in the model. Conventions relies on grammatical rules and standardized writing conventions and is less influenced by subject matter or domain knowledge.

4.4. Ablation Studies

In order to further validate the effectiveness of our method, ablation studies are conducted in this section, and the experimental results are shown in

Table 7. Ablation results.

NO.	Model	AVG QWK ↑
1	ExtCTS	0.602
2	w/o prompt representation $\mathbf{h}$	0.598 (−0.004)
3	w/o trait description representation $\mathbf{d}$	0.600 (−0.002)
4	w/o target prompt aware representation ${\mathbf{s}}^{″}$	0.595 (−0.007)
5	w/o all extra-essay knowledge representation	0.593 (−0.009)
6	w/o shallow level feature transfer	0.604 (+0.002)
7	w/o middle level feature transfer	0.597 (−0.005)
8	w/o deep level feature transfer	0.598 (−0.004)
9	w/o shallow and middle level feature transfer	0.598 (−0.003)
10	w/o shallow and deep level feature transfer	0.596 (−0.006)
11	w/o middle and deep level feature transfer	0.590 (−0.012)

The best measures are in bold. The upperarrow indicates that a higher value is better.

. The average QWK values for the ExtCTS model using BERT(base) over all prompts are given, the results with different external information removed are given in rows 1–5, and the results with different levels of transfer removed are given in rows 6–11. First, the effect of different external information on the model is verified. Comparing lines 1 to 4 of Table 7, removing $\mathbf{h}$, $\mathbf{d}$, and ${\mathbf{s}}^{″}$, the model performances all decrease, with ${\mathbf{s}}^{″}$ decreasing the most. It shows that using external knowledge can effectively enhance the model performance. Comparing lines 2 to 5 of Table 7, the models using the above three representations are all able to enhance, further demonstrating the effectiveness of fusing external knowledge.

Then, the effect of different layers of similarity transfer on the model is verified. Comparison of Table 7, lines 1 and 6 to 8 shows that the results of removing the shallow level feature transfer have been improved, and removing the middle and deep level feature transfer have been decreased. The above results indicate that the factor used at the shallow level is less effective; most likely, the model mainly contains sentence-level features at the shallow level, which are more semantically different from the deep-level features that are directly used for scoring, and thus it is more difficult to transfer features from the shallow level. In addition, migrating at a shallow level of the model may further affect the training of the overall representation of the model, thus degrading the model performance. Comparison of rows 1 and 9 to 11 of Table 7 shows that the results of using only similarity feature transfer at the deeper level of the model decreased less, and the results of using only shallow transfer decreased more and were lower than the results of the best baseline model, further suggesting that transfer at a shallow level of the model affects the model results. In contrast, similarity feature transfer at the deeper level of the model is more effective, indicating that our method is more suitable for feature transfer at the deeper level of the model.

4.5. Effect of Loss Function

This section analyzes the impact of the similarity loss function between traits proposed in this paper (TDS), and the experimental results are shown in

Table 8. Effect of similarity loss function.

$\mathit{\eta }$	Prompt								Avg QWK ↑
	1	2	3	4	5	6	7	8
1	0.635	0.580	0.643	0.642	0.677	0.607	0.407	0.522	0.589
0.7	0.665	0.583	0.646	0.656	0.675	0.611	0.421	0.562	0.602

The best measures are in bold. The upperarrow indicates that a higher value is better.

. When $\eta$ takes the value of 1, the training objective of the model contains only the scoring loss, and it can be seen that the effect of the model using only the scoring loss is slightly degraded; the results are slightly better only in Prompt 5, which proves the validity of the inter-trait similarity loss function.

Next, the trait similarity (TS) loss function proposed by the baseline [19] is compared. The difference between the two lies in the calculation of the similarity threshold, and our method proposes to utilize the cosine similarity between different trait descriptions instead of the Pearson correlation coefficient. The experimental results are shown in Figure 4 and Figure 5. The TDS loss proposed in this paper is closer to the results of the baseline modeling method, which proves that the proposed method is competitive.

4.6. Effect of Transfer Factors

This section verifies the effect of the two trait similarity factors of $f_p$ and $f_g$, and the results are shown in

Table 9. The effect of different transfer factors.

${\mathit{f}}_{\mathit{sim}}$	Prompts								AVG QWK ↑
	1	2	3	4	5	6	7	8
$f_p$	0.665	0.583	0.646	0.656	0.675	0.611	0.421	0.562	0.602
$f_g$	0.656	0.588	0.630	0.641	0.652	0.600	0.405	0.405	0.572

The best measures are in bold. The upperarrow indicates that a higher value is better.

, which shows that the effect of using the essay prompt similarity factor is better. The genre similarity factor has a negative effect on the model, which may be due to the fact that essays of similar genres have similar essay organization, language expression, etc., but there is no explicit distinction between the different information in the representation of ExtCTS, and the similarity factor will transfer all the information of the essays to the target prompt and thus have a negative effect. Compared to genre information, the prompt unifies the entire essay and the overall representation of the essay is influenced by the prompt information; therefore, using the essay prompt for transfer is effective.

5. Conclusions

In this paper, we propose extra-essay knowledge representation and similarity transfer to enhance shared representation learning in cross-prompt essay trait scoring. Our framework addresses two critical limitations of existing methods: (a) the oversight of external knowledge in measuring inter-prompt commonality, and (b) insufficient alignment between task traits. By explicitly incorporating external knowledge representation to enrich task-specific understanding and quantifying sample similarity through knowledge transfer, we effectively bridge the gap between source and target prompts. Experiments on the ASAP++ dataset show significant improvements over baselines, validating the effectiveness of the proposed method.

In the future, we will further refine the trait descriptions and explore effective combinations of large language models (LLMs), which can give fine-grained evaluations and timely feedback to enhance cross-prompt essay trait scoring. Integrating LLMs into essay scoring systems presents both challenges and opportunities. LLMs might prioritize shallow features (e.g., word choice) over critical thinking, stifling creativity. LLMs also complicate transparent scoring rationales, raising issues of credibility and fairness.

Cross-prompt scoring aims to generalize the evaluation of essays across different prompts or writing tasks, and our model suffers from the following limitations. First, the model is susceptible to prompt bias, and the training data are mostly focused on common prompts, which decreases the accuracy of the evaluation effect for prompts with large domain spans. Second, it is difficult to generalize the scoring criteria, as the core requirements of different writing tasks vary greatly and the scoring scales are different, making it difficult for a single model to accurately adapt to diverse scoring dimensions. Finally, data sparsity limits the robustness of the model, and insufficient samples for some topics may lead to overfitting or underfitting of the scoring rules.