A Multi-Level Feature Fusion Network Integrating BERT and TextCNN

Abstract

With the rapid growth of job-related crimes in developing economies, there is an urgent need for intelligent judicial systems to standardize sentencing practices. This study proposes a Multi-Level Feature Fusion Network (MLFFN) to enhance the accuracy and interpretability of sentencing predictions in job-related crime cases. The model integrates hierarchical legal feature representation, beginning with benchmark judgments (including starting-point penalties and additional penalties) as the foundational input. The frontend of MLFFN employs an attention mechanism to dynamically fuse word-level, segment-level, and position-level embeddings, generating a global feature encoding that captures critical legal relationships. The backend utilizes sliding-window convolutional kernels to extract localized features from the global feature map, preserving nuanced contextual factors that influence sentencing ranges. Trained on a dataset of job-related crime cases, MLFFN achieves a 6%+ performance improvement over the baseline models (BERT-base-Chinese, TextCNN, and ERNIE) in sentencing prediction accuracy. Our findings demonstrate that explicit modeling of legal hierarchies and contextual constraints significantly improves judicial AI systems.

1. Introduction

Job-related crimes refer to criminal offenses committed by public officials within state institutions, state-owned enterprises, or people’s organizations. These crimes involve the abuse of state-delegated powers through acts of corruption, fraud, dereliction of duty, infringement of citizens’ personal rights, or subversion of state regulations—all punishable under China’s Criminal Law. Suppose job-related crimes cannot be dealt with in a timely and fair manner. In that case, it will affect the credibility and efficiency of the government, which is why the anti-corruption campaign aims to crack down on job-related crimes. Although China’s rigorous anti-corruption campaigns have yielded significant results in recent years, the adjudication of job-related crimes faces systemic challenges, including protracted trial cycles and severe case backlogs, particularly in grassroots courts. Public judicial data reveals a consistent upward trend in such cases processed by courts and procuratorates. Within traditional judicial workflows, judges manually analyze voluminous historical precedents and legal provisions—a process that is labor-intensive, time-consuming, and prone to sentencing inconsistencies due to subjective interpretation. These deviations not only undermine judicial fairness but also risk eroding public trust in judicial transparency. Consequently, carrying out research to enhance sentencing efficiency, minimize human interference, and standardize judicial outcomes has become an urgent imperative.

The advances in research on artificial intelligence technology have been rapid in recent years, and they have a significant impact on practical work in the legal field. With the promulgation of the EU Artificial Intelligence Act, Legal Artificial Intelligence has gradually attracted widespread interest from judicial practice and artificial intelligence researchers, gaining rich research paradigms and traditional diversity from other disciplines, such as natural language processing, machine learning, mathematical statistics, and big data science. At the same time, Legal AI has brought efficiency improvements and fair judgments in judicial practices.

Legal AI should play a role in the governance of job-related crimes, especially by fully leveraging the legal semantic capture capabilities of BERT and TextCNN. To address urgent judicial challenges of job-related crimes in China’s anti-corruption campaign, this study pursues research objectives, including designing a hierarchical feature fusion network for legal semantics, optimizing domain adaptation for job-related crimes in legal classifications, and advancing AI’s role in China’s anti-corruption campaign through feature fusion and a hierarchical workflow.

2. Related Work

The rapid development of artificial intelligence technology has brought new possibilities to judicial AI [1]. Early systems mostly focused on expert rule libraries or traditional machine learning methods (such as support vector machines), matching historical precedents with structured case features to provide trial references for judges. Goncalves et al. annotated legal texts, including 42 rulings of the European Court of Human Rights [2]. Nallapati et al. proposed that traditional machine learning classifiers have shortcomings in the connection and extraction of relationships based on the argument from unstructured text to structured text [3]. However, such methods rely on manually defined feature engineering, which makes it difficult to deal with the complex semantics and unstructured features of legal texts. Especially when dealing with cases of job-related crimes, suggestions by judicial AI are often rigid due to the variability of the case and the cross-application of legal provisions, resulting in limited practical application effects.

Natural language processing technology, with its advantages in text understanding and semantic analysis, has shown potential in scenarios such as legal Q&A and contract review [4]. Mikolov et al. proposed the word2vec word embedding method to address the inherent shortcomings of traditional word representation methods in word order and idiomatic expression [5]. The CBOW (continuous bag of words) and Skip–Gram models use neural networks to predict words in contextual environments, capture a large number of accurate syntactic and lexical semantic relationships, and learn high-quality distributed representation methods for words, namely, word embedding representation. The introduction of the negative sampling hierarchical softmax method improves the learning speed of dense vectors for word embedding [6]. Aletras et al. combined legal provisions, word embeddings, and parts-of-speech tagging to provide predictions of judgment outcomes by describing the facts of the input legal case [7]. In order to capture more multi-level contextual information, Vaswani et al. proposed an attention mechanism. The Transformer model based on the attention mechanism makes gradient descent more effective and has a particular applicability [8]. BERT (Bidirectional Encoder Representations from Transformers) is composed of multiple layers of Transformer encoders stacked together, each layer containing a multi-head self-attention mechanism and a feedforward neural network [9]. The self-attention mechanism dynamically calculates the correlation weights between words, enabling the model to focus on global context and local key information simultaneously [10]. BERT uses bidirectional Transformer encoders, where the self-attention mechanism allows each token to directly attend to all other tokens in the input sequence. Every token representation output by BERT in later layers is inherently contextualized, incorporating global and coarse-grained information from the entire sequence to predict crime, thereby capturing long-range dependencies and semantic relationships beyond immediate neighbors. Kim proposed the TextCNN model [11], which uses a convolutional neural network (CNN) model to classify text. The text is transformed into a dense vector representation through word embedding. Then, the discrete vocabulary is mapped to a continuous semantic space, effectively capturing local feature information for the text to be classified [12]. In addition, TextCNN applies multiple convolutional filters of varying widths that slide over the word embedding sequence. Therefore, each filter specializes in detecting specific local patterns (n-grams) within its window size. At the same time, each filter produces a feature map that highlights where its local and fine-grained information is located, enabling quantitative analysis of the case.

In recent years, the LLM (large language model) has developed rapidly, which applies a large amount of text data to train deep learning models that can generate natural language text or understand the meaning of language text [13]. LLMs can fully leverage the advantages of pre-training on large datasets by using computing power and deep learning patterns [14]. Yue proposed the DISC-LawLLM model based on an LLM, which can solve specific tasks in the legal domain [15]. Moreover, Cui proposed using llama and a knowledge graph to construct the ChatLaw model, which has been iterated to the second generation ChatLaw2 MoE [16]. Represented by these models, multiple Chinese legal models have achieved excellent performance in Chinese legal Q&A and case matching [17]. The Lexis+AI platform launched by the Lexis company integrates large language models such as GPT-4, which can generate sentencing recommendations by analyzing variables such as case nature, involved amount, and defendant background [18]. The Law-AI tool developed by the Casetext group uses natural language processing technology to assist lawyers in legal retrieval, following human instructions to work [19]. LLMs have a wide range of application scenarios. They conduct pre-training on massive data through self-supervised learning or semi-supervised learning and then further optimize their performance and ability through fine-tuning and human alignment [20]. The existing legal intelligence applications mainly focus on general legal services scenarios [21], such as case retrieval or legal consultation, and support for sentencing decisions mostly stays at the macro suggestion level.

Expert systems and traditional machine learning methods have some limitations in the process of predicting job-related crimes. Manual rules failed to handle edge cases, e.g., novel corruption tactics, causing higher error rates in complex job-related crimes [22]. Typically, lawyers dedicate months to encoding rules and making systems scalable in response to evolving law [23]. Without contextual knowledge, ambiguous language, such as the terms “gift” and “bribe,” cannot be interpreted, leading to rigid outcomes [24]. Document failures of rule-based systems in common law show inaccuracy in statutory interpretation [25]. Traditional machine learning proved inadequate for job-related crimes, where case variability and contextual semantics, such as abuse of power, demand adaptive reasoning [26]. Compared with expert systems and traditional machine learning methods, word embeddings and convolutional neural networks are more of a focus for researchers. Word embeddings encode words as dense vectors, enabling similarity metrics and distributional semantics [27]. Due to the large size of the dataset, efficient training could reduce compute costs for the corpora through negative sampling [28]. Learned analogies of word embeddings could help parse legal relationships in job-related crimes, such as man and king or woman and queen [29]. However, due to overfitting in global feature associations, word embeddings are static embeddings [30], i.e., the same vector for all the same word senses, and they fail to encode the windows of words. Therefore, there may be deviations in understanding polysemous words, such as “gift” in legal and everyday contexts. Feature fusion is a method that fully considers both overfitting in global feature associations and the encoding of word windows. Feature fusion in deep learning integrates multiple cues or features representing different aspects of input characteristics to create a more comprehensive feature representation for tasks like classification. The linear combination of feature statistics and its extension to a general nonlinear fusion method involve minimizing a mean-square error and a regularization term [31]. As the combination of features from different layers or branches, it is often implemented via simple operations, such as summation or concatenation. Attentional feature fusion is described by short- and long-bound connections as well as feature fusion within the frontend layer [32].

However, the current research on legal classification prediction lacks an in-depth examination of auxiliary sentencing for the subdivision scenario of job-related crimes, which is characterized by a lack of logical analysis of criminal circumstances, factual determination, and legal relationships. In particular, insufficient attention is paid to research on the feature fusion of criminal circumstances. Job-related crime is a type of criminal offence that arises from the facts and legal consequences of a duty act. Its legal text includes both coarse-grained information for predicting crime and noncrime and fine-grained information for quantitatively analyzing the length of imprisonment of the defendant. This paper proposes a multi-level feature fusion method based on attention and convolutional kernels to address the problem of global and local feature segmentation in legal texts related to job-related crimes. The attention mechanism captures global features by embedding different semantic positions, while the convolutional kernel obtains local features through sliding windows. Multiple granularity features are then fused to enhance the model’s ability to perceive semantic information in job-related crime scenarios and improve its predictive performance for job-related crimes.

3. Features of Job-Related Crimes

3.1. Basis of Sentencing for Job-Related Crimes

The theory of sentencing for job-related crimes is based on the Criminal Law of the People’s Republic of China combined with judicial interpretations, criminal policies, and judicial practice experience to construct a clear hierarchy of punishment systems that balance fairness and efficiency. The sentencing levels for job-related crimes are divided into six categories, i.e., exemption from punishment, 0–1 year, 1–3 years, 3–5 years, 5–10 years, and more than 10 years. Its theoretical basis is rooted in the principle of proportionality between crime and punishment and the criminal trial policy of balancing leniency and severity, aiming to balance the corresponding relationship between social harm, subjective malice, and punishment through scientific and standardized standards.

In specific sentencing practices, the identification of punishment levels is mainly based on the objective harm and subjective malice of criminal behavior. Taking the crime of embezzlement as an example, Article 383 of the Criminal Law clearly shows that the amount of embezzlement between CNY 30,000 and CNY 200,000 is considered “relatively large” and can be sentenced to fixed-term imprisonment of up to 3 years or detention; CNY 200,000 to 3 million is regarded as a “huge amount” and shall be sentenced to imprisonment for not less than 3 years but not more than 10 years; and an amount of over CNY 3 million is considered “exceptionally large” and is subject to imprisonment for more than 10 years to life imprisonment. This is shown in

Table 1. Identification of harm caused by embezzlement.

Amount Involved (CNY)	Nature of Crime	Punishment
<5000	slight	not be registered
5000–30,000	general	aggravating circumstances
30,000–200,000	relatively large	less than 3 years
200,000–3,000,000	huge	between 3 and 10 years
>3,000,000	extremely huge	more than 10 years

. Although this amount standard has rigid requests, judicial interpretations further introduce flexible standards such as “serious circumstances” and “dire circumstances”. For example, in cases of embezzlement, disaster relief, poverty alleviation, or adverse social impact, even if the amount does not reach the threshold, it can still be upgraded to punishment, as shown in

Table 2. Aggravating circumstances in job-related crimes.

Specific Circumstances	Subjective Nature	Objective Harm
rescue	first offence	transferring illicit money
flood prevention	once embezzled	destruction of evidence
epidemic control	previously accepted bribes	escaping overseas
poverty alleviation	previously misappropriated	sponsoring others to commit crimes
illicit money	other criminal records	inability to recover illicit money

.

3.2. Global and Local Features of Sentencing

The global features of job-related crimes are the overall information about the legal relationship of job-related crimes obtained after analyzing the entire legal text. This information focuses on the legal relationship regulated by job-related crimes in the text, ignores local information, and has good stability, maintaining appropriate penalties and accountability. The criminal compositions, causal connections, and criminal accountability involved in job-related crimes have legality, specificity, and universality. The subjective purpose is adapted to the consequences of the crime, and the criminal behavior is consistent with the job-related responsibility. The global features of criminal embezzlement are shown in

Table 3. Global features in embezzlement.

Subjects	Objects	Subjective Factors	Objective Factors
governmental officials	state assets	directly	act of duty
trustees for public property	public property	intentionally	embezzlement
exploitation of officials	state nature	illegally	fraud
non-governmental officials	public nature	not-faulty	concealment

. The local features of job-related crimes refer to the features obtained in specific areas of the legal text. These can reflect the local structure and detailed information on the conviction and sentencing of job-related crimes. The determination of the amount of embezzlement belongs to local features, and aggravating factors also belong to local features. The combination of these two kinds of features can elevate the level of judgment in job-related crimes. The local features of the criminal embezzlement are shown in

Table 4. Local features in embezzlement.

Amount Involved (CNY)	Circumstances	Means	Regions
<5000	rescue	misappropriation	1-tier cities
5000–30,000	flood prevention	stealing	2-tier cities
30,000–200,000	epidemic control	swindle	3-tier cities
200,000–3,000,000	poverty alleviation	rebate	4-tier cities
>3,000,000	illicit money	indirect	5-tier cities

.

In judicial practice, the scientificity and adaptability of sentencing standards should be considered together. On the one hand, the fixed-amount standard may lead to sentencing differences between different areas due to unbalanced economic development. For example, the actual tolerance of “large amounts” in economically developed areas is higher than that in less developed regions, which damages legal unity and cannot demonstrate the power of legal punishment and deterrence. For example, the embezzlement of CNY 30,000 in economically underdeveloped areas affects the poverty relief of hundreds of people, and its criminal consequences are far greater than those in developed regions. On the other hand, the boundary of judges’ discretion needs to be further refined through judicial interpretation. The standard stipulates that probation or exemption shall not be applied to job-related crimes involving serious circumstances and destructive social impact, based on judicial opinions on several issues concerning the strict application of probation and exemption from criminal punishment in handling cases of job-related crimes, thus reducing the discretion space. The dynamic adjustment mechanism is introduced to link sentencing amount with regional per capita income to enhance the standard’s adaptability to the times.

3.3. Definition of Classification Task and Sentencing

According to the sentencing provisions for job-related crimes, they can be clearly classified into six categories, i.e., exemption from punishment, 0–1 year, 1–3 years, 3–5 years, 5–10 years, and more than 10 years. These six categories not only reflect the quantitative grading of the social harm of job-related crimes but also provide clear classification functions for the construction of machine learning models. In judicial practices, judges need to accurately match the text of the job-related crime to corresponding sentence ranges based on the facts of the case. Essentially, this involves mapping unstructured text (such as case descriptions in judgment documents) to preset category labels, which is highly consistent with the framework of multi-classification tasks from input features to output labels.

The choice of multi-classification tasks over regression or binary classification methods is mainly based on the legal requirement of discreteness and the normative needs of judicial decision-making. The law has strict boundaries for sentencing. For example, there is an unbreakable legal difference between “3–5 years” and “5–10 years”. Suppose a regression model is used to predict the specific length of a sentence. In that case, it may ignore such structural boundaries due to numerical continuity, resulting in prediction results deviating from legal standards. The regression model may output “4.8 years”, but according to criminal law regulations, this result needs to be classified as “3–5 years” instead of being used directly as an independent value. In addition, multi-classification tasks can generate probability distributions for each sentencing interval. For example, the probability of a case being sentenced to “5–10 years” is 75% and “3–5 years” is 20%. This probabilistic output not only provides a reference for judges but also assists them in balancing the rationality of different sentencing options, enhancing the transparency and scientific handling of decision-making.

4. Multi-Level Feature Fusion

4.1. Attention Mechanism for Global Features

In the text of job-related crimes, establishing the legal relationship that constitutes job-related crimes affects the adaptation of conviction and sentencing; for example, establishing the correlation between statements such as “the amount of embezzlement is particularly huge” and “causing national losses” is crucial for predicting the result of sentencing. The pre-trained language model represented by BERT captures deep semantic features related to sentencing in the text through the self-attention mechanism, such as the strong correlation between descriptions such as “multiple instances of embezzlement” and “refusal to return embezzled money” and severe penalty. After encoding the text into a high-dimensional vector, the model maps it to one of six sentencing labels through a fully connected layer, achieving end-to-end learning from text to category of sentencing, as is shown in Figure 1. In the pre-training stage, BERT learns the intrinsic rules of language through a Masked Language Model (LLM) and Next Sentence Prediction (NSP). The Masked LM forces the model to understand contextual logic by randomly masking some vocabulary and predicting the masked words. For example, in Figure 1, “amount” and “country” are masked. NSP enhances the model’s understanding of discourse structure by assessing the continuity of sentence pairs. This pre-training endows BERT with powerful semantic representation capabilities, enabling it to extract implicit legal elements from unstructured text.

Establishing the legal relationship of job-related crimes depends on encoding global features, and segment attributes can introduce more global information into the model. In establishing the legal relationship of job-related crimes, there are many aspects of identification, such as subject, object, subjective, and objective factors, for example, “from December 2012 to April 2013, the defendant Xia…”. The segment attributes for words are shown in

Table 5. Properties of case description fields in job-related crimes.

Token	Segment Property	Token	Segment Property	Token	Segment Property
2012	CT	在	CQ	以	CM
年	CT	担任	CQ	虚开	CM
12	CT	某市	CQ	发票	CM
月	CT	某局	CQ	方式	CM
至	CT	副局长	CQ	，	CM
2013	CT	期间	CQ	套取	CF
年	CT	,	CQ	公款	CF
4	CT	利用	CC	人民币	CF
月	CT	职务	CC	3	CF
,	CT	之	CC	万	CF
被告人	CQ	便	CC	元	CF
夏某	CQ	,	CC	。	CF

Chinese characters in the corpus have caused non-English content.

. The word is marked with a segment attribute, and each segment attribute identifies the structured metainformation in the legal relationship of job-related crimes. The meaning of the segment attribute mark is shown in

Table 6. Meanings of segment property tags.

Segment Property Tags	Meanings	Presenting Text
CT	time of crimes	digitals representing date
CQ	qualification of offences	names and official positions
CC	causes of crimes	text of causal relationships
CM	means of crimes	words such as “with” prepositions
CF	facts of crimes	digitals representing the amount

.

The self-attention mechanism can automatically identify key sentencing elements and generate high-dimensional semantic vectors through nonlinear transformations of multi-layer encoders. The Transformer utilizes the marker of CLS representing the global information of the entire legal text, which aggregates vectors of contextual embeddings and then outputs probability distributions of six labels of sentencing through a fully connected layer. For example, The probability of the model determining that a case belongs to the “5–10 year” category is 80%, providing a reference for judges. During the fine-tuning, by post-training on a specialized dataset for job-related crimes, self-attention can further adapt to the length of legal texts and optimize its ability to distinguish easily confused categories.

4.2. Convolutional Neural Networks for Local Features

Local features in the legal text of job-related crimes can be manifested as “after the offences”, “illicit funds”, “already refunded”, etc. Capturing such features can help determine the level of punishment. The architecture of TextCNN mainly includes the embedding layer, convolution layer, pooling layer, and fully connected classification layer, as shown in Figure 2. Firstly, the input text (such as the case description in the judgment document) is transformed into a dense vector representation through word embedding, mapping discrete vocabulary to a continuous semantic space. Subsequently, the convolutional layer uses convolutional kernels of different sizes, performs sliding-window scanning on the text, and then extracts local n-gram features. A 3 × 1 convolutional kernel may cover three consecutive words, capturing their combined semantics. The output of each convolutional kernel is subjected to a nonlinear activation function to generate a feature map. Next, the max-pooling operation is performed to reduce the dimensionality of each feature map, preserving the significant features in each convolutional channel. For example, the feature corresponding to “involved amount of CNY 30,000” is used as the key information of the amount dimension in the legal relationship of job-related crimes. Finally, all pooled features are concatenated into a global vector, which is input into a fully connected layer for classification, which then outputs probability distributions of six labels of sentencing.

4.3. MLFFN: Multi-Level Feature Fusion Network

The Multi-Level Feature Fusion Network needs to consider both the overall structural information of legal relationships in job-related crimes and capture fine-grained local circumstances. Therefore, MLFNN combines the global features on the encoder and the local features on the convolutional kernel by cascading Bert and TextCNN. It could improve the distribution of weighting and responsiveness of feature maps, obtain richer feature representations of job-related crimes, and enhance the model’s perception and understanding of conviction and sentencing for job-related crimes. MLFNN is shown in Figure 3.

The applicability of MLFFN in the task of sentencing for job-related crimes is reflected in its ability to map complex legal texts to discrete sentencing categories. Specifically, after the input for case fact description is segmented and embedded (word embedding, segment embedding, position embedding), the Transformer encoders extract semantic features layer by layer, as shown in Formulas (1)–(3), i.e.,

$$ET=\left[\begin{array}{ccccc}e{t}_{11}& \dots & e{t}_{1n}& \dots & e{t}_{1d}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{t}_{m1}& \dots & e{t}_{mn}& \dots & e{t}_{md}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{t}_{l1}& \dots & e{t}_{ln}& \dots & e{t}_{ld}\end{array}\right],$$

(1)

$$ES=\left[\begin{array}{ccccc}e{s}_{11}& \dots & e{s}_{1n}& \dots & e{s}_{1d}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{s}_{m1}& \dots & e{s}_{mn}& \dots & e{s}_{md}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{s}_{l1}& \dots & e{s}_{ln}& \dots & e{s}_{ld}\end{array}\right],$$

(2)

and

$$EP=\left[\begin{array}{ccccc}e{p}_{11}& \dots & e{p}_{1n}& \dots & e{p}_{1d}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{p}_{m1}& \dots & e{p}_{mn}& \dots & e{p}_{md}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ e{p}_{l1}& \dots & e{p}_{ln}& \dots & e{p}_{ld}\end{array}\right].$$

(3)

The hidden vector dimension in the embedding layer is 768. The input word vector can be a GloVe word vector with 100 dimensions, which is mapped into a word embedding ET through a 768-dimensional hidden layer called embeddings of the token, as shown in Formula (1). The input segment vector can be a vector involving the marker of segment property for job-related crimes, and the dimension of the segment vector depends on the legal relationship in the contextual information of job-related crimes. According to Table 6, the dimension of the segment vector with one-hot representation is 5. After 768 dimensions of hidden layer mapping, it is embedded into ES, as shown in Formula (2). The dimension of the input position vector depends on the length of the legal text. Generally, the dimension with one-hot representation is 512, which is mapped to a positional embedding EP through a 768-dimensional hidden layer, as shown in Formula (3). $e{t}_{11}$ represents the first-dimension value of the word embedding for the first token, where d is the dimension of the word embedding. $e{t}_l$ represents the word embedding of the last token in the input sentence, and $e{t}_s$ represents the word embedding of the m-th token. The first token can represent a [CLS] identifier with a special meaning, and the punctuation mark in the middle can be represented as [SEP]. $e{p}_m$ is the positional embedding of the m-th token. $e{s}_m$ is the segment embedding for the contextual information where the m-th token is located in the sentence. At the same time, segment embedding can introduce more global information into the model.

The MLFNN frontend encoder can automatically identify key sentencing elements through a self-attention mechanism, using the elements that constitute the legal relationship of job-related crimes as model inputs. After embedding hidden-layer mapping processing, the word embedding ET, segment embedding ES, and position embedding EP are added together, as shown in Formulas (4) and (5), i.e.,

$$EMB=\left[\begin{array}{ccccc}em{b}_{11}& \dots & em{b}_{1n}& \dots & em{b}_{1d}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ em{b}_{m1}& \dots & em{b}_{mn}& \dots & em{b}_{md}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ em{b}_{l1}& \dots & em{b}_{ln}& \dots & em{b}_{ld}\end{array}\right],$$

(4)

and

$$em{b}_{mn}=e{t}_{mn}+e{s}_{mn}+e{p}_{mn}.$$

(5)

After embedding segment properties that represent the legal relationship of job-related crimes, the MLFFN multi-layer encoders generate high-dimensional semantic vectors containing global feature information through nonlinear transformation. The output encoding corresponding to each token maps word embedding, segment embedding, and position embedding in the opposite end token. Moreover, it includes attention allocation for contextual information, i.e.,

$$MultiHead(Q,K,V)=Concat(hea{d}_1,\dots ,hea{d}_i,\dots ,hea{d}_h)W^o,$$

(6)

$$hea{d}_i=Attention(Q_i,K_i,V_i),$$

(7)

and

$$\begin{array}{c}\hfill Q_i=EMB\times W_i^Q,\\ \hfill K_i=EMB\times W_i^K,\\ \hfill V_i=EMB\times W_i^V,\\ \hfill Attention(Q_i,K_i,V_i)=softmax({\displaystyle \frac{Q_i{K}_i^T}{\sqrt{d_k}}})V_i.\end{array}$$

(8)

It shows that the different attention heads $hea{d}_1,\dots ,hea{d}_i,\dots ,hea{d}_h$ are concatenated by $Concat(hea{d}_1,\dots ,hea{d}_i,\dots ,hea{d}_h)$ and transformed into the embeddings input of the next layer by $W^o$ in Formula 6. The three matrices of $Q_i$, $K_i$, and $V_i$ are obtained by linearly transforming the input vector of embeddings, and the corresponding transformation matrices are $W_i^Q$, $W_i^K$, and $W_i^V$, where $W_i^Q$, $W_i^K$, and $W_i^V$ are pre-training parameters of the model, and $W_i^Q\in {\mathbb{R}}^{d_{model}\times d_k}$, $W_i^K\in {\mathbb{R}}^{d_{model}\times d_k}$, $W_i^V\in {\mathbb{R}}^{d_{model}\times d_v}$. The feature map encoded with global information on job-related crimes is output according to Formulas (6)–(8). The MLFNN backend TextCNN uses convolution kernels from sliding windows to filter the global feature map and extract the plot of job-related crimes, making the semantic understanding of local features more accurate. Embedding segment properties that represent the legal relationship of job-related crimes, the MLFFN multi-layer encoders generate high-dimensional semantic vectors containing global feature information through nonlinear transformation. After each token is encoded and output in the MLFNN frontend, it contains segment information on the legal relationship structure. The size of the global feature map is $l\times d$ (usually 512 × 768). The MLFNN backend convolutional layer adopts dual channels with a convolution kernel size of $m\times d$. The output of the channel convolutional layer is calculated according to Formula (9), i.e.,

$$OutConv(i,c,k)=\sum _{p=1}^r\sum _{q=1}^d{w}_{p,q}{x}_{p+i-1,q}.$$

(9)

where $w_{p,q}$ represents the connection weight of the convolutional kernel, with a dimension of $r\times d$, and $x_{p+i-1,q}$ represents the value of the $p+i-1$th row of the $qth$ dimension in the global feature map output by the MLFNN frontend. The pooling layer adopts the maximum pooling method and outputs the larger one. Then, the output of the weight by the convolutional kernel is fully concatenated to form a fully connected layer and linked to a softmax layer, which is used for the classification of labels of job-related crimes.

The workflow of the MLFNN model is shown in Figure 4, which includes an input layer, a frontend BERT layer, a backend TextCNN layer, and an output layer. In the input layer, legal documents like the indictment and verdict are split into tokens with position IDs and segment IDs, such as [FACT] and [STATUTE]. Tokenization utilizes WordPiece and legal lexicon expansion, and segment embeddings assign unique IDs to document sections. In the frontend BERT layer, the information of the tokens, the positions, and the segments passes through BERT’s Transformer encoders, and multi-head attention captures long-range dependencies to extract global contextual features. The output of the frontend BERT layer is the global encoding sequence, comprising token embeddings, position embeddings, and segment embeddings. In the backend TextCNN layer, parallel convolutions are applied with varying kernel sizes to extract local n-gram patterns from fused features, and max-pooling retains the most salient local features. The output of the backend TextCNN layer concatenates all pooled features. In the output layer, a fully connected layer and softmax activation predict job-related crime categories.

The MLFNN model differs from other existing models in two aspects. On the one hand, the input of the frontend Bert of MLFNN utilizes tags for words, segments, and positions, corresponding to word embeddings, segment embeddings, and position embeddings, respectively. The segment embeddings are related to criminal circumstances, as indicated in Table 5 and Table 6, according to the encoding. The labels for criminal circumstances differ from those in general knowledge and are designated according to legal provisions. Therefore, the output vectors of the hidden layer of the MLFNN frontend statistically include logical relationships between legal entities. On the other hand, the frontend of MLFNN freezes model parameters except for the last hidden layer, which can be fine-tuned during training based on the crime dataset to provide more contextual information for the MLFNN backend TextCNN to capture important information for legal application classification. The MLFNN algorithm is shown in Algorithm 1.

Algorithm 1 MLFNN algorithm.

Require:  bert_model: pre-trained bert model; textcnn_model: input_dim is equal to     bert_output_dim; loss_fn: criteria of training with cross entropy loss; Ensure:  optimizer: optimal state or approximate optimal state;  1: bert_model = AutoModel.from_pretrained(“bert-base-chinese”)  2: # frozen bert parameters  3: for $param$ in $bert\_model.parameters\left(\right)$ do  4:       $param.requires\_grad=False$  5: end for  6: $inputs$=tokenizer($texts$,$token$,$segment$,$position$)  7: $outputs$=bert_model($inputs$,$output\_hidden\_states$=$True$)  8: $dataset$=load_dataset($outputs$)  9: $optimizer$=optim.Adam($textcnn\_model.parameters\left(\right)$,$lr$=lr) 10: $criterion$=nn.CrossEntropyLoss() 11: for $epoch$ in $range\left(epochs\right)$ do 12:       textcnn.train() 13:       $total\_loss$=0 14:       for i in $range(0,len(train\_texts),batch\_size)$ do 15:             $embeddings$=bert_model($batch\_texts$) 16:             $logits$=textcnn_model($embeddings$) 17:             $loss$=criterion($logits$,$batch\_labels$) 18:             optimizer.zero_grad() 19:             loss.backward() 20:             optimizer.step() 21:             $total\_loss$+=loss.item() 22:       end for 23: end for

5. Experiments and Results

5.1. Data Description

The current datasets of legal cases mostly come from China Judgments Online [33], the Bank of the National Judicial Examination Center [34], and Challenge of AI in Law [35]. China Judgments Online covers various types of cases, including civil, criminal, administrative, etc., involving factual determination, evidence and basis for the determination, and verdict results. The National Judicial Examination Center provides teaching cases and standardized question banks selected by legal scholars. The case dataset from China Judgments Online and National Judicial Examination Center needed to be obtained through web crawler tools. Unlike the two dataset sources mentioned above, Challenge of AI offers an authoritative dataset of the judicial field that can be publicly downloaded. Therefore, the datasets of job-related crimes come from cases from China Judgments Online, the question bank from the National Judicial Examination Center, and the CAIL2018 (Challenge of AI in Law) dataset.

Data cleaning and data preprocessing are important and interconnected steps in preparing data for experiments. Personal privacy in cases involving sensitive data should not be exposed to the public; therefore, the content of cases related to personal privacy has been desensitized through technological means. The facts of the cases and sentencing standards, after being cleaned and preprocessed, comply with the sentencing circumstances stipulated in the Criminal Law. The structured data ensures a one-to-one correspondence between criminal facts and sentencing recommendations, providing natural attributes and labels for the corresponding relationship between input and output during modeling. Each case only contains two fields, namely, “statement of fact” and “range of verdict and sentence”. The ranges of verdicts and sentences are strictly divided into six categories according to the criminal law, namely, exemption from punishment, 0–1 year, 1–3 years, 3–5 years, 5–10 years, and more than 10 years.

The datasets of job-related crimes involved in this paper are specialized resources containing approximately 270,921 Chinese legal documents related to job-related crimes. The cases used for the experiments are divided into a training set, a verification set, and a test set, with a ratio of 6:2:2. This includes 162,552 cases in the training dataset, 54,185 cases in the verification dataset, and 54,184 cases in the test dataset.

5.2. Evaluation Metrics

On the one hand, legal judgments require both precision and reliability. On the other hand, legal predictions rely on extensive historical case data, and artificial intelligence is revolutionizing the legal profession by enabling attorneys to predict case outcomes with greater accuracy. Although accuracy reflects the overall correctness of the prediction, it is sensitive to data with imbalanced categories, as shown in Formula (10). TP represents a true positive, where the model predicts a positive sample that is actually positive. TN represents a true negative, where the model predicts a negative sample that is actually negative. FP represents a false positive, where the model predicts a positive sample that is actually negative. FN represents a false negative, where the model predicts a negative sample that is actually positive. To objectively describe the model’s performance, precision, recall, and F1 score are used as evaluation metrics, as shown in Formulas (11)–(13). Precision emphasizes the reliability of predicting positive classes to avoid wrongful judgments, suitable for scenarios that require reducing false positives. Recall ensures that sufficient positive class samples are identified to avoid missed judgments. The F1 value is the harmony between precision and recall, meeting the rigid demand for fairness in judicial scenarios.

The experimental results are related to the multi-classification task of predicting the sentence of related job crimes, and the dataset has a class imbalance. For example, the number of cases with judgments more than 10 years reached 36,653 in the test set, accounting for 70.2%. Therefore, weighted average precision (WAP), weighted average recall (WAR), and weighted average F1 value (WAF1) are used as performance evaluations, as shown in Formulas (14)–(16). Additionally, $P_i$ and $R_i$ indicate the precision and recall of label i, respectively. $count\left(labe{l}_i\right)$ and $count\left(dataset\right)$ represent the count of label i and the count of the dataset in Formulas (14), (15) and (16), respectively.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},$$

(10)

$$Precision={\displaystyle \frac{TP}{TP+FP}},$$

(11)

$$Recall={\displaystyle \frac{TP}{TP+FN}},$$

(12)

$$F1=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}},$$

(13)

$$WAP=\sum _{i=1}^C({\displaystyle \frac{count\left(labe{l}_i\right)}{count\left(dataset\right)}}{P}_i),$$

(14)

$$WAR=\sum _{i=1}^C({\displaystyle \frac{count\left(labe{l}_i\right)}{count\left(dataset\right)}}{R}_i),$$

(15)

and

$$WAF1=\sum _{i=1}^C({\displaystyle \frac{count\left(labe{l}_i\right)}{count\left(dataset\right)}}F{1}_i).$$

(16)

5.3. Experimental Setup

Bert-base-Chinese, TextCNN, ERNIE, and MLFFN (our model) were selected to conduct experiments evaluating the performance of different models on the dataset of job-related crimes. The dimension of segment embedding of MLFFN frontend is 5, the word embedding adopts a 100-dimensional glove, and the convolution core of the MLFFN backend adopts a dual-channel setting. The experimental environment is provided by AutoDL’s cloud service using PyTorch 2.3.0 (https://pytorch.org/get-started/previous-versions/, accessed on 23 August 2025) in the framework and an NVIDIA V100 GPU in the hardware. The hyperparameter setup is shown in

Table 7. Setup of Hyperparameters.

Models	Bert-Base-Chinese	TextCNN	ERNIE	MLFFN
pre-training	Bert-base-Chinese	GloVe	ernie-3.0-base-zh	Bert&GloVe
batchsize	128	128	128	128
training criteria	CE	CE	CE	CE
optimizer	AdamW	AdamW	AdamW	AdamW
learning rate	5 × 10−5	1 × 10−3	5 × 10−5	1 × 10−5

, and the selection of hyperparameters takes into account practical convention, individual experience, and computational power. The dimension of GloVe pre-trained word embeddings used as input to TextCNN is 300, and the convolution kernel size covers two, three, and four windows, with each size configured with 100 filters. The segment embedding dimension of the MLFFN frontend is 5, the positional embedding dimension is 512, the word embedding adopts a 100-dimensional GloVe, and the MLFFN backend convolution kernel adopts a dual-channel setting.

5.4. Results and Discussion

Large models used for comparative analysis have a huge number of parameters and are prone to overfitting training data, remembering noise rather than learning patterns. It is necessary to evaluate the performance of the validation set (such as loss, accuracy, etc.) after each training epoch. Suppose the training set metrics continue to improve while the validation set metrics decrease. In that case, it indicates that the model is beginning to overfit, which requires a timely adjustment of the training strategy.

The training will automatically end when the performance of the validation set no longer improves for more consecutive epochs. At the start of model training, the cross-entropy loss value rapidly decreases. By observing the accuracy curve of models on the validation set, the model reached a stable state after approximately three epochs and 6000 iterations of training, as shown in Figure 5 and Figure 6. The evaluation metrics, including ACC (accuracy), WAP, WAR, and WAF1, of each model on the test set are shown in

Table 8. Experimental result of the fine-tuning experiments.

Models	ACC	WAP	WAR	WAF1
Bert-base-Chinese	0.7654	0.7317	0.7654	0.7177
TextCNN	0.7132	0.7293	0.7132	0.6344
ERNIE	0.7625	0.7502	0.7625	0.7144
MLFFN (ours)	0.7683	0.7525	0.7683	0.7300

and

Table 9. Performances of Bert-base-Chinese, TextCNN, ERNIE, and MLFFN.

Models	ACC	WAP	WAR	WAF1
Bert-base-Chinese	0.7654	0.7317	0.7654	0.7177
TextCNN	0.7231	0.7293	0.7132	0.6344
ERNIE	0.7152	0.7502	0.7625	0.7144
MLFFN (ours)	0.8156	0.8052	0.8156	0.7854

.

From Table 8, it can be seen that compared to Bert-base-Chinese, MLFFN has a 10%, 6%, and 9% improvement in WAP, WAR, and WAF1, respectively; compared to TextCNN, MLFFN shows improvements of 10%, 14%, and 23% on WAP, WAR, and WAF1, respectively; and compared to ERNIE, MLFFN has improvements of 6%, 7%, and 10% on WAP, WAR, and WAF1, respectively.

Experimental data shows that the MLFFN model has a performance improvement of over 6% in weighted accuracy, weighted recall, and weighted F1 value, further indicating that the fusion of global and local features is effective in predicting job-related crimes. The model adopted the framework of frontend Bert and backend TextCNN, enhanced with segment embeddings for legal classification in job-related crimes. Key findings include three points as follows.

Firstly, segment embeddings could improve performance by encoding structural boundaries in legal relationships, e.g., sections, clauses, etc. Secondly, the frontend Bert captures long-range contextual semantics in legal relationship clauses of job-related crimes, especially modifiers, while the backend TextCNN extracts discriminative local patterns of job-related crimes, such as legal phrases, particularly those related to duty positions. Thirdly, frozen parameters in the frontend Bert and fine-tuning parameters in the backend TextCNN could help converge training loss and validation accuracy in the training and validation sets.

6. Conclusions

In this work, a Multi-Level Feature Fusion Network (MLFFN) is proposed, aiming at the problem of consistency of punishment and offenses in job-related crimes. The word vector, segment vector, and position vector are input to the MLFFN frontend, and the global feature encoding that can represent the legal relationship of job-related crimes is formed through nonlinear mapping of the attention mechanism. Then the MLFFN backend uses convolution kernels with sliding windows to filter the global feature map, and it obtains the local feature information on the special circumstances in job-related crimes. Experiments on the dataset of job-related crimes showed that the MLFFN model has significant performance improvements compared to models such as Bert-base-china, TextCNN, and ERNIE, verifying that the feature fusion method is in line with the applicable rules of benchmark penalties in job-related crimes. In addition, there is some uncertainty in the feature fusion in the post-training of the job-related crimes dataset, which will be the next research direction.

7. Limitations and Future Work

Despite MLFFN improving the model’s performance, this work has limitations. Segment embedding heavily relies on predefined legal relationships and document structures in job-related crimes, and it may fail to handle them when there are multiple legal relationships. Evaluation focused on the specific dataset, while performance based on other datasets remains unknown. The model utilizes machine learning to learn statistical distributions and patterns from data, replacing legal decision-making logic and reasoning, which, to some extent, poses legal risks.

Future work will be carried out in the following aspects to address these limitations. Distilled BERT variants and dynamic TextCNN filters may be considered to enhance adaptation in other datasets. Learnable segmentation embeddings will be explored for legal relationships in job-related crimes without clear boundaries. Additionally, the framework of Bert and TextCNN will be coupled with attention visualization to provide a rational explanation for predictions.