AgentsBench: A Multi-Agent LLM Simulation Framework for Legal Judgment Prediction

Abstract

The justice system has increasingly applied AI techniques for legal judgment to enhance efficiency. However, most AI techniques focus on decision-making outcomes, failing to capture the deliberative nature of the real-world judicial process. To address these challenges, we propose a large language model-based multi-agent framework named AgentsBench. Our approach leverages multiple LLM-driven agents that simulate the discussion process of the Chinese judicial bench, which is often composed of professional and lay judge agents. We conducted experiments on a legal judgment prediction task, and the results show that our framework outperforms existing LLM-based methods in terms of performance and decision quality. By incorporating these elements, our framework reflects real-world judicial processes more closely, enhancing accuracy, fairness, and societal consideration. While the simulation is based on China’s lay judge system, our framework is generalizable and can be adapted to various legal scenarios and other legal systems involving collective decision-making processes.

1. Introduction

In recent years, Artificial Intelligence (AI) has emerged as a powerful tool in the legal domain. The court and judicial system has increasingly used AI techniques across various legal tasks, including legal case retrieval, legal summarization, and legal judgment prediction (LJP). Among these, LJP stands out as one of the core challenges, aiming to predict the outcomes of court decisions, such as charges, liabilities, or sentencing, based on factual descriptions and legal statutes [1]. With the increasing digitalization of legal texts and the growing demand for efficiency, LJP systems have become a focal point of legal AI research, holding promise for improved consistency and accessibility in judicial processes [2].

Despite these benefits, current AI techniques for judicial application face several challenges. Many AI techniques developed for LJP mainly focus on outcome decision making, which is far removed from real-world judicial decision-making processes, such as discussion, debate, and consensus-building among judges [3]. Moreover, these single LJP models focus heavily on accuracy metrics and often suffer from biases, a lack of explainability, and fail to consider the ethical and social effect of algorithm judgment [4]. These issues are really significant for not only the quality of AI techniques but also the transparency of the digital justice system.

The use of a deliberative bench composed of multiple judges is crucial in judicial systems. For example, China’s lay judge system incorporates both professional and lay judges in the decision-making process [5]. The lay judges will participate in the trial and deliberations of the case under the presidency of professional judges. The bench will jointly decide on the final judgment, for example, the prison term punishment in criminal cases. The bench plays an essential role in maintaining fairness and minimizing individual biases. The collaborative nature of these deliberations ensures that multiple viewpoints are considered before reaching a final decision, thereby increasing the reliability of judicial outcomes [6].

Inspired by the deliberative bench design of the lay judge system in China, this paper explores how to mimic this structure using AI techniques. Agent-based models (ABMs) are frequently used in computational social science [7]. Early work on agent-based model (ABM) simulation attempted to introduce a multi-agent system in the legal field [8]. However, the ABM approach alone is insufficient to simulate the legal process, which entails nuanced legal reasoning and deliberation expressed through natural language.

Large language model (LLM)-based multi-agent systems have recently demonstrated remarkable potential in various domains, such as mathematical problem solving [9] and code generation [10], by leveraging collaboration and discussion frameworks [11]. Compared to traditional ABMs, LLMs are better suited for simulating legal scenarios, as they can generate natural language responses that resemble human legal discourse. Building on these advantages, we propose applying an LLM-based multi-agent framework to the legal domain to enhance fairness, depth, and quality in judicial decision making.

In this paper, we introduce a LLM-based multi-agent framework named AgentsBench to simulate the collaborative nature of a judicial bench for legal judgment prediction tasks. Our framework consists of multiple LLM-driven agents, each representing different judicial roles, who deliberate collectively to form a decision. By incorporating discussion, debate, and consensus-building among these agents, our approach better reflects the actual judicial process, thereby increasing both accuracy and transparency. The AgentsBench framework combines the capabilities of LLMs with the structured deliberation of real judicial environments, ultimately enhancing the quality and trustworthiness of AI-supported digital justice.

2. Related Work

2.1. LLM-Based Multi-Agent System

LLM-based multi-agent systems have recently shown great potential in a variety of domains [12]. By allowing multiple agents to work collaboratively, these frameworks leverage cognitive diversity and interaction to solve complex tasks, such as mathematical problem solving [9] and code generation [10], more effectively than single LLM systems [13]. In general-purpose contexts, LLM-based multi-agent frameworks have improved problem-solving capabilities by utilizing the combined reasoning and knowledge of multiple agents [14]. In specialized domains such as healthcare, systems like Agent Hospital have demonstrated the benefits of using multiple LLM-driven agents for simulating hospital environments and facilitating medical decision-making processes [15]. These systems have consistently achieved superior outcomes compared to standalone LLMs by fostering dynamic interaction and shared learning [16]. Inspired by such advancements, our work aims to explore the application of an LLM-based multi-agent framework in the legal domain, using multiple agents to simulate a collegial bench, enhancing the fairness, depth, and quality of justice decision making.

2.2. Legal Judgment Prediction

The task of legal judgment prediction is originally defined to predict the results of a legal judgment given the descriptions of a fact [17]. Earlier research focused on collecting legal case datasets for different jurisdictions and improving deep learning algorithms on them. For example, Xiao et al. [18] introduced large-scale Chinese legal data, CAIL2018, for predictive tasks involving charges, penalties, and relevant articles. Chalkidis et al. [19] built an English LJP dataset that contains cases from the European Court of Human Rights. Luo et al. [20] proposes an attention-based neural network that jointly models charge prediction and relevant article extraction. Hong and Chang [21] introduces PekoNet, a framework that integrates abstractive text summarization to improve the predictive accuracy of LJP models for colloquial case descriptions. Chien et al. [22] develops TWLJP, a dataset of indictments for judgment prediction, and improves charge prediction through interactive message passing and prompt-based learning, benefiting prosecutors in detecting discrepancies and managing legal knowledge. Recently, more work has begun to reflect on the value of the above approaches. An et al. [23] proposes a new framework to assess whether LJP models conform to legal theories, revealing gaps between the law and the techniques in existing models. Medvedeva and Mcbride [24] argued that many LJP studies do not adequately address practical needs, emphasizing the importance of explainability, user-centric approaches, and proper data usage for reliable LJP systems. While performance is important, interpretability and transparency are critical for trustworthy legal AI systems. Most existing studies lack adequate reasoning behind their predictions, which limits their practical applicability.

2.3. Large Language Models in Law

AI applications in the legal domain have seen significant advancements with the development of LLMs. LLMs are transformer-based neural networks trained on massive corpora of text to predict and generate human language [25]. LLMs show capability in tackling various legal scenarios, such as legislation, legal literacy, and justice. For instance, Liga and Robaldo [26] demonstrated that GPT-3 can effectively classify legal and deontic rules, such as obligations, permissions, and constitutive rules, even with limited data, outperforming prior models on the same task. Jiang et al. [27] presented a novel application of LLMs to improve legal literacy for non-experts through storytelling. Deroy et al. [28] explored the applicability of LLMs for summarizing lengthy and complex legal case judgments. Jiang and Yang [29] explored legal syllogism prompt engineering to improve the legal reasoning performance of LLMs. LLMs can be further trained through fine-tuning on domain-specific data, enabling them to perform specialized tasks for specific domains. Licari and Comandè [30] fine-tuned a BERT model on Italian legal data to improve NLP tasks within the Italian legal domain. Huang et al. [31] explored adapting LLMs to the legal domain via pretraining and supervised fine-tuning, while mitigating hallucination by integrating a retrieval module for relevant legal articles. While these works highlight significant progress in using LLMs for legal tasks, there remain gaps with the real-world justice system. Current LLM-based research often works on specific tasks but lacks the exploration of complexity and dynamics of justice and the legal system. Hamilton [32] trained nine separate models with the respective authored opinions of each supreme court judge, shedding light on our research. But it only gives the results of the judges’ votes and does not model the complex decision-making process.

3. Agents on the Bench

This section outlines the components of our proposed framework. AgentsBench uses LLM-based model-based agents to simulate the complex dynamics of a collegial bench in the judicial decision-making process. The framework consists of four primary stages: (i) Bench Selection, (ii) Independent Sentencing, (iii) Deliberation, and (iv) Final Decision Making. Figure 1 provides an overview of the AgentsBench process.

3.1. Bench Selection

3.1.1. Preliminary: LLM Agent

Each agent within the framework is driven by an LLM and is designed to mimic the different actors present in a typical judicial bench. Inspired by other work on agent simulation [10], we set each agent to have certain generic capabilities that empower them to operate autonomously during simulations. Specifically, the agents are capable of (1) Planning: Each agent formulates a plan that outlines the actions it will take, such as finding legal rules, reviewing case facts, or engaging with other agents’ opinions. (2) Acting: The agents execute their plans, contributing to the deliberation by presenting arguments, questioning each other, and providing insights. (3) Reflecting: After each round of deliberation, agents reflect on their contributions, learning from their observations and from the input of other agents. (4) Memory: Agents can store encountered precedents, allowing them to summarize past experiences and apply them to similar future cases.

To closely simulate a real-world collegial bench, we set up different types of agents, each fulfilling a distinct judicial role. Specifically, we designed judge agents and juror agents to replicate the collaborative judicial process. The professional judge agents are responsible for overseeing the entire decision-making process, moderating discussions, summarizing arguments, and ultimately facilitating or making a consensus-based final decision. This role is guided by a specific prompt that encourages moderation, evaluation, and adherence to legal standards. In contrast, lay judge agents act as lay persons, providing perspectives that reflect community values and societal norms. Lay judge agents actively participate in discussions, contributing to the collective decision-making process.

We use an LLM role-playing approach as a cold-start, where each agent is initialized with a role-specific prompt generated via the LLM itself. These prompts emphasize different aspects of the task. For example, lay judges are instructed to focus on ethical and societal concerns, while professional judges are instructed to follow legal rules, principles, and landmark cases. Lay judges also set up different professions, ages, personalities, and other backgrounds through the prompt to add variety. To further differentiate expertise levels over time, we implement a memory mechanism: judge agents are allowed to store and access more structured legal reasoning patterns from prior deliberations, while lay judge agents accumulate intuitive or commonsense-driven observations. This enables the judge agents to simulate legal experience learning across rounds of discussion, in contrast to the more socially grounded memory traces of the lay judges.

We chose to use general-domain large language models (LLMs) for role-playing legal agents, rather than domain-specific fine-tuned models. This decision is grounded in recent benchmark results [33], which show that general-domain models such as GPT-4 consistently outperform fine-tuned legal models in legal task benchmarks. These findings suggest that the legal knowledge and reasoning capabilities acquired through large-scale pretraining are sufficiently robust to support legal role simulation via context-based role-playing, without requiring additional supervised training.

3.1.2. Bench Selection

In this initial stage, AgentsBench forms a collegial bench by selecting agents to represent both professional judges and lay judges. The composition of the bench is carefully designed to reflect the diversity present in real-world judicial decision-making processes. To achieve this, a professional judge with extensive legal expertise is assigned as the presiding judge, providing a central figure with deep knowledge of legal standards and procedures. In addition, a diverse pool of lay judge agents is maintained, each representing different backgrounds, areas of expertise, and social perspectives. Lay judges are then randomly selected from this pool for each simulation, ensuring that the resulting bench is dynamic and reflective of the varied experiences found in real-world judicial panels. This selection process helps simulate the interplay between professional legal reasoning and the broader viewpoints contributed by lay members of the bench.

3.2. Independent Sentencing

Once the collegial bench is formed, each agent independently evaluates the case and proposes an initial sentence. During this stage, all agents—including both professional and lay judges—thoroughly review the case details, applying their unique backgrounds, perspectives, and knowledge to form an individual understanding of the situation.

Mathematically, we denote the initial sentencing decisions as

$$S^{\left(0\right)}=\{s_1^{\left(0\right)},s_2^{\left(0\right)},\dots ,s_n^{\left(0\right)}\}$$

where $s_i^{\left(0\right)}$ represents the initial sentencing decision made by agent i. Each agent i derives $s_i^{\left(0\right)}$ based on their independent analysis of the case, which can be described by the function

$$s_i^{\left(0\right)}=A_i(C,P_i)$$

Here

• C represents the case details, including the fact description and legal statutes, provided to all agents.
• $P_i$ denotes the unique personal factors of agent i, including their background, experience, and perspectives.
• $A_i$ is the analysis function used by agent i, which takes into account both C and $P_i$ to generate the initial sentencing decision $s_i^{\left(0\right)}$.

Importantly, each agent documents the rationale behind their decision, capturing their interpretation of the law as well as their subjective views on the specifics of the case. This process ensures that the diversity of perspectives among the agents is preserved, forming the set $S^{\left(0\right)}$, which serves as the foundation for the subsequent deliberation stages.

3.3. Deliberation

During each round of deliberation, all agents independently update their sentencing decisions based on the deliberation outcomes. The update function for agent i in round t + 1 can be represented as

$$s_i^{(t+1)}=U(s_i^{\left(t\right)},D^{\left(t\right)})$$

where $s_i^{\left(t\right)}$ is the sentencing decision of agent i in the previous round and $D^{\left(t\right)}$ contains the discussion content from round t. The function U prompts each agent to reconsider their initial sentencing, incorporating the new arguments and perspectives shared by others. This process allows each agent to either modify their decision or provide further justification.

After each round, the presiding judge evaluates whether the bench has reached a consensus. Instead of relying on a strict numerical threshold for agreement, the presiding judge—an LLM agent—uses its interpretative capabilities to determine whether the agents’ updated positions converge sufficiently in terms of content and rationale. This is represented by

$$C^{\left(t\right)}=\mathrm{JudgeEval}(S^{(t+1)},D^{\left(t\right)})$$

where $C^{\left(t\right)}$ is a binary indicator of consensus ($C^{\left(t\right)}=1$ if consensus is achieved, otherwise $C^{\left(t\right)}=0$). The presiding judge’s evaluation considers not only the similarity of the updated decisions $S^{(t+1)}$ but also the coherence and convergence of arguments presented during discussion $D^{\left(t\right)}$.

The deliberation proceeds iteratively through rounds until consensus is reached or the maximum allowed number of rounds is completed. This adaptive approach enables the presiding judge to assess agreement dynamically, reflecting the fluid and nuanced nature of human judicial decision making.

3.4. Decision Making

In the concluding stage, the presiding judge synthesizes the discussion outcomes to reach a final decision. This process can be described as follows. Once all rounds of deliberation are complete, the presiding judge agent analyzes the various points raised during the discussions. This analysis includes evaluating arguments, identifying recurring themes, and integrating the perspectives of both the professional and lay judges. If a consensus was reached during the deliberation, the presiding judge ratifies that consensus as the final sentencing decision. However, if significant disagreements persist, the presiding judge must weigh all contributions and utilize their expertise to determine an appropriate final judgment.

Mathematically, the final sentencing decision $S_{\mathrm{final}}$ can be represented as

$$S_{\mathrm{final}}=g(S^{\left(T\right)},D_{\mathrm{history}})$$

where

• $S^{\left(T\right)}$ represents the set of updated sentencing decisions from all agents after the final round of deliberation.
• $D_{\mathrm{history}}$ includes the complete history of discussion points raised during all deliberation rounds.
• g is the synthesis function, executed by the presiding judge to combine all available input and formulate the final sentencing decision.

Finally, the presiding judge provides a comprehensive justification for the final judgment, incorporating the insights gathered from each round of deliberation. This justification captures both the rational analysis of legal principles and the subjective viewpoints contributed by different bench members, ensuring transparency and thoroughness in the decision-making process.

The AgentsBench framework aims to capture the nuanced interplay between professional legal knowledge and diverse societal perspectives in judicial decision making. By simulating this complex process, AgentsBench provides a novel approach to studying the impact of bench composition on sentencing outcomes and the dynamics of judicial deliberations.

4. Experiment

4.1. Task and Dataset

We chose the LJP task to evaluate the ability of our framework for judicial decision making. In criminal legal judgment, the court determines the article of law, charge of crime, and prison term based on the facts of the case. Hence, LJP tasks could be divided into Legal Articles Prediction, Charge Prediction, and Prison-Term Prediction. Determining the article of law and the charge for a crime is largely based on knowledge and application of the law, leaving judges and the bench with limited room for discretion. In contrast, Prison-Term Prediction requires deciding the appropriate length of a prison term, which offers the bench significantly more discretionary space. Prison-Term Prediction is the task of predicting the potential criminal sentence of the defendant based on the given case facts and legal provisions. In civil law countries, the criminal law typically provides a range rather than a fixed term for each charge. For example, in cases of less severe intentional homicide in China, the prison term is from 3 to 10 years, specified by Article 232 of the Chinese Criminal Law.1 This broader range of discretion in sentencing allows judges to consider factors such as the severity of the crime, mitigating circumstances, and the defendant’s background, making it an ideal task to evaluate the nuanced decision-making abilities of our methodology. Hence, we select the subtask of LJP, Prison-Term Prediction, as it provides a more comprehensive assessment of the framework’s capability to support the judicial decision-making process, especially in scenarios where balancing various factors and applying judicial discretion are crucial.

The Prison-Term Prediction task can be formally defined as follows. The inputs are the facts and description of a case X with the applicable legal article l and charge c. Since prison-term prediction is the final step in criminal judgment, we assume that the articles and charges have already been determined. Therefore, they are used as inputs along with the facts of the case. The task is to predict the potential length of the prison term, represented as $y=n$. Example data can be found in the Appendix A. In the AgentsBench framework, the decision-making process involves a collegial bench composed of multiple agents, including both professional and lay judges. Each agent independently analyzes the input $(\mathbf{X},l,c)$ and proposes an initial sentence $s_i^{\left(0\right)}$. The collective deliberation process of the bench then refines these initial proposals to arrive at a final sentencing decision y.

$$y=f(\mathbf{X},l,c)$$

For our experiments, we selected evaluation data from the LawBench dataset, which is a benchmark dataset designed to assess the legal capabilities of LLMs in Chinese [33]. Specifically, LawBench includes various legal AI tasks, among which tasks 3–5 involve the Prison-Term Prediction task. The data of tasks 3–5 is based on the Chinese AI and Law Challenge dataset (CAIL2018) [18]. It is a Chinese criminal case dataset widely used for LJP research. They are all real cases collected from China Judgments Online, the official website that publishes cases and decisions from Chinese courts. Each case involves both a factual description and a legal judgment. The judgment further includes three parts: legal articles, charges, and prison terms. LawBench removes cases involving the death penalty and a life sentence in prison, and then randomly chooses 500 cases as the test dataset. To assist the LLMs perform the prison-time prediction task, it added the charge name and the full content of the applicable article at the end of the fact description when creating the test dataset as follows:

The evaluation results of numerous LLMs have been publicly shared by LawBench, such as in GPT-4, GPT-3.5, and Qwen [34], offering an objective baseline reference for our research. This enables us to benchmark our AgentsBench framework against existing models in a fair and transparent manner.

4.2. Setup

4.2.1. Baselines

We compare our framework with the following baselines to evaluate its performance:

• Standard Prompt: This baseline involves prompting LLMs to output only the prison term, without any additional contextual information or step-by-step reasoning. This approach aims to measure the basic decision-making capability of LLMs without specialized prompting strategies.
• CoT: The zero-shot Chain of Thought (CoT) method enhances reasoning by incorporating the phrase “Let’s think step by step” into the prompt. This method encourages the LLM to engage in a reasoning process before providing an answer, potentially improving the quality and accuracy of the prediction by prompting the model to articulate intermediate reasoning steps [35].
• LS: Legal syllogism (LS) prompting is a zero-shot approach that instructs the LLM to apply syllogistic reasoning to legal judgment prediction tasks. The prompt first defines the legal syllogism structure and then guides the model through applying it to the given case [29]. This approach tests the model’s ability to logically derive the outcome based on legal articles. By comparing against LS prompting, we can evaluate how effective our multi-agent deliberation process is compared to a purely logical and structured method.

Each of these baselines represents a different aspect of the capabilities of LLMs, ranging from basic prediction to advanced logical reasoning and formal legal reasoning. To the best of our knowledge, there are no other multi-agent LLM frameworks designed for LJP tasks. COT and LS are the current SOTA methods for LJP with LLMs. Therefore, we choose them as baselines for comparison. These comparisons will provide insight into the performance of our multi-agent framework relative to other established approaches in this field.

4.2.2. Implementation

We used the closed-source model GPT-4, GPT-3.5-Turbo, and the open-source Chinese model Qwen-7B for all experiments. All experiments were conducted in a zero-shot setting, meaning that the models received only the question prompts at inference time without being shown any example inputs or answers for the task beforehand. This setting evaluates the model’s ability to generalize from pretraining alone. To ensure reproducibility, we set the temperature to 0 and the top_p to 1.0 for all prediction steps. We use right truncation to the input of some cases, where the fact description exceeds the length limitation.

4.3. Evaluation

The evaluation is divided into two parts: performance evaluation and quality evaluation. The first part focuses on the quantitative metrics that assess the accuracy of predictions made by these methods, while the second part involves qualitative assessments made by automated and human evaluators to assess the legality, rationality, and morality considerations of the outcomes.

4.3.1. Performance Evaluation

The performance evaluation assesses the accuracy of the predicted outcomes generated by different methods compared to the gold labels, which are the ground-truth prison terms in the dataset. As shown in Appendix A, the gold label is the numeric prison months given by judges in real-case judgment decisions. We use the data from CAIL2018, as stated before. The dataset already has extracted prison terms from the case judgment documents, and we do not need more data pre-processing work. Since the model outputs often include extra details like reasoning and explanations, we extract the numeric prison term from the LLM output and convert it into a standard format. First, Chinese numbers are converted to Arabic numerals. Then, we extract the values before time units, month and year, converting every term into months. The ground-truth labels are also standardized to months to ensure consistency in evaluation.

We evaluate the prison term prediction task using the normalized log distance (nLog-distance) as the scoring metric. It is used to capture the continuity of sentence lengths. First, we calculate the logarithm of the absolute difference between the predicted term and the gold standard prison term. We then normalize this value to fall within the range of 0 to 1 using the following formula:

$$scor{e}_{=}{\displaystyle \frac{log\left(\right|\mathrm{predicted}\mathrm{term}-\mathrm{gold}\mathrm{label}|+1)}{log(\mathrm{maximum}\mathrm{possible}\mathrm{difference}+1)}}$$

(1)

4.3.2. Quality Evaluation

Given the intricate nature of legal decision making, it is challenging to rely solely on automated metrics for a full assessment of the framework’s capabilities. Therefore, we also conducted a human evaluation to qualitatively assess the deliberations and final decisions of these methods. For this evaluation, we employed a panel of three legal professionals who independently reviewed a random sample of 100 case outputs.

The evaluators assessed each case based on three criteria: legality, logicality, and morality. Legality is evaluated by determining whether the decisions complied with relevant legal articles and legal theory. Logicality is judged based on the coherence and logical progression of arguments provided by the method. Morality is assessed to ensure that some moral, ethical, and social factors are properly taken into account in the decision-making process. Each criterion was rated as either “True” (1) or “False” (0) to capture whether the method’s output met the expected standard, and inter-rater reliability was measured using Cohen’s kappa to evaluate agreement among evaluators.

4.4. Results

Table 1. Comparison of different methods on the Prison-Term Prediction task.

Model	Method	Performance (%)	Legality (%)	Logicality (%)	Morality (%)
Qwen	Standard Prompt	74.22
	CoT	73.32	49.3	53.6	49.2
	LS	72.82	53.2	52.5	48.9
	AgentsBench	78.25	55.4	53.4	68.7
GPT-3.5	Standard Prompt	75.13
	CoT	76.45	51.0	55.0	50.1
	LS	74.72	54.8	54.4	49.6
	AgentsBench	80.81	55.3	55.2	72.1
GPT-4	Standard Prompt	80.98
	CoT	79.76	54.1	57.3	51.5
	LS	80.28	56.8	56.6	52.2
	AgentsBench	86.33	56.5	58.2	76.2

compares different methods on the Prison-Term Prediction task across three models: Qwen, GPT-3.5, and GPT-4. The evaluation is based on performance, legality, logicality, and morality.

AgentsBench consistently outperformed the other methods, achieving the highest scores across all three models. Specifically, GPT-4 with AgentsBench reached 86.33%, significantly higher than the other methods, highlighting the strength of the LLM agent-based deliberation framework. Interestingly, the CoT and LS methods did not always yield better results compared to direct output (Standard Prompt); in some cases, these methods even led to decreased performance, which confirms the findings from other studies [36]. We believe that this is related to the specificity of prison-term prediction. Unlike charge prediction, which relies more on legal rules and logic, a prison term is not exclusively determined by law and logic. In contrast, the AgentsBench approach did not harm performance. This emphasizes the effectiveness of structured multi-agent deliberation in enhancing predictive accuracy without the risks associated with other prompting techniques. With AgentsBench, GPT-3.5 can reach the same performance level of pure GPT-4, which proves our framework could enhance the capabilities of small LLMs.

Where AgentsBench truly excelled was in morality. It achieved a score of 76.2% with GPT-4, significantly outperforming the other methods. This highlights the strength of the multi-agent deliberative approach in achieving ethically balanced outcomes, which are crucial in legal decision-making processes that require careful consideration of fairness, goodness, and social justice. This advantage reflects AgentsBench’s ability to integrate diverse perspectives to produce morally robust and balanced decisions.

In terms of legality and logicality, AgentsBench performed competitively compared to the baseline methods. While other frameworks like Legal Syllogism (LS) showed slightly higher scores in these dimensions, AgentsBench still demonstrated solid adherence to legal principles, achieving 56.5% in legality and 58.2% in logicality with GPT-4. This indicates that AgentsBench remains consistent in delivering reliable legal and logical evaluations.

4.5. Case Study

To further evaluate the performance of our AgentsBench framework, we conducted a detailed case study focused on a complex bribery and fraud case. Details of the case and the full bench deliberation content are in the Appendix A and Appendix B.

In this case, we can see the sentencing deliberation process where multiple agents, including professional judges and lay jurors, evaluated the facts of the case. Initially, each agent independently proposed a sentencing decision. The presiding judge A proposed a harsher sentence of 60 months based on the severity of Liu’s offenses, emphasizing the need for a strong deterrent. In contrast, Judge B proposed 48 months, citing mitigating factors such as the defendant’s remorse and the fact that this was a first offense. Lay juror C suggested 54 months, balancing the severity of the crime with the defendant’s remorse.

The deliberative rounds demonstrated the collaborative strength of the AgentsBench framework. During the initial round of discussions, the presiding judge summarized the various perspectives, and the agents collectively discussed key factors, including the social impact of the crime and the defendant’s repentance. After thorough debate, a consensus was reached in the second round, with all agents agreeing on a 54-month sentence. Considering the gold label is 58 months, the bench results are very close to the gold label in the dataset. This outcome illustrated how a deliberative approach facilitates nuanced decision making that takes into account diverse viewpoints while balancing legal rigor and societal considerations. The collaborative discussions led by multiple agents effectively captured the complexities of the case. The presiding judge acted as a moderator, synthesizing the input from the other agents, which enabled a more rounded assessment compared to individual agent evaluations.

The final decision balanced the need for punishment with a recognition of the defendant’s remorse and the desire to provide a rehabilitative opportunity. Legal experts rated the deliberation process highly in terms of its reasoning quality, logical consistency, and legality, as the agents applied the relevant provisions from Articles 266 and 385 of the Chinese Criminal Law appropriately and coherently. The agents’ use of statutory law, combined with careful consideration of the defendant’s background and mitigating factors, led to a final judgment that was both legally sound and ethically considerate.

5. Conclusions

In this paper, we presented AgentsBench, an LLM-based multi-agent framework that simulates judicial decision making by incorporating deliberative discussions among multiple agents. Our findings show that AgentsBench not only improves the accuracy of legal judgment predictions but also enhances fairness and ethical considerations in legal decision-making processes. Our framework is highly extensible and can be adapted to other types of legal cases and broader judicial scenarios, providing a pathway for more comprehensive and realistic applications of AI in the justice system.