Moral Judgment with a Large Language Model-Based Agent

Abstract

The ethical reasoning capability of large language models (LLMs) directly impacts their societal applicability, and enhancing this capacity is critical for developing trustworthy and secure artificial intelligence (AI) systems. The existing moral judgment methods based on LLMs rely on a single cognitive theory and lack an information aggregation and transmission mechanism, which affects the accuracy and stability of moral judgment. In this paper, we propose MoralAgent, an agentic approach that utilizes LLMs for moral judgment. First, the moral judgment process is planned based on various moral judgment theories. Second, the four dynamic prompt templates and the memory module are designed, and the moral principle is constructed to assist the analysis. Finally, the memory module is coordinated with the dynamic prompt template to optimize data transmission efficiency. This method significantly outperforms three types of traditional methods on the MoralExceptQA dataset. Compared to the two existing categories of methods based on LLMs, the F1 score of the proposed method is at least 4.13% higher, with slightly lower variance. Extensive experiments and evaluation metrics demonstrate the effectiveness of the proposed method, and sample analysis shows how the judgment process works to ensure that the results are reliable.

1. Introduction

The swift advancement of large language models (LLMs) signifies a wholly novel phase in artificial intelligence systems. The model markedly enhances the efficiency and precision of the decision-making process. However, because LLMs depend on learning from vast amounts of data, they may cause moral and ethical problems such as bias and discrimination when they capture and use harmful or sensitive information in the training data. At the same time, LLMs can learn and internalize some moral principles during training. However, they may show a mechanical adherence to these principles and a lack of flexibility. These risks may not only exacerbate social conflicts and reinforce group segregation but also have far-reaching negative impacts on values.

The primary emphasis of discourse on moral judgment is ensuring that artificial intelligence (AI) systems conform to human morality and stay under human oversight. Consequently, prior research has sought to attain specific goals such as debiasing and detoxification through the modification of model parameters, fine-tuning a limited set of parameters, or incorporating supplementary modules. These lightweight methodologies seek to regulate model behavior to align its outputs more closely with human morality [1,2,3,4]. Nevertheless, as model sizes perpetually increase, the efficacy of these methodologies has progressively diminished. Subsequently, some researchers used fully supervised fine-tuning to fine-tune the model with end-to-end instructions by manually constructing datasets that satisfy different values. These methods have markedly enhanced alignment efficacy [5,6,7,8]. While fully supervised fine-tuning has enhanced stability in LLMs, it is deficient in generalization abilities and continues to show inconsistencies with human evaluations in intricate real-world situations. To tackle these challenges, certain scholars have commenced the integration of multidisciplinary methodologies by amalgamating theories from philosophy and cognitive science to augment the moral judgment faculties of LLMs. Moral judging methods, such as MORALCoT [9] and ECMoral [10], direct LLMs through a sequence of reasoning processes to facilitate moral reasoning and assessment. Experimental findings demonstrate that these models possess a degree of moral judgment ability. Nonetheless, these methods rely on a single theory, indicating potential for enhancement in their applicability and comprehensiveness. Furthermore, the substantial volumes of produced textual data are devoid of effective management and transmission, resulting in the potential for information ambiguity, which ultimately influences their moral judgment performance.

To solve the above problems, this study proposes MoralAgent, an agentic approach that utilizes LLMs for moral judgment. The agent is composed of the large language model (LLM), memory, planning, and tools [11], and its architecture enables the LLM to have a more comprehensive analyzing ability [12,13,14,15]. The design of the intelligent body in this paper will begin by simulating the human moral judgment process to construct an automated analysis process centered around the LLM. We summarize and refine based on many human moral judgment theories and design a set of analysis plans. It includes key information extraction, preliminary analysis, multidimensional analysis, and comprehensive decision-making. Based on this four-step analysis plan, four dynamic prompt templates applicable to the LLM are constructed. The dynamic prompt template will provide details of the tasks to be performed, with the analysis results to be received. At the same time, this study organized and summed up the ethical guidelines for AI that were put forward by governments, institutions, and academic groups. It then turned them into callable moral principles that could be used to support the analysis. This strengthens the comprehensiveness of the LLM’s moral judgment.

Moreover, there is uncertainty in the information transmission of LLMs. To avoid confusion and omission of important information during the reasoning process, this method designs the analysis process as four dynamic prompt templates. These templates serve as an interactive bridge between the input and output of the LLM, running through the four-step analysis process. These can dynamically update based on the information transmitted during the reasoning process, ensuring the coherence of the analysis process and accurately conveying all important information. The complexity of moral scenarios and the diversity of analysis tasks generate a large amount of analysis data. This study constructs a memory module and collaborates with dynamic prompt templates to support LLMs in storing and summarizing important information related to analysis tasks and their generation during the analysis process. Moreover, the memory module will use the summarized information to update the next prompt template. Dynamic prompt templates and the memory module complement each other, further alleviating the issues of information confusion and loss that LLMs in agents might encounter when processing vast amounts of information, and thereby enhancing the efficiency and reliability of analysis.

• MoralAgent is proposed as an agentic approach using LLMs for moral judgment. We systematically sort and summarize many moral judgment theories and design a four-step analysis plan. In addition, dynamic prompt templates, moral principles, and the memory module are designed to support the analysis.
• An information processing method is designed. The memory module summarizes and stores the results obtained from the analysis plan. Subsequently, the variable fields in the next prompt template are automatically updated based on what is memorized. The two interact synergistically to ensure the orderly progression of each step in the analysis plan and the efficient delivery of important information.
• The paper’s experiments demonstrate that the MoralAgent method outperforms previous related methods, and the moral judgment process is demonstrated through a sample.

The remainder of this paper is organized as follows: Section 2 reviews related work, including theoretical research on human moral judgment, studies on moral judgment in LLMs, and research on LLM agents. Section 3 details our proposed method, covering problem definition, the design rationale of the MoralAgent approach, and its execution process. Section 4 introduces the Dataset, Baseline Methods, and Experimental Metrics. Section 5 presents the experimental results comparing MoralAgent with baseline methods, along with ablation studies and generalization tests. Section 6 concludes the paper and discusses its limitations as well as future research directions.

2. Related Works

2.1. Theoretical Research on Human Moral Judgment

Early studies of moral judgment focused on causal and emotionally driven analyses. The responsibility and blame model, which Shaver [16] came up with in 1985, says that moral judgments are based on causality and responsibility. Since then, researchers like Weiner [17], Cushman [18], and Knobe [19] have investigated deeper aspects of moral judgment from the points of view of emotion, behavior that can be controlled, and cause and effect.

Subsequent studies have focused on the interaction between intuition and rationality in complex moral decision-making and their theoretical models. Haidt and Hersh [20] suggested fast emotional intuition usually comes before slowing post hoc reasoning and rules over moral judgments. Greene [21], on the other hand, combined emotion and logic to explain how people make choices when they are faced with moral dilemmas. Malle et al.’s [22] pathway model of blame combines intuition and reasoning, clarifying how final blame decisions are made, and providing a framework for understanding the dynamic processes involved in moral judgment.

With the continuous development of moral judgment theories, researchers have further refined and explored new theoretical frameworks based on existing results. The Social Information Processing–Moral Decision-Making (SIP-MDM) framework by Garrigan et al. [23] is based on the Social Information Processing Theory [24]. It combines affective processes and moral domain knowledge through six steps, from cue encoding to behavior execution, to show how moral judgments are made. Van Bavel et al. [25] also said that traditional static theories should be replaced with dynamic system models that focus on the interaction among emotion, cognition, and context. This offers a new way to understand how moral cognition changes over time. Using only a single theory to guide an LLM can lead to a limited analytical perspective, which in turn limits its information processing capabilities. Therefore, it is necessary to systematically sort out and integrate multiple theories and construct the steps of moral judgment applicable to the LLM.

2.2. Research on Moral Judgment in LLMs

Early research on moral judgments in LLMs focused on parameter training, i.e., achieving the alignment of moral values without modifying or adjusting only a small number of parameters in the model. Sheng et al. [1] searched and optimized adversarial training to obtain discrete strings as triggers to be spliced into cues of a language model to reduce the model from generating discriminatory content with respect to gender, ethnicity, etc. Cheng et al. [2] trained a filtering layer on top of BERT’s output using an information bottleneck-based loss function to remove gender-related information. Similarly, Qian et al. [4] and Yang et al. [26] optimize the fairness and harmlessness of the generated content through prefix fine-tuning and information-theoretic approaches, respectively. While these methods achieved some success in small-scale models, their effectiveness diminished when applied to LLMs.

With the increasing scale of LLMs, supervised fine-tuning has gradually become a mainstream approach. Supervised fine-tuning can yield more accurate alignment results for LLMs than parameter training. To fine-tune a language model and get rid of the gender bias in the pre-training data, Lu et al. [27] created attribute values that were semantically similar but different (for example, male vs. female). Liu et al. [8] added positive and negative examples to the training data and improved the model’s ability to spot subtle value differences through comparative learning. However, such methods show limited generalization ability when dealing with unfamiliar scenes.

In recent years, researchers have begun to explore the use of prompt learning methods to enhance the performance of LLMs in tasks involving moral judgment. Ganguli et al. [28] improve the model’s performance by adding clear value constraint statements to the prompts, like “Please make sure your answer is fair.” This makes the content that is generated more in line with what is expected to happen. Sun et al. [7], through contextual learning, design manually written guidelines to help the model produce useful and harmless content. In addition, Saunders et al. [29] used a LLM to self-critique the responses generated for a given question and revise the responses again based on the problems they found. However, the analysis process and moral judgment results obtained by these methods still differ from actual human moral judgments. This highlights the importance of incorporating human moral judgments into the moral judgment mechanism.

To solve the problems listed above and get the most out of prompt learning, Jin et al. [9] first created a set of prompt methods for moral judgment in LLMs from a contractualist point of view. Wu et al. [10] tried to include emotional and mental factors in the way that LLMs decide what is right and wrong, drawing from psychology and cognitive science. Such methods still have room for improvement, as they rely solely on a theoretical framework to guide LLMs in moral judgment.

2.3. Research on LLM Agents

LLMs like GPT-4 have already demonstrated their exceptional capabilities in complex reasoning, intent recognition, and code writing. These abilities are usually considered manifestations of intelligence that appear only when the model’s parameter count reaches a certain level. Currently, numerous studies are dedicated to exploring the potential of large language models (LLMs), driving their evolution toward agent-oriented development [30,31,32,33,34]. Therefore, several agent systems such as AutoGPT, BabyAGI, AgentGPT, and HuggingGPT have emerged on the market [35]. LLMs serve as the core brain of agents, working in conjunction with multiple key components [11]. Some studies have utilized LLMs to combine foundational models with existing application architectures to accomplish tasks in both digital and physical domains. For example, ChemCrow [36] integrates 17 expert-designed tools to enhance the capabilities of LLMs in the field of chemistry. Math agents [37] use LLMs to explore, reveal, solve, and demonstrate mathematical problems. In this paper, the method of MoralAgent utilizes the advantages of agents to design a more comprehensive moral judgment process centered on an LLM. It includes callable moral principles to assist in analysis. The memory module and dynamic prompt templates work together to make sure that the analysis process runs efficiently. This improves the moral judgment abilities of LLMs in the end.

3. Methodology

Because moral judgments are complicated and flexible, traditional approaches based on LLMs only use one theory of moral judgment as a prompt template for analysis and judgment. This makes it hard to fully utilize the potential of LLMs. Also, the huge amount of text data that is generated during the analysis process does not have a good way to summarize and transfer information, which can easily cause information confusion. To solve the above problems, this paper proposes MoralAgent, an agent method for moral judgment using LLMs. The method constructs a set of moral judgment processes that simulate human beings to achieve a systematic analysis of moral scenarios. At the same time, a formatting tool is constructed to assist the analysis with a knowledge graph containing moral principles, which increases the comprehensiveness of the analysis. Additionally, we synergize the memory module and dynamic prompt templates to optimize the information transfer process, thereby ensuring the reliability of the analysis results. This section provides a detailed description of the specific structural design of each module in this methodology, as well as the interaction process between them.

3.1. Task Definition

Moral judgment belongs to a special classification task in natural language processing. By analyzing the given input scene text and behavior description, we predict whether the behavior is appropriate (the output is “Yes”) or not (the output is “No”).

3.2. Design of the MoralAgent Method

As a deep learning-based model, the LLM is trained with a large amount of human textual data to exhibit human-like cognitive abilities. There are some problems with relying only on the LLM’s abilities, though. Therefore, this study proposes a framework for agents that integrates memory, dynamic prompt templates, and external tool modules with a LLM at its core. As shown in Figure 1, this study first analyzes the basic process of human moral judgment and then identifies the analysis plan and competency needs of MoralAgent.

3.2.1. Design of the Analysis Process

Researchers have a better idea of how people make moral decisions thanks to progress in cognitive science. Out of many theories and models of moral decision-making, we chose six that best represent how people do it: the Social Intuitionist Model [20], the Causal-Intentional Model [18], the Dual Process Theory [21], the Path Model of Blame [22], the Dynamic Systems Model of Moral Cognition [25], and the SIP-MDM framework [24].

Although these theories and models differ in content, they have similarities in process. We broke the process down into four steps and combined the best parts of each theory and model to achieve the process of moral judgment shown in Figure 1. The process is as follows: Step 1: Moral Scenario Information Identification and information recognition. Step 2: Intuitive and superficial thinking. Step 3: Causal analysis, emotional cognition assessment, and analysis of purpose and intent. Step 4: Make a judgment based on all the analysis results.

MoralAgent’s analysis process is based on the human moral judgment process, resulting in a four-step analysis plan applicable to LLMs. The process is as follows: Step 1: Text retrieval and key information extraction. Step 2: Intuitive preliminary analysis. Step 3: Multidimensional analysis based on emotions, causality, perspective-taking, and moral principles. Step 4: Synthesize all analyses for final decision-making. These steps will serve as the basis for the subsequent construction of the dynamic prompt template.

3.2.2. Design of the Memory Module

The human memory capacity can effectively recall and summarize the information in the process of moral judgment, thus ensuring coherent thinking. The process of analyzing moral scenarios generates a large amount of data, and LLMs often show a tendency to confuse or forget important information when dealing with large datasets. To ensure the effective transfer of important information during the analysis, the model can continuously refer to this information throughout the runtime. Our method will design the memory module (denoted as $\mathrm{M}\mathrm{M}$). This process can be represented as:

$$\mathrm{M}\mathrm{M}=\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}(\mathrm{l}\mathrm{e}\mathrm{n}(\mathrm{I}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}))$$

(1)

$${\mathrm{M}}_{\mathrm{P}\mathrm{i}}=\mathrm{M}\mathrm{M}(\mathrm{I}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t})$$

(2)

where $\mathrm{I}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}$ includes the analysis task, the output information after the LLM performs the analysis task, and the external information that is called. $\mathrm{l}\mathrm{e}\mathrm{n}()$ represents the total number of characters for storing the information and is intended to be the storage length for creating the memory. $\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}$ is the module in LangChain that is used to create the memory module $\mathrm{M}\mathrm{M}$. ${\mathrm{M}}_{\mathrm{i}}$ denotes the information summarized by the memory, where $\mathrm{i}$ = 1, 2, 3. Firstly, the important information $\mathrm{I}\mathrm{n}\mathrm{p}\mathrm{u}\mathrm{t}$ in the analysis process is stored, and, subsequently, $\mathrm{M}\mathrm{M}$ summarizes the important information to obtain ${\mathrm{M}}_{\mathrm{i}}$ through the LLM, which provides support for the subsequent analysis tasks. The introduction of the memory module not only effectively improves the efficiency of information processing but also enhances its stability and reliability in complex tasks. It lays the foundation for the efficient operation of the analysis process.

3.2.3. Formulation of the Moral Principles

Existing human moral principles also influence the judgment process. According to cognitive development theory experts, the imperfection of judgment standards will lead to immature moral judgments [38]. Moreover, a lack of understanding of moral principles and judgment standards will make it difficult for individuals to make accurate and appropriate decisions when facing complex moral situations. So, it is important to make sure that clear moral principles are used for analysis and evaluation.

This study excerpts internationally recognized ethical and moral norms for AI proposed by governments, institutions, and academic bodies. To mitigate regional or cultural biases, we extracted the moral principles consistently addressed across these norms and structured them into a knowledge graph. This knowledge graph stores the principle information in the form of nodes, which not only cover the basic quasi-moral principles, such as fairness, justice, peace, freedom, cooperation, respect for others, and safety, but also include guiding principles that are analyzed for the needs of special situations and specific groups of people. The content of the moral principles is more concise and standardized with this structured representation. This makes calling moral principles in LLMs more efficient and accurate. This design provides a clear reference basis for an in-depth analysis of LLMs and further enhances the reasonableness of moral judgments.

3.3. Execution Process of the MoralAgent

In the information processing process of moral judgment, it is crucial to ensure the clear transmission of important information at each step. Efficient transmission of information is the basis for avoiding confusion and omission of information. To effectively address this challenge, we will design the dynamic prompt template and collaborate with the memory module as a bridge between inputs and outputs in the four-step analysis process to realize the efficient transfer of information. The dynamic prompt template provides formatted tasks with the content of the results of the tasks performed and passes this information to the memory module. Subsequently, the memory module then summarizes the important information and uses it as input for the next prompt template.

As shown in Figure 2, the four dynamic prompt templates are notated as P₁, P₂, P₃, P₄. Fixed fields are designed in the instruction information to give the specific analysis task and to prompt the LLM to start the execution. At the same time, the variable fields are designed to take over the important information in the memory module, the external information, and the results obtained by the LLM after executing the analyzing instructions in this step. The variable fields include raw text (Text), formatted templates (Json), key information (T₁), results of preliminary analysis (T₂), results of multidimensional analysis (T₃), moral principle (T_k), and the information summarized in the memory module (M_P1, M_P2, M_P3), which will be dynamically replaced according to the progress of the task. The updated command information is noted as P₁′, P₂′, P₃′, P₄′, which will be passed into the memory module for memory storage and summarization, providing important information for analysis and the decision-making of subsequent tasks. Next, the specific process will be introduced from the four parts of key information extraction, preliminary analysis, multidimensional analysis, and comprehensive decision-making.

3.3.1. Key Information Extraction

Moral scenarios are usually of high complexity, containing not only diverse subjects and their behaviors, but also influenced by the time and place of the event, as well as other contextual information. In the face of such complex textual information, the extraction of key information helps to improve the understanding of the content of that text [39,40]. Therefore, we must first systematically extract the scene’s content to ensure the accuracy of our subsequent analysis and decision-making process.

First, the LLM receives instructions from the dynamic prompt template ${\mathrm{P}}_1$ and invokes the format template as a qualification for recognizing the content and format. Then, the language comprehension capability of LLM is utilized to finally obtain the key information ${\mathrm{T}}_1$ of $\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}$. These include who, what, when, where, why, and how. By applying this extraction method, it helps to capture the core information systematically about the details and context of the event in the text. This process can be represented as:

$${\mathrm{P}}_1=(\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_1,\{\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}\},\{{\mathrm{T}}_1\},\{\mathrm{J}\mathrm{s}\mathrm{o}\mathrm{n}\})$$

(3)

$${\mathrm{T}}_1=\mathrm{L}\mathrm{L}\mathrm{M}({\mathrm{P}}_1)$$

(4)

where the fixed field $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_1$ represents the specific task of key information extraction and $\left\{\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}\right\}$, $\left\{{\mathrm{T}}_1\right\}$, $\left\{\mathrm{J}\mathrm{s}\mathrm{o}\mathrm{n}\right\}$ are all variable fields.

This kind of key information based on the strictly formatted processing preserves the integrity of the information in the subsequent transmission process. During execution, $\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}$ as well as the output key information ${\mathrm{T}}_1$ are automatically populated with the content of the dynamic prompt template ${\mathrm{P}}_1$ updated to ${\mathrm{P}}_1^{\prime }$ and passed to the memory module to be summarized to get ${\mathrm{M}}_{\mathrm{P}1}$.

3.3.2. Preliminary Analysis

In the absence of a clear analytical focus and tendency, a direct multidimensional analysis of key information could put judgments in a dilemma, thus increasing the uncertainty of the results. Simultaneously, intuition frequently influences the final outcome of human moral judgment. Therefore, it is crucial to conduct an intuitive preliminary analysis of the extracted key information. At this step, it is possible to see where the analysis is going and what the judgment results are likely to be. It is also possible to build a strong logical base for the next multidimensional analysis.

${\mathrm{M}}_{\mathrm{P}1}$ is summarized by the memory module, replacing the corresponding variable fields in the dynamic prompt template ${\mathrm{P}}_2$. The LLM receives this information and performs an intuitive preliminary analysis, ultimately generating the result ${\mathrm{T}}_2$. During this process, the information generated by the LLM will be passed to ${\mathrm{P}}_2$ as variable fields. Finally, the updated dynamic prompt template ${\mathrm{P}}_2^{\prime }$ will be passed to the memory module for storing and summarizing to get ${\mathrm{M}}_{\mathrm{P}2}$. This process can be represented as:

$${\mathrm{M}}_{\mathrm{P}1}=\mathrm{M}\mathrm{E}\mathrm{M}({\mathrm{P}}_1^{\prime })$$

(5)

$${\mathrm{P}}_2=(\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_2,\{{\mathrm{M}}_{\mathrm{P}1}\},\{{\mathrm{T}}_1\},\{{\mathrm{T}}_2\})$$

(6)

$${\mathrm{T}}_2=\mathrm{L}\mathrm{L}\mathrm{M}({\mathrm{P}}_2)$$

(7)

$${\mathrm{M}}_{\mathrm{P}2}=\mathrm{M}\mathrm{E}\mathrm{M}({\mathrm{P}}_2^{\prime })$$

(8)

where the fixed field $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_2$ indicates the specific task of the preliminary analysis and $\left\{{\mathrm{M}}_{\mathrm{P}1}\right\}$, $\left\{{\mathrm{T}}_1\right\}$, $\left\{{\mathrm{T}}_2\right\}$ are all variable fields.

3.3.3. Multidimensional Analysis

Multidimensional analysis is a necessary step to realize deep moral judgment. Its purpose is to conduct a comprehensive and detailed analysis of moral situations from different perspectives to reveal the potential motives behind behaviors. In addition, LLM invokes external moral principles to simulate the subconscious moral judgment process of human beings relying on moral principles. This process ensures the depth of moral judgment, thus avoiding one-sided conclusions.

In this phase, ${\mathrm{P}}_3$ instructs the LLM to call the moral principles ${\mathrm{T}}_{\mathrm{m}}$ from the knowledge graph and extract the important information ${\mathrm{M}}_{\mathrm{P}2}$ summarized in the memory module. Subsequently, based on this information, the LLM conducts an in-depth analysis from multiple dimensions, such as emotional, causal, moral principles, and perspective-taking aspects, and finally obtains the output ${\mathrm{T}}_3$. ${\mathrm{P}}_3$ will be automatically updated to ${\mathrm{P}}_3^{\prime }$ with the introduction of external information and the generation of analysis results, and will be deposited into the memory module again for summarization, resulting in ${\mathrm{M}}_{\mathrm{p}3}$, which will support the final decision. This process can be represented as:

$${\mathrm{P}}_3=(\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_3,\{{\mathrm{M}}_{\mathrm{P}2}\},\{{\mathrm{T}}_{\mathrm{k}}\},\{{\mathrm{T}}_3\})$$

(9)

$${\mathrm{T}}_3=\mathrm{L}\mathrm{L}\mathrm{M}({\mathrm{P}}_3)$$

(10)

$${\mathrm{M}}_{\mathrm{P}3}=\mathrm{M}\mathrm{E}\mathrm{M}({\mathrm{P}}_3^{\prime })$$

(11)

where the fixed field $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_3$ denotes the specific task of the multidimensional analysis, and $\left\{{\mathrm{M}}_{\mathrm{P}2}\right\}$, $\left\{{\mathrm{T}}_{\mathrm{k}}\right\}$, $\left\{{\mathrm{T}}_3\right\}$ are all variable fields.

3.3.4. Comprehensive Decision-Making

The aforementioned steps process the $\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}$, yielding a series of analysis results. Integration of these results with the final prompts makes a moral judgment.

The comprehensive decision-making stage will receive a summary of the information from the previous three steps ${\mathrm{M}}_{\mathrm{P}3}$. The content of ${\mathrm{P}}_4$ will also instruct the LLM to make a final judgment based on the available information, outputting the judgment result ${\mathrm{T}}_4$. The content in ${\mathrm{P}}_4$ will also be updated to provide an analysis basis for subsequent experiments. Then this process can be represented as:

$${\mathrm{P}}_4=(\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_4,\{{\mathrm{M}}_{\mathrm{P}3}\},\{{\mathrm{T}}_4\})$$

(12)

$${\mathrm{T}}_4=\mathrm{L}\mathrm{L}\mathrm{M}({\mathrm{P}}_4)$$

(13)

where the fixed field $\mathrm{R}\mathrm{e}\mathrm{g}\mathrm{u}\mathrm{l}\mathrm{a}{\mathrm{r}}_4$ represents the specific task of comprehensive decision-making, and $\left\{{\mathrm{M}}_{\mathrm{P}3}\right\}$, $\left\{{\mathrm{T}}_4\right\}$ are both variable fields.

During the moral judgment process, the memory module and dynamic prompt templates work together to speed up the transfer of information during the analysis process. This makes sure that important information is communicated correctly in difficult analysis tasks. This collaborative mechanism provides a solid guarantee for the orderly execution of the LLM in multi-task collaboration.

4. Experimental Design

In this section, we introduce the experimental dataset, baseline methodology, and experimental metrics for MoralAgent. To be consistent with the experimental setup of the SOTA model ECMoral [10], this study conducts experiments by calling the GPT-3.5-turbo-instruct model from the InstructGPT family of models via OpenAI’s API.

4.1. Introduction to the Dataset

The dataset used in this study is the MoralExceptQA [9], released by the Max Planck Institute, Stanford University, and ETH Zurich. MoralExceptQA originated from a series of recent moral psychology studies designed to investigate the flexibility of human moral cognition. The dataset scale is detailed in

Table 1. MoralExceptQA dataset Size.

Dataset	Number of Scenes
Do not cut in line (Line)	66
Do not damage others’ property (Prop)	54
Do not throw firecrackers into the pool (Cann)	28
Total	148

. It consists of three moral scenarios: (1) Do not cut in line (Line); (2) Do not damage others’ property (Prop); (3) Do not throw firecrackers into the pool (Cann.). These three moral scenarios are designed based on existing psychological survey questionnaires and represent different moral cognitive processes: (1) social learning, (2) socio-cultural evolution, and (3) individual reasoning.

4.2. Baseline Methods

To evaluate the effectiveness of MoralAgent, we introduce three categories of traditional methods and two categories of LLM-based methods for performance comparison. The comparative methods are detailed below:

1.

Simple Baseline Methods. This includes Random Baseline and Always No. Random Baseline is the result of randomly choosing in moral scenarios. Always No is the result of choosing inappropriate in all scenarios.

2.

BERT Family of Models. We reference the experimental results of BERT-base [41], BERT-large [41], RoBERTa-large [42], and ALBERT-xxlarge [43] as reported in the MORALCoT study.

3.

Delphi Family of Models [44]. Delphi is trained on 1.7 million moral judgment datasets, while Delphi++ is a model trained on top of Delphi with an additional 200,000 pieces of data.

4.

GPT Family of Models. These include GPT-3 [45], text-davinci-002 [46], and GPT-3.5-turbo-instruct. We directly ask the GPT family of models whether the behavior of the scenario is appropriate or not and obtain the answer.

5.

Chain of Thought (CoT). The CoT method [47] adds “Let’s think step by step” to the instruction to guide the model in step-by-step reasoning. MORALCoT [9], based on contractualism, evaluates whether an action violates moral principles. Self-Ask [48] prompts the model to generate and answer relevant questions sequentially before making moral judgments. The “emotion-cognition” collaborative reasoning method proposed by ECMoral [10]. Auto-ECMoral [10] is an automatic reasoning chain for “emotion-cognition” collaboration, which enables LLMs to automatically generate and answer reasoning questions under guidance.

4.3. Experimental Metrics

We followed the metrics of the SOTA method ECMoral in the literature [9]. In this study, the weighted F1 score as well as the accuracy are used as the main evaluation metrics. Meanwhile, in order to show the model performance in different scenarios, we calculate the F1 score in three types of scenarios (Line, Prop, and Cann) separately.

$$\mathrm{F}1={\displaystyle \frac{2\times \mathrm{T}\mathrm{P}}{2\times \mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(14)

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(15)

Specifically, $\mathrm{T}\mathrm{P}$ (True Positive) refers to the number of correctly identified appropriate cases, while $\mathrm{T}\mathrm{N}$ (True Negative) refers to the number of correctly identified inappropriate cases. $\mathrm{F}\mathrm{P}$ (False Positive) represents the number of cases incorrectly judged as appropriate, and $\mathrm{F}\mathrm{N}$ (False Negative) represents the number of cases incorrectly judged as inappropriate. Furthermore, the $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}$ metric represents the percentage of errors caused by rigidly adhering to moral principles and incorrectly judging a case as “inappropriate”.

$$\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}={\displaystyle \frac{\mathrm{F}\mathrm{N}}{\mathrm{F}\mathrm{N}+\mathrm{F}\mathrm{P}}}$$

(16)

To present a more detailed picture of the difference between the model’s moral judgments and human moral judgments, we compare the probability (${\mathrm{P}}_{\mathrm{moralagent},\mathrm{i}}$) of the model output being appropriate with the probability(${\mathrm{P}}_{\mathrm{human},\mathrm{i}}$) of a human choosing whether it is appropriate or not in a moral scenario. The difference between the probability of the model’s output and that of the human judgment was calculated for each problem using the mean absolute error ($\mathrm{M}\mathrm{A}\mathrm{E}$) as a measure of the model’s bias in moral judgments, and the cross-entropy ($\mathrm{C}\mathrm{E}$) between these two probability distributions was calculated.

$$\mathrm{M}\mathrm{A}\mathrm{E}={\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}\left|{\mathrm{P}}_{\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{a}\mathrm{n},\mathrm{i}}-{\mathrm{P}}_{\mathrm{moralagent},\mathrm{i}}\right|}$$

(17)

$$\mathrm{C}\mathrm{E}=-{\displaystyle \frac{1}{\mathrm{N}}}{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}\left[{\mathrm{P}}_{\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{a}\mathrm{n},\mathrm{i}}\mathrm{l}\mathrm{o}\mathrm{g}({\mathrm{P}}_{\mathrm{moralagent},\mathrm{i}})+(1-{\mathrm{P}}_{\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{a}\mathrm{n},\mathrm{i}})\mathrm{l}\mathrm{o}\mathrm{g}(1-{\mathrm{P}}_{\mathrm{moralagent},\mathrm{i}})\right]}$$

(18)

5. Analysis of Experimental Results

5.1. Comparative Experiment

As shown in

Table 2. Comparative experimental results on MoralExceptQA.

Model/Method		Overall Performance					F1 on Each Subset
		F1(↑)	Accuracy (↑)	Cons	MAE (↓)	CE (↓)	Line (↑)	Prop (↑)	Cann (↑)
Simple Baseline Methods	Random Baseline	49.37 (4.50)	48.82 (4.56)	40.08 (2.85)	0.35 (0.02)	1.00 (0.09)	44.88 (7.34)	57.55 (10.34)	48.36 (1.67)
	Always No	45.99 (0.00)	60.81 (0.00)	100.00 (0.00)	0.258 (0.00)	0.70 (0.00)	33.33 (0.00)	70.60 (0.00)	33.33 (0.00)
BERT Family of Models.	BERT-base	45.28 (6.41)	48.87 (10.52)	64.16 (21.36)	0.26 (0.02)	0.82 (0.19)	40.81 (8.93)	51.65 (22.04)	43.51 (11.12)
	BERT-large	52.49 (1.95)	56.53 (2.73)	69.61 (16.79)	0.27 (0.01)	0.71 (0.01)	42.53 (2.72)	62.46 (6.46)	45.46 (7.20)
	RoBERTa-large	23.76 (2.02)	39.64 (0.78)	0.75 (0.65)	0.30 (0.01)	0.76 (0.02)	34.96 (3.42)	6.89 (0.00)	38.32 (4.32)
	ALBERT-xxlarge	22.07 (0.00)	39.19 (0.00)	0.00 (0.00)	0.46 (0.00)	1.41 (0.04)	33.33 (0.00)	6.89 (0.00)	33.33 (0.00)
Delphi Family of Models	Delphi	48.51 (0.42)	61.26 (0.78)	97.70 (1.99)	0.42 (0.01)	2.92 (0.23)	33.33 (0.00)	70.60 (0.00)	44.29 (2.78)
	Delphi++	58.27 (0.00)	62.16 (0.00)	76.79 (0.00)	0.34 (0.00)	1.34 (0.00)	36.61 (0.00)	70.60 (0.00)	40.81 (0.00)
GPT Family of Models	GPT3	52.32 (3.14)	58.95 (3.72)	80.67 (15.50)	0.27 (0.02)	0.72 (0.03)	36.53 (3.70)	72.58 (6.01)	41.20 (7.54)
	text-davinci-002	53.94 (5.48)	64.36 (2.43)	98.52 (1.91)	0.38 (0.04)	1.59 (0.43)	42.40 (7.17)	70.00 (0.00)	50.48 (11.67)
	GPT-3.5-turbo-instruct	53.13 (6.27)	62.84 (3.41)	70.45 (10.10)	0.39 (0.03)	1.57 (0.37)	37.66 (6.12)	48.01 (3.04)	65.75 (7.32)
Chain of Thought	CoT	62.02 (4.68)	62.84 (6.02)	58.46 (17.5)	0.40 (0.02)	4.87 (0.73)	54.4 (4.30)	72.50 (11.11)	59.57 (5.07)
	MORALCoT	64.47 (5.31)	66.05 (4.43)	66.96 (2.11)	0.38 (0.02)	3.20 (0.30)	62.10 (5.13)	70.68 (5.14)	54.04 (1.43)
	Self-Ask	53.58 (2.46)	62.84 (1.23)	93.62 (1.14)	0.40 (0.02)	4.57 (0.85)	42.50 (4.26)	72.44 (2.68)	46.90 (1.20)
	ECMoral	71.98 (1.76)	72.13 (1.50)	50.16 (12.87)	0.29 (0.02)	1.78 (0.27)	66.24 (3.90)	85.56 (8.03)	53.95 (4.44)
	Auto-ECMoral	67.70 (2.14)	68.58 (2.79)	59.53 (19.75)	0.31 (0.01)	1.75 (0.37)	59.46 (2.94)	81.30 (5.69)	55.26 (4.94)
	MoralAgent	76.11 (1.60)	81.76 (1.54)	53.26 (8.87)	0.29 (0.02)	1.77 (0.29)	75.76 (3.23)	80.00 (5.78)	74.07 (3.92)

Note: “↑” indicates that the larger the value, the better the performance; “↓” indicates that the smaller the value, the better the performance. The values in parentheses represent the variance of 10 experimental results. The bold values indicate the best values.

, MoralAgent is more effective in moral judgment tasks. Its F1 score and accuracy score are higher than those of all other methods. MoralAgent introduced a formatted tool and utilized a memory module combined with a dynamic prompt module to facilitate the transmission of important information, thereby suppressing, to some extent, the uncertainty in model outputs. Therefore, this method exhibited low variance in various experiments, demonstrating good stability.

We found that methods such as the GPT family of models, CoT [47], and Self-Ask [48], which mainly rely on the intrinsic capabilities of LLMs with minimal external guidance, tend to produce moral judgments that reflect high-standard moral norms. However, this tendency also results in a lack of necessary flexibility when handling complex scenarios. In contrast, methods such as MORALCoT, ECMoral, and Auto-ECMoral improve their judgment capabilities in complex contexts by enhancing the design of their reasoning chains. However, these methods rely solely on a specific human-defined rule, lack effective control over the data transmission process, and do not consider existing moral principles. As a result, they fail to fully explore and utilize the intelligence of LLM. The MoralAgent method proposed in this paper designs a reasonable moral analysis and judgment process and collaborates with the memory modules and dynamic prompt templates to ensure efficient data transmission. Meanwhile, it leverages moral principles to enhance the comprehensiveness of the analysis. This method fully strengthens the moral judgment capability of LLMs.

To ensure more reliable experimental results, this study further increased the number of experiments in comparative analysis compared to methods in previous research. A total of 10 rounds of experiments were conducted. As shown in Figure 3, the results demonstrate that the judgment results generated by this method exhibit high stability. Specifically, the variance of the F1 score is 1.60, and no extreme values caused by the uncertainty of LLM outputs were observed. From the second round onward, the average F1 score in each round fluctuated slightly around 76. Therefore, based on a comprehensive consideration of the experimental results and resources, the final report adopts the average results of the 10 rounds of experiments as the final evaluation metric.

5.2. Ablation Study

To verify the effect of different components of the method in this paper on the experimental results, we conduct ablation experiments on the MoralExceptQA dataset. We use the F1 score as a measure, and

Table 3. Ablation study.

Ablation Component	F1	Line	Prop	Cann
MoralAgent	76.11	75.76	80.00	74.07
Ablation of Key Information Extraction	66.67	61.54	73.68	74.07
Ablation of Preliminary Analysis	73.21	73.85	76.19	69.23
Ablation of Multidimensional Analysis	59.65	59.70	55.56	62.07
Ablation of Multidimensional Analysis	61.11	56.25	66.67	69.23

Note: The bold numbers are the optimal results.

displays the results.

Ablation of Key Information Extraction. If we do not perform key information extraction on the original text at the beginning, the F1 score is 66.67%. After analysis, the extraction of key information from the original text can significantly enhance the LLM’s in-depth understanding of the text content, thus avoiding the neglect of key content due to the gradual deepening of communication in the subsequent interactive reasoning process. It can reduce the phenomenon of blurring of actors and behaviors in the judging process and, at the same time, avoid the neglect of important background information beyond the main events. However, this information is usually the key element to focus on in moral judgment.

Ablation of Preliminary Analysis. The experimental results indicate that the lack of preliminary analysis in this method, MoralAgent, will appear to have a certain degree of influence on the final judgment results, in which the F1 score decreases from 76.11% to 73.21%, and the performance in all three types of scenarios decreases. Since many moral scenarios are such that a sound judgment can be given quickly and unilaterally, a direct multidimensional analysis may lead to over-complicating the analysis and instead lead to a dilemma in judgment. This suggests that making an intuitive analysis of the moral scenario first helps subsequent analysis and judgment.

Ablation of Multidimensional Analysis. We observed that the F1 score of the ablated MoralAgent method decreased by about 16%, especially in the Prop scenario, where its F1 score was the lowest among all ablation studies. This study found that although the big language model itself has powerful data as a foundation, it still leads to too one-sided analysis results when dealing with complex situations due to the lack of necessary information such as moral principles and the single dimension of consideration. Especially when faced with rules that are shaped by social conventions and at the same time possess exceptions, the performance drops significantly. This demonstrates the importance of the multidimensional analytical capabilities of LLMs when dealing with complex information.

Ablation of Memory Module. The F1 score decreased by about 15%. In the Line scenario, the F1 score was only 56.25%. This study found that without a dedicated memory module for information storage and summarization, the large amount of information generated by the LLM during the analysis process is directly transferred to the subsequent tasks. This direct transfer of information can lead to information buildup, which can trigger the confusion of important information. Especially in the final judgment stage, the model has forgotten the important information, which leads to the instability of the output results. Therefore, to ensure the logic of the judgment process and the reliability of the final decision result, the memory module is indispensable.

5.3. Generalization Ability of the MoralAgent

The model is continuously updated iteratively, while the moral scenarios are characterized by diversity. In view of this, we will verify the generalization performance of the MoralAgent method based on these two dimensions.

Model Generalizability. The GPT-3.5-Instruct model has become the most representative model in the InstructGPT family. To verify the generalization ability of the MoralAgent method proposed in this paper, the GPT-4o model was selected for experimental analysis and compared with other methods, as it is the most utilized and widely used model in the ChatGPT family. The experimental results are shown in

Table 4. Results of MoralAgent used on GPT4o.

Method	F1	Line	Prop	Cann
GPT4o	76.64	71.43	85.71	80.00
MORALCoT	76.92	76.17	75.68	75.68
ECMoral	77.78	76.19	86.96	75.68
MoralAgent	81.16	77.65	85.71	87.50

Note: The bold numbers are the optimal results.

, where the MoralAgent method performs the best among all methods on the GPT-4o model. Its F1 score reaches 81.16%, which indicates that this method is able to deeply integrate with the current mainstream models and significantly improve its performance. In addition, this experiment also reveals that, although the performance gap between GPT-3.5-Instruct and GPT-4o is large, GPT-3.5-Instruct, based on the MoralAgent method, is close to GPT-4o in terms of moral judgment ability. This further demonstrates the effectiveness of the current method.

Scenario Generalization. The ETHICS dataset is known for its large size and wide coverage. To verify the effectiveness of the proposed method, we randomly selected 10% of the samples for prediction on the moral judgment test set with the category “commonsense” in the ETHICS dataset. GPT-3.5-turbo-instruct is used as the base model. The experimental results are shown in

Table 5. Results of MoralAgent’s use on the ETHICS dataset.

Method	F1	Accuracy
GPT-3.5-turbo-instruct	61.41	60.23
MORALCoT	51.33	57.52
ECMoral	63.89	61.17
MoralAgent	68.67	63.32

Note: The bold numbers are the optimal results.

. On the selected dataset, the MoralAgent method demonstrated superior performance compared to other methods. This result confirms that MoralAgent possesses wider applicability, thus providing a more reliable solution for ethical judgment.

5.4. Sample Analysis

To illustrate and demonstrate the method of this paper more intuitively, we select representative samples, as shown in Figure 4.

Step 1: Key Information Extraction. The LLM receives key information extraction instructions from the dynamic prompt templates and calls the formatting templates to extract the six key pieces of information: who, what, when, where, why, and how. The main points of the moral scenarios become clearer through this structured extraction process. This approach gives a solid base for further analysis and making the final decision. At this step, we update the dynamic prompt template’s content with key information. Further, these are stored in memory and summarized in a memory module to support subsequent analysis. During the subsequent analysis, it can be seen that the extracted objects and their behaviors, as well as the contextual information of the scene, are fully taken into account.

Step 2: Preliminary Analysis. The LLM extracts the information obtained earlier from memory and makes an intuitive analysis of this information to generate a brief analysis result. The results of the preliminary analysis reveal the motivation behind the maintenance worker’s behavior and give a general direction for the analysis by making basic considerations about the relevant group (customers who are standing in line) that the behavior may affect. This information will also be remembered and summarized in the memory module and passed on to the next stage as input for the multidimensional analysis.

Step 3: Multidimensional Analysis. In this stage, the big language model utilizes ethical principles and analyzes the problem comprehensively in multiple dimensions, such as sentiment analysis, causal reasoning, and transpersonal thinking. It enriches the analysis and enhances the interpretability of moral judgment. As can be seen from the results of the analysis, this stage indicates the necessity of the maintenance worker’s work, the benefits that such behavior will eventually bring, and gives a clearer judgment tendency. At the same time, the interests and feelings of the customers in the queue are considered more fully, and the idea that specific problems need to be analyzed specifically is shown. The memory module then stored and summarized these results as the foundation for the final judgment.

Step 4: Comprehensive Decision-Making. The LLM receives the information summarized into the memory module at each stage and outputs the final judgment results and reasons. The result of this moral judgment is yes, which indicates that the maintenance worker’s behavior of jumping the queue is reasonable in the scenario.

This method fully simulates human thinking in the process of moral analysis and judgment, and it generates a more comprehensive and reasonable analytical result.

6. Conclusions

6.1. Summary of This Study

To promote the sustainable development of artificial intelligence and society, and to address the deficiencies in the existing moral judgment methods based on LLMs, this paper proposes MoralAgent, an agent method for moral judgment using LLMs. This method simulates the human analysis process through a small amount of cueing information and creates moral principles to assist in the analysis. In addition, collaborative memory modules and dynamic prompt templates provide efficient control of information flow. This method provides analysis results one layer at a time by extracting key information, conducting preliminary analysis, and then performing multidimensional analysis. Finally, it gives moral judgment. The experimental results indicate that the proposed method has an obvious improvement in F1 score, accuracy, and stability, and proves the effectiveness of the proposed method on different models and datasets. This work provides valuable insights for developing more trustworthy AI systems, particularly benefiting developers building ethically aligned applications and end-users relying on AI decisions in sensitive domains. The enhanced moral reasoning capability addresses critical needs for the responsible deployment of LLMs in real-world scenarios.

6.2. Limitations and Future Directions

However, this method still has certain limitations. First, while this method employs an agent-based approach to enhance the LLM’s moral judgment capabilities via external prompt templates, memory module, and tools, the model’s inherent moral foundation itself remains unaltered by training. Secondly, while the method is designed based on contemporary advanced moral judgment theories, it is important to note that human morality itself evolves with time, and moral judgment theories will continue to evolve accordingly. Lastly, in real-time application scenarios, there remains room for improvement in the system’s response efficiency and resource utilization.

Therefore, future research could explore three key directions: First, by constructing high-quality moral dilemma datasets to directly train the inherent moral judgment abilities of LLMs, enabling the development of end-to-end moral alignment training frameworks. Second, establishing dynamic moral adaptation systems that utilize incremental learning and time-varying knowledge bases to allow models to continuously track shifts in societal ethics. Third, optimizing real-time decision-making architectures and investigating efficient inference methods to enhance system responsiveness. These directions will advance AI moral judgment systems toward greater autonomy, adaptability, and practical utility.