Differentiated Effects of Agent Diversity on Collective Decision-Making in LLM-Based Multi-Agent Delphi Systems

Abstract

Large language model (LLM)-based multi-agent systems are increasingly being explored as tools for supporting structured collective decision-making, but it remains unclear how the diversity structure of agent groups affects decision-making performance. This study examined the effects of agent diversity in a Delphi-based multi-agent system by applying Harrison and Klein’s framework of variety, separation, and disparity. An LLM-based Multi-Agent Delphi System (MADS) was implemented, and 11 agent-group conditions were constructed by manipulating personality-based variety, belief-orientation-based separation, and influence asymmetry. Each condition consisted of eight agents and was tested across four open-weight LLMs with repeated experimental runs. Performance was evaluated in terms of idea exploration extent, consensus quality, and consensus efficiency, using a human expert Delphi study as an external benchmark. The results showed that diversity had differentiated effects across performance dimensions. The maximum diversity group achieved the highest idea exploration but showed low consensus quality and efficiency, whereas the homogeneous group achieved stable consensus quality and efficiency but limited exploration. The separation diversity conditions formed Pareto-efficient alternatives, indicating balanced trade-offs between idea exploration and consensus. These findings indicate that agent diversity does not uniformly improve collective decision-making performance; instead, its appropriate structure depends on whether the task prioritizes exploration, consensus quality, or consensus efficiency.

1. Introduction

Organizations today operate in environments characterized by high levels of uncertainty and complexity, making strategic decision-making increasingly difficult for a single decision-maker. To address this challenge, organizations have adopted Group Decision Support Systems (GDSSs) to support collective decision-making [1]. Among structured group decision-making approaches, the Delphi method is widely used to facilitate consensus through iterative surveys, controlled feedback, and expert anonymity [2]. By reducing the influence of dominant individuals and mitigating groupthink, the Delphi method supports collective judgment in complex decision-making contexts [3,4,5].

Recent advances in Large Language Models (LLMs) have created new possibilities for extending structured group decision-making processes. LLMs can be designed as artificial agents capable of generating alternatives, evaluating options, summarizing opinions, and processing feedback through natural language interaction [6]. These capabilities suggest the possibility of constructing LLM-based multi-agent decision support systems that can repeatedly simulate collective deliberation at relatively low cost while enabling the exploration of multiple perspectives and the formation of consensus.

Despite this potential, LLM-based multi-agent systems face important limitations when applied to group decision-making. A key challenge in this context is algorithmic homogeneity in agent responses. Because agents generated from the same foundation model share similar training data, model weights, and probability distributions, they may converge toward similar responses unless users deliberately design them with heterogeneous attributes. Therefore, improving the quality of LLM-based group decision-making requires treating agent diversity not as a superficial difference in assigned roles, but as a structural attribute of group composition.

Existing studies on LLM agents have often implemented diversity through functional role assignments, such as programmer, planner, critic, or manager [6,7]. However, in group decision-making, diversity cannot be fully captured by functional role assignments alone. Organizational behavior research has shown that differences in personality, values, attitudes, experience, and status can influence information processing, creativity, conflict, and consensus formation [8,9]. Thus, agent diversity in LLM-based multi-agent systems should be defined and operationalized from a theoretically grounded perspective on group composition.

Another unresolved issue concerns the trade-off among idea exploration, consensus quality, and consensus efficiency. Greater agent diversity may broaden the range of generated alternatives and improve the exploration of problem spaces, but it may also increase disagreement, slow convergence, and raise computational costs. Conversely, low diversity may facilitate rapid agreement but restrict the range of ideas and reduce the quality of collective judgment. Therefore, it is necessary to examine how different structures of agent diversity affect multiple dimensions of decision-making performance rather than assuming that diversity is uniformly beneficial.

To address these issues, this study conceptualizes agent diversity as a structural variable of group composition based on Harrison and Klein’s framework. Harrison and Klein classify diversity into three dimensions—variety, separation, and disparity—depending on how member attributes are distributed within a group [10]. In this study, these three dimensions are operationalized as experimentally manipulable conditions in the design of LLM agent groups.

The purpose of this study is to empirically analyze how structural diversity in an LLM-based multi-agent Delphi system affects collective decision-making performance. Specifically, this study examines whether and how the diversity dimensions of variety, separation, and disparity produce differences in three performance dimensions: Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. Through this analysis, the study investigates whether agent diversity consistently improves decision-making outcomes or whether its effects vary depending on the type of diversity and the performance criterion being considered.

To this end, this study implements a Multi-Agent Delphi System (MADS) and conducts experiments with agent groups designed to vary in diversity structure. The variety dimension is implemented through differences in personality types based on the Big Five model [11], the separation dimension through differences in belief and value orientations based on the Competing Values Framework (CVF), and the disparity dimension through asymmetry in agent influence during the consensus process. The decision-making performance of these agent groups is evaluated using indicators of idea exploration, consensus quality, and consensus efficiency, with human expert Delphi results used as an external benchmark for evaluation.

This study contributes by theoretically structuring agent diversity in LLM-based multi-agent systems and empirically examining its effects on idea exploration, consensus quality, and consensus efficiency. It also proposes a benchmark-based approach for evaluating LLM agent group decisions against human expert Delphi results.

2. Theoretical Background and Related Work

2.1. LLM-Based Multi-Agent Systems and Algorithmic Homogeneity

The emergence of Large Language Models (LLMs) has expanded the role of artificial intelligence from language generation to reasoning, problem solving, and autonomous interaction. Recent studies have shown that LLMs can function as agents capable of perceiving their environment, planning actions, using tools, and interacting with other agents [12,13]. Based on these capabilities, LLM-based multi-agent systems have been developed to solve complex tasks through collaboration among multiple agents.

Frameworks such as CAMEL [6] and MetaGPT [14] demonstrate that multiple LLM agents can collaborate by taking on different occupational or functional roles, such as planner, programmer, reviewer, or product manager. However, these approaches often implement agent diversity mainly as a division of labor. Even when agents are assigned different role labels, they may still rely on the same foundation model, similar training data, and comparable alignment mechanisms. As a result, they may converge quickly toward similar reasoning patterns and conclusions. This issue can be understood as algorithmic homogeneity, in which apparent role differences do not necessarily produce meaningful cognitive diversity.

This limitation is important because group decision-making performance depends not only on the number of agents, but also on whether agents generate sufficiently different perspectives, assumptions, and evaluative criteria. Merely instructing an agent to “act as a critic” or “serve as an expert” may not be enough to reproduce the cognitive conflict and multi-perspective deliberation observed in human expert groups. Therefore, improving LLM-based collective decision-making requires moving beyond role-based differentiation and toward the structural design of diversity among agents.

Previous research suggests that LLMs can express heterogeneous social and cognitive characteristics when appropriately conditioned. Argyle et al. (2023), for example, showed that LLM-based agents conditioned with demographic and attitudinal attributes can approximate response patterns of specific human sub-populations [15]. This implies that LLMs may contain latent distributions of diverse human-like responses. However, the existence of latent diversity does not guarantee better decision-making. The key research question is how such diversity should be systematically designed and how it affects collective decision-making outcomes. Therefore, this study treats agent diversity not as arbitrary persona assignment, but as a theoretically structured property of group composition.

2.2. Organizational Diversity Theory and Its Application to Multi-Agent Systems

Organizational diversity refers to differences among members of a social unit in terms of objective or subjective attributes [16]. Early diversity research mainly focused on surface-level attributes such as gender, age, and race, and explained their effects through social categorization processes. From this perspective, diversity can increase in-group and out-group distinctions, reduce cohesion, and generate relational conflict, thereby negatively affecting group performance [16,17].

Later studies, however, emphasized that diversity can also improve decision quality, especially in complex and uncertain tasks. Deep-level diversity, including differences in knowledge, expertise, values, attitudes, and cognitive styles, can expand the range of information and perspectives available to a group [18,19]. From the information processing perspective, heterogeneous groups may outperform homogeneous groups when members’ diverse perspectives are actively exchanged, elaborated, and integrated into the decision-making process.

The Categorization-Elaboration Model provides a useful explanation of this dual effect. Diversity can generate social categorization and conflict, but it can also enhance performance when it promotes information elaboration [20]. Thus, diversity does not automatically improve group outcomes. Its effect depends on whether differences among members are transformed into constructive information processing rather than unproductive conflict.

This perspective is particularly relevant to LLM-based multi-agent systems. Human groups may experience relational conflict, status anxiety, or communication barriers when diversity is high. LLM agents, by contrast, do not possess emotions or social identities in the same way humans do. This creates the possibility of designing agent groups that increase cognitive diversity while reducing some of the interpersonal costs associated with human group interaction. However, this potential can only be realized when agent diversity is deliberately structured in ways that influence reasoning, evaluation, and consensus formation.

To conceptualize such diversity, we adopt Harrison and Klein’s (2007) framework as shown in Table 1, which distinguishes diversity into three dimensions: variety, separation, and disparity [10]. These dimensions define diversity not simply by the content of member attributes, but by their distributional structure within a group.

Table 1. Three Dimensions of Diversity and Their Application to LLM Agent Design.

Dimension	Definition	Application to LLM Agent Design
Variety	Differences in categorical attributes	Assigning different personality types or expertise profiles to agents
Separation	Differences in beliefs, values, or attitudes	Assigning conflicting value orientations or cognitive biases
Disparity	Unequal distribution of valued resources, status, or influence	Assigning different levels of influence or decision weight among agents

This framework provides a theoretical basis for translating organizational diversity into LLM-based multi-agent system design. Although LLM agents do not possess biological or social attributes in the same way humans do, their structural characteristics can be explicitly designed through prompts, role configurations, and influence mechanisms. Accordingly, we operationalize agent diversity through variety, separation, and disparity, and empirically examine how each dimension affects collective decision-making performance.

2.3. Delphi-Based Consensus Building and Performance Evaluation in Multi-Agent Systems

The Delphi method is a structured communication process designed to elicit expert consensus on complex problems for which uncertainty is high or no single correct answer exists [3]. Its core mechanisms include anonymity, iterative feedback, and statistical aggregation. By allowing participants to review others’ opinions anonymously and revise their judgments across multiple rounds, the Delphi method reduces social noise such as conformity pressure, halo effects, and dominance by high-status individuals [4].

Because the Delphi method is often applied to future-oriented or ill-structured problems, objective ground truth is not always available. Therefore, prior studies have evaluated Delphi performance using proxy indicators such as consensus, stability, convergence, reliability, validity, accuracy, and efficiency [21,22]. Among these, consensus and convergence capture whether expert opinions move toward agreement, while efficiency reflects the resources required to reach a stable outcome, such as time, cost, or the number of iterative Delphi rounds.

Recent studies have begun to explore the application of Delphi-like procedures to multi-agent systems. Prior to the emergence of LLM-based agents, García-Magariño et al. (2008) proposed a multi-agent-based implementation of the Delphi process, showing that structured expert consensus procedures could be computationally modeled through autonomous agents [23]. More recently, LLM-based studies such as DelphiAgent and DeLLMphi have extended this direction by using multiple LLM agents to emulate Delphi-like workflows involving expert judgment, iterative feedback, synthesis, and consensus formation [24].

In parallel, related studies on LLM-based collective judgment and multi-agent deliberation have also demonstrated the usefulness of aggregation, debate, and iterative feedback. Schoenegger et al. (2024) showed that aggregating multiple LLM forecasts can improve predictive performance and approach the performance of human crowds, describing this phenomenon as the “wisdom of the silicon crowd” [25]. Du et al. (2023) demonstrated that multi-agent debate can improve decision quality by allowing agents to identify and correct errors generated by other agents [26]. Similarly, ChatEval showed that debate among agents with different personas can produce evaluation results that align with human expert judgments [27]. Although these studies do not directly implement the Delphi method, they provide further evidence that mechanisms associated with Delphi-based consensus building, such as aggregation, iterative feedback, opinion revision, and collective evaluation, can be operationalized in LLM-based multi-agent systems.

However, existing studies have primarily focused on whether Delphi-like workflows, debate mechanisms, or multi-agent aggregation can improve task performance, rather than on how the structural composition of agent groups shapes collective decision-making. In particular, little attention has been paid to how agent diversity affects the generation, evaluation, and integration of different perspectives. Without an explicit design of diversity, multiple agents may still rely on similar assumptions, share common model biases, or converge prematurely on the same conclusion. Therefore, applying Delphi-based procedures to LLM-based multi-agent systems requires not only implementing iterative feedback and aggregation mechanisms, but also examining how the diversity structure of agents affects collective decision-making performance.

3. Methodology

3.1. Overview of the Research Design

We systematically analyze how diversity in an LLM-based multi-agent group affects collective decision-making performance. Based on the organizational diversity theory of Harrison and Klein [10], we define three dimensions of agent diversity—Variety, Separation, and Disparity—as independent variables and examine their effects on collective decision-making outcomes.

We conceptualize collective decision-making performance as a multidimensional construct rather than a single outcome indicator. Specifically, we evaluate performance in terms of three dimensions: Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. Idea Exploration Extent captures the extent to which the group explores diverse conceptual domains during problem solving. Consensus Quality refers to both the internal convergence of group opinions and their alignment with an external benchmark. Consensus Efficiency reflects the temporal cost and process efficiency involved in reaching consensus.

These performance dimensions may involve structural trade-offs. Higher diversity may broaden idea exploration but also increase disagreement, making consensus formation more difficult. Conversely, lower diversity may facilitate rapid consensus while limiting the scope of exploration. Therefore, we analyze not only whether agent diversity improves decision-making performance, but also how different diversity structures shape the balance between exploration and convergence.

To examine these relationships, we construct agent groups in which the three diversity dimensions are systematically manipulated. Each group performs the same decision-making task through a modified Delphi procedure consisting of an initial idea-generation stage and subsequent consensus-building stages. In Round 1, agents generate potential opportunity and challenge factors through an open-ended survey. From Round 2 onward, agents conduct closed-ended evaluations, receive feedback on group response distributions, and revise their judgments iteratively. This design enables comparison of both final decision-making outcomes and convergence patterns across agent groups with different diversity structures.

3.2. Benchmark Study

We use the Delphi study by Bayor et al. (2025) as an external benchmark for evaluating the decision-making performance of the agent groups [28]. Their study examined the potential opportunities and challenges associated with the integration of artificial intelligence (AI) into the talent acquisition (TA) process using a Delphi method. The expert panel consisted of eight C-level executives and TA leaders with extensive industry experience, and the study produced structured consensus outcomes through iterative surveys and feedback.

We do not treat this benchmark as an absolute ground truth. Rather, we use it as a reference point that represents the judgment structure of a human expert group in a comparable decision-making task. Accordingly, the benchmark comparison is not intended to determine whether the agent groups produce a correct answer. Instead, it evaluates the extent to which their outcomes are semantically and structurally aligned with the consensus judgments of human experts. In this study, this alignment is quantitatively captured through External Consensus Quality (ECQ). Table 2 summarizes the Delphi experiment conducted in the benchmark study.

Table 2. Summary of the Delphi Experiment in the Benchmark Study.

Category	Description
Evaluation items	A total of 45 items derived from the open-ended survey in Round 1, consisting of 26 opportunity factors and 19 challenge factors
Evaluation dimensions	For opportunity factors, Benefit and Feasibility were used as evaluation dimensions. For challenge factors, Likelihood and Impact were used as evaluation dimensions. All evaluations were quantified using a 5-point Likert scale
Consensus process	Opinions were collected over three rounds, and a standard Delphi convergence process was conducted in which differences among experts decreased as the rounds progressed

3.3. Agent Diversity Design

3.3.1. Variety Diversity: Personality-Based Diversity

Variety refers to categorical differences among members, which arise from differences in the “types” of individual-level backgrounds such as personality, knowledge, and role [10]. In this study, the Big Five model from psychology was used to implement diversity in the variety dimension to secure diversity in the linguistic patterns and thinking styles of the texts generated by agents. The model established by McCrae and John (1992) explains individual behavioral patterns through five factors: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism [11]. When different Big Five personality profiles are assigned to LLM agents, qualitative differences may occur in expression style, risk-taking tendency, and information-processing style even when they respond to the same agenda. This can function as a factor that expands the scope of information exploration by providing diverse behavioral repertoires within the agent group [29]. In this study, the Blau Index, which is used in ecology and the social sciences to measure categorical diversity, was used to quantify the level of heterogeneity in the agent group in order to quantitatively control the distribution of personality diversity among agents. A Blau Index value closer to 1 indicates a state in which group members are more evenly distributed across multiple categories, meaning that diversity is high. In this study, when agent personality is defined as $k$ discrete categories, the diversity index of the variety dimension for the entire multi-agent system, Variety, is defined as shown in Equation (1).

$$Variety=1-{\displaystyle \sum _{i=1}^k}{p}_i^2$$

(1)

where $p_i$ denotes the proportion of agents belonging to the $i$-th personality type, and $k$ denotes the total number of personality types. A Blau Index value of 0 indicates a perfectly homogeneous group. In this study, values between 0 and 0.3 are interpreted as low diversity, values between 0.3 and 0.6 as moderate diversity, and values of 0.6 or above as high diversity. A value of 1 indicates the highest level of diversity, in which all categories are equally represented.

To operationalize variety diversity, we mapped the five Big Five personality traits onto five agent personas. These personas were designed to induce differences in response style, attention focus, and discussion behavior, rather than to represent fixed psychological traits of the agents. For example, the Visionary persona emphasizes creative and unconventional alternatives, whereas the Architect persona emphasizes logical consistency, evidence, and implementation feasibility. This persona-based design was intended to increase cognitive and behavioral heterogeneity in the discussion process [29]. Table 3 summarizes the personality-based agent personas used to implement variety diversity.

Table 3. Personality-Based Agent Persona Design for Implementing Variety Diversity.

Dominant Trait	Agent Persona	Intended Behavioral Orientation
Openness	Visionary	Generates creative and unconventional ideas; expands the range of alternatives
Conscientiousness	Architect	Emphasizes logical consistency, evidence, and implementation feasibility
Extraversion	Driver	Actively leads the discussion and maintains conversational momentum
Agreeableness	Mediator	Identifies common ground and supports consensus formation
Neuroticism	Sentinel	Detects risks, uncertainty, and potential failure scenarios

3.3.2. Separation Diversity: Cognitive Bias-Based Diversity

Separation refers to horizontal differences or disagreement among members’ values, beliefs, or positions [10]. To operationalize separation diversity in the LLM-based multi-agent setting, we assign agents different initial belief orientations. These belief orientations are designed to prevent uniform responses and to expand the range of perspectives considered during the discussion process.

We draw on the Competing Values Framework (CVF) proposed by Cameron and Quinn (2011) to design these belief orientations [30]. The CVF classifies organizational values into four broad orientations: efficiency and performance, safety and stability, fairness and ethics, and innovation and change. These orientations can be interpreted as distinct value-based lenses through which agents evaluate the same decision-making problem.

Although Harrison and Klein conceptualize separation diversity as distance along a continuum of values, applying a continuous value scale directly to LLM agents may not reliably produce meaningful behavioral differences. For example, subtle numerical differences in assigned belief scores may not be consistently reflected in agent responses. Therefore, to enhance the internal validity of the experimental manipulation and improve the semantic clarity of agent personas, we adopt a categorical approach based on four CVF-derived belief archetypes. Table 4 summarizes the belief-oriented agent personas used to implement separation diversity.

Table 4. CVF-Based Agent Persona Design for Implementing Separation Diversity.

CVF-Based Orientation	Agent Persona	Intended Behavioral Orientation
Efficiency–Performance	Pragmatist	Prioritizes feasibility, efficiency, measurable outcomes, and implementation practicality.
Safety–Stability	Guardian	Emphasizes risk minimization, stability, safeguards, and potential failure modes.
Fairness–Ethics	Advocate	Focuses on ethical legitimacy, fairness, accountability, and stakeholder impact.
Innovation–Change	Innovator	Emphasizes novelty, long-term value, alternative exploration, and strategic change.

The diversity index for the separation dimension is calculated using the Blau Index, following the same logic used for variety diversity. When agents’ belief orientations are defined as $k$ discrete categories, separation diversity is calculated as shown in Equation (2).

$$Separation=1-{\displaystyle \sum _{i=1}^k}{p}_i^2$$

(2)

where $p_i$ denotes the proportion of agents belonging to the $i$-th belief-orientation category, and $k$ denotes the total number of belief-orientation categories. A higher value indicates that agents are more evenly distributed across multiple belief orientations.

3.3.3. Disparity Diversity: Influence Asymmetry

Disparity refers to inequality in power, status, or access to valuable resources within a group [10]. In group decision-making, such inequality may affect whose opinions are more strongly reflected in the final outcome. Prior research suggests that members with greater expertise often exert stronger influence, particularly when the decision-making task is complex [31]. Based on this perspective, this study operationalizes disparity diversity in the LLM-based multi-agent system as the asymmetry of influence exercised by agents during the consensus process.

Unlike variety and separation diversity, which are expected to broaden the range of perspectives and values represented in the group, disparity diversity concerns the vertical distribution of influence among agents. A high-disparity structure may allow the group to reach consensus more quickly because the opinions of highly influential agents can guide the decision-making process. However, it may also reduce decision quality if the final outcome is disproportionately shaped by a small number of agents and fails to sufficiently reflect diverse perspectives. Therefore, this study treats disparity as a distinct diversity dimension that may improve consensus efficiency while potentially constraining collective judgment quality.

To quantify influence asymmetry, this study calculates an Influence Score ($IS$) for each agent in each Delphi round. The Influence Score combines two components: the agent’s pre-assigned Competence Weight ($CW$) and its Round Average Consistency Score ($\overline{CS}$). Competence Weight represents the role-based expertise assigned to the agent through the system prompt. In this study, agents are assigned one of three role levels: Leader, Manager, or Junior Staff. These roles are given different weights to reflect differences in potential influence. Specifically, Leader, Manager, and Junior Staff are assigned weights of 1.0, 0.5, and 0.1, respectively. Therefore, the competence weight was assigned according to the agent’s role type, as shown in Equation (3).

$$C{W}_i=\{\begin{array}{ll}1.0,& \mathit{if} Rol{e}_i=\mathit{Leader}\\ 0.5,& \mathit{if} Rol{e}_i=\mathit{Manager}\\ 0.1,& \mathit{if} Rol{e}_i=\mathit{Junior}\mathit{Staff}\end{array}$$

(3)

where $C{W}_i$ denotes the competence weight assigned to agent $i$, and $Rol{e}_i$ denotes the role type assigned to agent $i$ through the system prompt.

The Round Average Consistency Score reflects the stability of the agent’s responses in a given round, calculated based on the degree of consistency among multiple responses generated for the same evaluation items. When an LLM agent has sufficient task-relevant knowledge or confidence, it is likely to produce relatively stable reasoning paths and conclusions across repeated responses. Conversely, when the agent lacks confidence or enters a hallucination-prone state, response variability tends to increase [32]. Therefore, this study uses response consistency as a proxy for the agent’s demonstrated response stability in a given round, rather than as a direct measure of expertise.

The Influence Score of agent $i$ in round $t$ is calculated as shown in Equation (4).

$$I{S}_{i,t}=C{W}_i\times {\overline{CS}}_{i,t}$$

(4)

where $i$ denotes an individual agent, $t$ denotes the Delphi round, $C{W}_i$ is the competence weight assigned to agent $i$, and ${\overline{CS}}_{i,t}$ is the round-average consistency score of agent $i$ in round $t$.

This formulation allows the model to reflect both the agent’s assigned expertise and its demonstrated response stability. Even if an agent is assigned a high competence weight, its actual influence decreases when its responses are inconsistent. Conversely, an agent with a lower competence weight may still contribute meaningfully to the consensus process when it provides stable and consistent responses.

After calculating the Influence Score for each agent, this study measures the Disparity Score ($DS$) of the agent group using the coefficient of variation in the Influence Scores in each round. In Equation (5), ${\mu }_{IS,t}$ and ${\sigma }_{IS,t}$ represent the mean and standard deviation of the agents’ Influence Scores in round $t$, respectively. A higher $D{S}_t$ indicates stronger concentration of influence among specific agents.

$$D{S}_t={\displaystyle \frac{{\sigma }_{IS,t}}{{\mu }_{IS,t}}}$$

(5)

Finally, to evaluate the overall level of influence asymmetry across the entire Delphi process, this study calculates the Average Disparity Score ($ADS$) by averaging the Disparity Scores across all Delphi rounds. Let $M$ denote the total number of Delphi rounds. The $ADS$ is calculated as shown in Equation (6).

$$ADS={\displaystyle \frac{1}{M}}{\displaystyle \sum _{t=1}^M}D{S}_t$$

(6)

The ADS is used as the final indicator of disparity diversity in this study. Through this measure, the study compares how different degrees of hierarchical influence within agent groups affect consensus quality and consensus efficiency.

3.4. Construction of Agent Group Diversity Conditions

In order to systematically analyze how agent group diversity affects collective decision-making performance, this study designed multiple agent group composition conditions based on the three diversity dimensions of variety, separation, and disparity. In this study, the results of the human expert benchmark study were used to calculate the metrics for evaluating the collective decision-making performance of the agent groups. Given that the expert panel in the benchmark study consisted of eight participants, each experimental group in this study was also composed of eight expert agents. This design was intended to ensure consistency in the conditions under which the benchmark-based evaluation metrics were calculated and to control for the potential effects of differences in the number of agents within a group on idea generation and consensus formation. The homogeneous group (H) served as the reference condition, while Groups A, B, and C were designed to gradually increase variety, separation, and disparity diversity, respectively. In addition, the maximum mixed group (D) combined high levels of all three diversity dimensions. This design allows comparison of both the independent effects of each diversity dimension and the combined effects that emerge when multiple dimensions operate simultaneously. Table 5 summarizes the agent group diversity conditions defined in this study.

Table 5. Summary of Experimental Group Composition.

Condition	Purpose	Variety (Blau Index)	Separation (Blau Index)	Hierarchy Composition (Leader:Manager:Junior)
H	Homogeneous reference group	0.00	0.00	0:8:0
A-1/A-2/A-3	Gradual increase in variety diversity	0.32/0.66/0.80	0.00	0:8:0
B-1/B-2/B-3	Gradual increase in separation diversity	0.00	0.32/0.66/0.74	0:8:0
C-1/C-2/C-3	Gradual increase in disparity diversity	0.00	0.00	1:6:1/2:2:4/1:0:7
D	Maximum mixed diversity condition	0.80	0.74	1:0:7

3.5. Evaluation Metrics

In this study, in order to evaluate the effects of agent group diversity on decision-making performance in a multi-agent group, performance was divided into three dimensions: Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. Each performance dimension is measured not by a single indicator but as a composite indicator that combines multiple sub-metrics. Table 6 summarizes the sub-metrics for each evaluation dimension.

Table 6. Evaluation Metrics by Performance Dimension.

Evaluation Dimension	Metrics
Idea Exploration Extent	Semantic Diversity (SeD), Agenda Coverage (AC), Idea Validity (IV)
Consensus Quality	Internal Consensus Quality (ICQ), External Consensus Quality (ECQ)
Consensus Efficiency	Convergence Speed (CoS), Initial Consensus Rate (ICR), Variance Reduction Rate (VRR)

3.5.1. Idea Exploration Extent

Idea Exploration Extent is an indicator of how extensively the ideas generated in the open-ended round cover diverse semantic domains. In this study, it is measured through three sub-metrics: Semantic Diversity (SeD), Agenda Coverage (AC), and Idea Validity (IV).

First, Semantic Diversity ($SeD$) is defined as the average semantic distance among generated ideas. It indicates how conceptually dispersed the ideas proposed by the agents are. To measure this, each idea is transformed into a sentence embedding vector, and the cosine distance between all pairs of ideas is calculated. Semantic Diversity is defined as shown in Equation (7).

$$SeD={\displaystyle \frac{2}{N(N-1)}}\sum _{i<j}\left(1-\mathrm{c}\mathrm{o}\mathrm{s}({\mathit{e}}_i,{\mathit{e}}_j)\right)$$

(7)

where ${\mathit{e}}_i$ and ${\mathit{e}}_j$ denote the embedding vectors of ideas $i$ and $j$, respectively, and $N$ denotes the total number of generated ideas. The term ${\displaystyle \frac{2}{N(N-1)}}$ is a normalization coefficient used to average the pairwise cosine distances across all non-duplicated idea pairs, where the number of possible pairs is $\left({\displaystyle \genfrac{}{}{0pt}{}{N}{2}}\right)={\displaystyle \frac{N(N-1)}{2}}$.

A higher $SeD$ value indicates that the group explored more diverse conceptual domains. Because the indicator is calculated as an average pairwise distance, it reduces the simple cumulative effect of generating more ideas and instead captures the conceptual dispersion of the idea set.

Second, Agenda Coverage ($AC$) measures how comprehensively the agent group reflects the agendas presented in the human expert benchmark. To calculate this indicator, this study constructs a cosine similarity matrix between the benchmark item set ($\mathcal{B}$) and the agent-generated idea set ($\mathcal{A}$). One-to-one greedy matching is then performed by sequentially selecting the item pairs with the highest similarity. Once an item is matched, it is removed from subsequent matching candidates, ensuring that each item is matched at most once. Among the matched pairs, those with a cosine similarity above the predefined threshold (${\tau }_{sim}\ge 0.7$) are defined as Valid Hits. Agenda Coverage is calculated as shown in Equation (8).

$$AC={\displaystyle \frac{\left|\mathrm{Valid}\mathrm{Hits}\right|}{\left|\mathcal{B}\right|}}$$

(8)

where $\mathcal{B}$ denotes the benchmark item set, and $\mathcal{A}$ denotes the agent-generated idea set. $\left|\mathrm{Valid}\mathrm{Hits}\right|$ denotes the number of benchmark items judged to be valid matches, and $\left|\mathcal{B}\right|$ denotes the total number of benchmark items.

A higher $AC$ value indicates that the agent group covered a larger proportion of the agendas identified by the human expert benchmark.

Third, Idea Validity ($IV$) measures the proportion of agent-generated ideas that semantically correspond to benchmark items. While Agenda Coverage evaluates how much of the benchmark agenda is covered by the agent group, Idea Validity evaluates how many of the generated ideas are aligned with meaningful benchmark agendas. Based on the same one-to-one matching results used for Agenda Coverage, IV is calculated as shown in Equation (9).

$$IV={\displaystyle \frac{\left|\mathrm{Valid}\mathrm{Hits}\right|}{\left|\mathcal{A}\right|}}$$

(9)

where $\mathcal{A}$ denotes the agent-generated idea set, and $\left|\mathcal{A}\right|$ denotes the total number of generated ideas. $\left|\mathrm{Valid}\mathrm{Hits}\right|$ denotes the number of generated ideas that are validly matched to benchmark items.

A higher $IV$ value indicates that a larger proportion of the ideas generated by the agent group correspond to meaningful benchmark agendas.

3.5.2. Consensus Quality

Consensus Quality refers to the extent to which the final decision-making outcome of the agent group converges stably and aligns with an external reference. In this study, Consensus Quality is conceptualized as a multidimensional performance dimension that includes both internal convergence among agents and external alignment with the human expert benchmark. Accordingly, Consensus Quality is measured using two indicators: Internal Consensus Quality (ICQ) and External Consensus Quality (ECQ).

First, Internal Consensus Quality ($ICQ$) measures the extent to which judgments among agents converge in the final round. Following the evaluation structure of the benchmark study by Bayor et al. (2025), this study defines consensus as being achieved only when each item satisfies the consensus criteria on two evaluation axes simultaneously [28]. Specifically, for opportunity factors, consensus is judged to be achieved when the standard deviations for both Benefit and Feasibility satisfy the predefined threshold ($\sigma \le 0.5$). For challenge factors, consensus is judged to be achieved when the standard deviations for both Impact and Likelihood satisfy the same threshold. Based on this criterion, ICQ is calculated as the proportion of items satisfying the consensus criteria among all evaluation items, as shown in Equation (10).

$$ICQ={\displaystyle \frac{N_{consensus}}{N_{total}}}$$

(10)

where $N_{consensus}$ denotes the number of items satisfying the multidimensional consensus criteria in the final round, and $N_{total}$ denotes the total number of evaluation items.

A higher $ICQ$ value indicates that the agent group reached stable consensus across a larger proportion of the evaluation items.

Next, External Consensus Quality ($ECQ$) evaluates how closely the final judgment of the agent group aligns with the human expert benchmark. In research on LLM-based text generation and evaluation, alignment with human evaluation is commonly used as an important criterion for validating model outputs [33,34]. In this study, ECQ is measured using Spearman’s rank correlation coefficient ($\rho$), which captures the degree of agreement between the item rankings produced by each agent group and those derived from the human expert benchmark. Spearman’s $\rho$ is calculated as shown in Equation (11).

$$\rho =1-{\displaystyle \frac{6{\displaystyle {\sum }_{i=1}^n}{d}_i^2}{n\left(n^2-1\right)}}$$

(11)

where $n$ denotes the number of items being compared, and $d_i$ denotes the rank difference between the agent group result and the human expert benchmark result for item $i$.

Since Spearman’s $\rho$ ranges from −1 to 1, this study normalizes the coefficient to the range between 0 and 1 to define ECQ, as shown in Equation (12).

$$ECQ={\displaystyle \frac{\rho +1}{2}}$$

(12)

A higher $ECQ$ value indicates stronger alignment between the final judgment ranking of the agent group and that of the human expert group. Specifically, a value closer to 1 indicates high alignment, a value closer to 0 indicates opposite ranking tendencies, and a value of 0.5 indicates no meaningful rank alignment.

Overall, Consensus Quality in this study captures both the stability of internal convergence and the validity of the final judgment relative to an external expert benchmark. This allows the study to evaluate not only whether the agent group reaches agreement, but also whether the resulting consensus is aligned with human expert judgments.

3.5.3. Consensus Efficiency

Consensus Efficiency captures the temporal cost of reaching consensus and the efficiency with which disagreement is resolved during the Delphi process. In this study, we measure Consensus Efficiency using three indicators: Convergence Speed ($CoS$), Initial Consensus Rate ($ICR$), and Variance Reduction Rate ($VRR$). These indicators respectively capture the number of rounds required for consensus, the extent of initial agreement before feedback, and the degree to which disagreement decreases through iterative feedback.

The termination condition is defined as the point at which all closed-ended survey items simultaneously satisfy the multidimensional consensus criteria. If any item exceeds the predefined variance threshold on either of the two evaluation axes, consensus is regarded as not yet established, and the item proceeds to the next feedback round. To prevent excessive iteration, the maximum number of closed-ended evaluation rounds is limited to three.

First, Convergence Speed (CoS) measures how rapidly an agent group reaches the final consensus state. In this study, CoS is operationalized as the final round of the Delphi process. When all decision items satisfy the predefined consensus criteria before reaching the maximum number of rounds, CoS corresponds to the round in which consensus is achieved. Otherwise, CoS is assigned the predefined maximum round. Thus, a smaller CoS value indicates greater consensus efficiency.

Second, Initial Consensus Rate ($ICR$) refers to the proportion of items that satisfy the consensus criteria in the first closed-ended evaluation round, before any Delphi feedback is provided. Let $N_{initial}$ denote the number of items satisfying the consensus criteria in the first closed-ended evaluation round, and let $N$ denote the total number of evaluation items. $ICR$ is calculated as shown in Equation (13).

$$ICR={\displaystyle \frac{N_{initial}}{N}}\times 100$$

(13)

A higher $ICR$ value indicates that the group reached agreement on a larger proportion of items without requiring iterative feedback.

Third, Variance Reduction Rate ($VRR$) measures how effectively opinion variance decreases through the Delphi feedback process. Whereas $ICR$ captures initial agreement, $VRR$ focuses on the efficiency of disagreement adjustment among items that require additional feedback. For each evaluation item and evaluation axis, the variance reduction rate is calculated by comparing the variance in the first closed-ended round with the variance in the final round, as shown in Equation (14).

$$VR{R}_{i,k}={\displaystyle \frac{{\sigma }_{i,k,R_2}^2-{\sigma }_{i,k,R_{final}}^2}{{\sigma }_{i,k,R_2}^2}}$$

(14)

where $i$ indexes evaluation items, and $k$ indexes the evaluation axis. In this study, each item has two evaluation axes: Benefit and Feasibility for opportunity factors, and Impact and Likelihood for challenge factors. ${\sigma }_{i,k,R_2}^2$ denotes the score variance for item $i$ on axis $k$ in Round 2, the first closed-ended evaluation round. ${\sigma }_{i,k,R_{final}}^2$ denotes the corresponding variance in the final round.

A $VR{R}_{i,k}$ value closer to 1 indicates that the initial disagreement on the corresponding item-axis pair has been substantially reduced. A value close to 0 indicates little reduction in disagreement, while a negative value indicates that variance increased during the feedback process.

Because each item is evaluated on two axes, the item-level variance reduction rate is calculated by averaging the two axis-level values, as shown in Equation (15).

$$VR{R}_i={\displaystyle \frac{VR{R}_{i,1}+VR{R}_{i,2}}{2}}$$

(15)

This item-level aggregation prevents overestimating convergence when disagreement decreases on only one evaluation axis. Finally, the group-level ${VRR}_{final}$ is calculated by averaging the item-level $VR{R}_i$ values across the set of conflict items, as shown in Equation (16).

$$VR{R}_{final}={\displaystyle \frac{1}{N_{conflict}}}{\displaystyle \sum _{i=1}^{N_{conflict}}}VR{R}_i$$

(16)

where $N_{conflict}$ denotes the number of items that did not satisfy the consensus criteria in the first closed-ended evaluation round and therefore entered the iterative feedback process.

In this study, $VRR$ is calculated only for conflict items, excluding items that already satisfied the consensus criteria in the initial closed-ended round. This prevents the feedback-based variance reduction effect from being underestimated by items that did not require disagreement adjustment.

Overall, Consensus Efficiency captures not only whether consensus is reached, but also how quickly and effectively the agent group moves toward consensus. This enables comparison of how different agent group compositions affect the efficiency of the consensus-building process.

4. System Implementation

4.1. Implementation Overview

We implemented an LLM-based Multi-Agent Delphi System (MADS) to analyze how diversity in multi-agent groups affects decision-making. MADS is designed not merely to call multiple LLM agents in parallel and collect their responses, but to reproduce the core procedures of the Delphi method in software, including iterative opinion convergence, controlled feedback, and statistical consensus judgment.

In particular, we focused on embedding the structural diversity of agent groups as a designable variable to address the problem of algorithmic homogeneity that can arise in existing LLM-based multi-agent systems. To this end, we incorporated the three dimensions of organizational diversity theory—variety, separation, and disparity—into the agent generation and response mechanisms.

To operationalize this design, MADS incorporates three core implementation mechanisms: agent generation controlled by variety and separation conditions, dynamic operationalization of disparity diversity, and statistical consensus judgment. For agent generation, the Blau Index is used to construct groups whose personality and belief distributions approximate predefined target values. To operationalize disparity dynamically, MADS calculates a Consistency Score from repeated responses to the same item and combines it with each agent’s Competence Weight, allowing disparity diversity to vary according to the reliability and quality of agent responses. Finally, consensus is assessed using weighted means and weighted standard deviations, and the process terminates when dispersion falls below a predefined threshold.

In this way, MADS integrates diversity-controlled group composition, dynamic disparity modeling, and statistical consensus judgment into a unified decision-making pipeline. This enables reproducible multi-agent Delphi experiments. Rather than functioning simply as a system for aggregating LLM responses, MADS models the collective decision-making process itself and provides the basis for the experiments conducted in this study.

4.2. System Architecture

MADS consists of four modules: an experiment control module, an expert agent generation module, a mediation and consensus-building module, and a data management module. Each module performs an independent function, while the overall system is designed to process the execution, storage, and consensus judgment of multi-agent Delphi experiments as a unified pipeline. Figure 1 illustrates the overall architecture and execution flow of MADS.

Figure 1. System architecture of the MADS (Multi-Agent Delphi System).

The experiment control module initializes the overall experimental context and controls the order of interactions across rounds. This module loads key experimental hyperparameters, such as the maximum number of rounds, consensus threshold, Likert scale range, number of agents, and target diversity values, and passes them to the relevant modules. This enables experiments to be conducted in a consistent manner. It also fixes the procedural sequence of the open-ended survey, closed-ended survey, mediation, closed-ended re-evaluation, and final consensus derivation. As a result, procedural consistency is maintained across different experimental conditions.

The expert agent generation module creates multiple expert agents by reflecting the variety, separation, and disparity conditions defined in Section 3. Each agent is defined based on a persona that includes attributes such as personality, belief, professional background, role, response generation parameters, and initial expertise weight. Table 7 presents the persona attributes assigned to expert agents.

Table 7. Agent Persona Configuration.

Component	Description	Type of Diversity
Profession	Expert role and industry	Disparity
Background	Years of experience and domain expertise	Disparity
Belief (Goal)	Core value driving decisions	Separation
Personality (Method)	Thinking and communication style	Variety
Competence Weight	Influence factor in aggregation	Disparity

The mediation and consensus-building module receives responses generated by expert agents through a moderator agent, calculates the mean and standard deviation for each item, and, when consensus has not been reached, provides this information to expert agents to support convergence. It also determines whether consensus has been formed by comparing the results with the predefined consensus threshold.

The data management module stores various types of data generated throughout the experiment and supports subsequent analysis. By storing agent-level responses, round-level statistics, consensus outcomes, and changes in influence in a hierarchical structure, this module ensures the traceability and reproducibility of experimental results.

The overall execution flow of the system can be summarized as follows. First, the experiment control module sets the initial conditions. Then, the agent generation module constructs a group of expert agents that reflects the specified diversity condition. Next, ideas are generated in the open-ended round and stored in the database. In the closed-ended rounds, opinions converge through repeated evaluation and feedback. Finally, when the consensus judgment criterion is satisfied, the experiment terminates and the results of the closed-ended rounds are stored in the database.

4.3. LLM Integration and Prompting Strategy

The LLM integration and prompting strategy is a core implementation element that operationalizes agent diversity and the consensus formation process. In multi-agent experiments using the same foundation model, agent responses may converge toward similar distributions if no separate conditioning is applied. Therefore, even when the same foundation model is used, we designed prompts hierarchically so that each agent would behave as if it had a distinct perspective, value criterion, and level of expertise.

To reduce dependence on a specific LLM and mitigate biases associated with the characteristics of a single model, we conducted experiments using multiple open-weight language models from different model families and parameter scales. Model selection considered both computational feasibility and reported response-generation performance. Given the computing resources required for repeated multi-agent experiments, the candidate pool was limited to recently released models in the approximately 120B-or-smaller range. Within this range, models reported to demonstrate competitive instruction-following and response-generation performance in technical reports, model cards, or publicly available benchmarks were prioritized. Practical availability and operational feasibility for repeated execution were also considered. Based on these criteria, four open-weight models were used: gpt-oss-120B, qwen3.5-122B-A10B, gemma4-26B-A4B, and qwen3.6-35B-A3B.

The same system and user prompt structure, round-specific instructions, output format, and Likert-scale evaluation criteria were applied to all models. Among the decoding parameters, temperature and the maximum output-token limit were explicitly configured. Temperature was set to 0.7 in Round 1 to support broader exploration during open-ended idea generation and to 0.2 from Round 2 onward to support more stable reassessment and consensus formation. The maximum output length was set to 32,768 tokens. Other decoding parameters were not explicitly specified and therefore followed the default settings of each model-serving environment.

No random seed was specified. The purpose of this study was not to reproduce identical individual responses, but to examine group-level performance differences across agent-diversity conditions under a common experimental procedure. Therefore, the seed was not fixed so that the results would not depend on a single sampling trajectory and so that the stochastic variation inherent in LLM generation could be reflected in repeated execution. Instead, the same prompt structure, agent composition, round-specific temperature settings, and output configuration were applied across all experimental conditions, and each condition was executed repeatedly. The resulting variation was incorporated into the group-level statistical analysis.

The system used a standardized API-based calling method to interact with external LLM inference environments. This allowed us to maintain the same input format and output structure without depending on a specific model provider or execution environment. In addition, we fixed the prompt structure, output format, and evaluation scale for each round to increase comparability across experimental conditions.

The prompts given to expert agents consisted of two main components: a system prompt and a user prompt. The system prompt defined each agent’s personality, belief, professional background, expertise level, and role. Through this structure, the variety, separation, and disparity conditions were reflected in the actual response generation process. The user prompt specified the task to be performed in each round. In the open-ended round, agents were asked to freely propose opportunity factors and challenge factors related to a given topic. In the closed-ended round, agents were asked to evaluate structured items using a five-point Likert scale.

Finally, we applied a response normalization procedure to address the formal instability of LLM outputs. LLM responses may be returned in various formats, including explanatory text, Markdown, or incomplete JSON fragments. Therefore, they must be converted into a structured format for subsequent analysis and storage. To this end, we applied a post-processing procedure that extracts key values from responses, repairs incomplete syntax, and normalizes outputs into a common schema. This process plays an important role in reducing information loss caused by abnormal outputs and ensuring data integrity in large-scale repeated experiments. Figure 2 illustrates the structure of the prompt used to obtain LLM responses. In this study, prompt design functions not merely as an input template, but as a core implementation mechanism through which the diversity of agent groups is reflected in the actual decision-making process.

Figure 2. Prompt structure for expert-agent response generation.

4.4. Outputs of the Implemented System

Based on the MADS architecture and implementation described above, we developed an executable system and conducted open-ended and closed-ended Delphi experiments. During the experiment, the open-ended Delphi round produced raw idea data generated by agents, whereas the closed-ended Delphi rounds produced repeated evaluations of benchmark items and consensus judgment results. These outputs served as the units of analysis for calculating the performance indicators defined in Section 3.

First, responses generated by agents in the open-ended Delphi round consisted of lists of opportunity factors and challenge factors. Each response was output in JSON format, and each opportunity or challenge factor contained the fields id, title, and description. This structure was designed to ensure that agents did not simply produce keyword lists, but instead generated specific agenda items for the problem domain based on their roles and perspectives.

Each experimental group consisted of eight agents, and each agent generated ten opportunity factors and ten challenge factors. Accordingly, each experimental group produced 80 opportunity factors and 80 challenge factors, for a total of 160 raw ideas. Based on these raw ideas, we measured the level of idea exploration among the decision-making performance indicators defined in Section 3.

Figure 3 shows an example of agent responses generated in the open-ended round. As shown in the figure, responses in the open-ended round consist of two sections: opportunities and challenges. These sections represent the positive possibilities and potential risks of AI-based talent acquisition, respectively. Each item consists of id, title, and description. In this study, we used the information stored in the title field as the basic unit of analysis.

Figure 3. Example of Agent-Generated Responses in the Open-Ended Survey Round.

The title field concisely summarizes the core content of each opportunity or challenge factor proposed by an agent. Therefore, we treated each title as a raw idea and used it as the unit of analysis for measuring the level of idea exploration. In contrast, the description field was used as supplementary information for interpreting the context and content of each response, but it was not directly used to calculate the evaluation indicators. Table 8 summarizes the units of analysis for response data generated in the open-ended survey round.

Table 8. Units of Analysis for Response Data in the Open-Ended Survey Round.

Item	Description
Output format	JSON
Response structure	Opportunities and challenges
Number of responses per experimental run	80 opportunity factors, 80 challenge factors, and 160 items in total
Related evaluation indicators	SeD, AC, IV

Responses generated in the open-ended Delphi round were stored in the database at the agent-response level. Each response record included the experimental group, round information, agent attributes, response type, and actual response content. In addition, experiment-level metadata included the experimental topic, LLM used, group type, Variety Blau Index, Separation Blau Index, and completion status. This storage structure made it possible to trace which agent generated which idea under a specific diversity condition and to analyze the relationship between group composition conditions and output results.

Next, in the closed-ended Delphi rounds, agents repeatedly evaluated predefined benchmark items (see Table 9). The benchmark items consisted of opportunity and challenge factors for AI-based talent acquisition derived from a Delphi experiment with human experts. Each item included item_id, title, description, and category information. These items were presented identically to all experimental groups, enabling comparison of the evaluation convergence process and consensus formation patterns according to group diversity conditions.

Table 9. Units of Analysis for Evaluation Data in the Closed-Ended Delphi Rounds.

Item	Description
Input items	Benchmark-based opportunity and challenge factors
Response format	Likert-scale evaluation scores
Round structure	Repeated evaluation for up to three rounds
Round-level outputs	Item-level mean, standard deviation, and weighted mean
Consensus judgment criterion	Whether the item-level standard deviation is below a predefined threshold
Items evaluated in the next round	Items that did not reach consensus in the previous round
Basic unit of analysis	Item-level round statistics and agent-level evaluation scores
Related evaluation indicators	ICQ, ECQ, CoS, ICR, VRR

In the closed-ended rounds, each agent evaluated the benchmark items using a Likert scale. At the end of each round, statistical values such as the mean, standard deviation, and weighted mean were calculated for each item. If the standard deviation satisfied the predefined consensus criterion, the item was considered to have reached consensus. Items that reached consensus were excluded from evaluation in subsequent rounds, while only items that had not reached consensus were re-evaluated in the next round. This procedure was designed to analyze whether opinion dispersion within an agent group decreases across rounds and whether groups under specific diversity conditions reach consensus more quickly and stably.

The data generated in the closed-ended Delphi rounds consisted mainly of agent-level evaluation scores, item-level round statistics, and consensus judgment results. Agent-level evaluation scores refer to the scores assigned by individual agents to each item. Item-level round statistics include the mean and standard deviation of group evaluations for each item. Consensus judgment results indicate whether the standard deviation for each item satisfies the consensus criterion and identify the round in which the item reached consensus.

The results of the closed-ended rounds were stored as item-level aggregate statistics and consensus judgment results. For each item, the evaluation scores assigned by the eight agents were aggregated into the round-level mean and standard deviation. If the standard deviation satisfied the predefined consensus criterion, the item was treated as having reached consensus. Items that reached consensus were excluded from subsequent rounds, while non-consensus items were re-evaluated in the next round. Table 10 shows how the evaluation scores of selected items changed across rounds in the closed-ended Delphi process and how consensus outcomes were determined.

Table 10. Example of Item-Level Evaluation Convergence in the Closed-Ended Delphi Rounds.

Item ID	Item Title	Round 1 Mean (SD)	Round 2 Mean (SD)	Round 3 Mean (SD)	Consensus
O1	Support HR in interview scheduling	4.73 (0.45)	4.88 (0.33)	-	Reached in R2
O7	AI reduces time-to-hire	4.50 (0.82)	4.63 (0.52)	-	Reached in R2
O14	Algorithmic bias in screening	3.13 (0.99)	3.00 (0.89)	2.88 (0.84)	Not reached by R3

As shown in Table 10, O1 and O7 reached consensus in Round 2 as their standard deviations decreased, whereas O14 did not satisfy the consensus criterion even after evaluations were repeated through Round 3. The hyphen (-) indicates that the item had already reached consensus in a previous round and was therefore excluded from evaluation in subsequent rounds. These round-level aggregate statistics were used to calculate the performance indicators defined in Section 3, including consensus quality, consensus speed, initial consensus rate, and variance reduction rate.

The evaluation data generated in the closed-ended rounds were used to calculate indicators related to consensus formation and decision-making quality as defined in Section 3. Item-level means and standard deviations show the extent to which group evaluations converged in each round, while consensus status and the round in which consensus was reached were used to assess how quickly the group arrived at a stable judgment. In addition, the Consistency Score derived from repeated agent responses was used to assess the stability of individual judgments, and the influence score was used to adjust the extent to which each agent’s evaluation was reflected in aggregation.

In summary, the outputs of the open-ended survey round were used as data for evaluating the diversity and comprehensiveness of ideas generated by the agent group, whereas the outputs of the closed-ended Delphi rounds were used as data for analyzing repeated evaluations of benchmark items and the process of consensus convergence. This distinction shows that MADS was designed to process the problem exploration stage and the consensus formation stage using different data structures and evaluation indicators.

5. Experimental Results

5.1. Experimental Design and Analysis Procedure

This experiment was conducted using the MADS described in Section 4 to examine how agent-group diversity affects collective decision-making performance. Following the diversity composition strategy defined in Section 3, 11 experimental groups were constructed: the homogeneous group H; the variety diversity conditions A-1, A-2, and A-3; the separation diversity conditions B-1, B-2, and B-3; the disparity diversity conditions C-1, C-2, and C-3; and the maximum diversity condition D.

To improve statistical reliability, each condition was repeatedly tested using four large language models with different architectures and parameter scales. The experiment was repeated 10 times for each model, resulting in 40 independent runs per group.

Performance was evaluated across three dimensions: Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. These dimensions were measured using semantic diversity ($SeD$), agenda coverage ($AC$), idea validity ($IV$), internal and external consensus quality ($ICQ$, $ECQ$), convergence speed ($CoS$), initial consensus rate ($ICR$), and variance reduction rate ($VRR$). For idea exploration metrics, ideas were vectorized using sentence-transformers/all-mpnet-base-v2 [35].

Group differences were examined using one-way ANOVA for each performance indicator. When significant differences were found, post-hoc tests with multiple-comparison correction were conducted, and statistical significance was determined based on adjusted p-values. In addition, Pareto frontier analysis was conducted to assess the trade-offs among Idea Exploration Extent, Consensus Quality, and Consensus Efficiency and to identify agent-group configurations that were not dominated across these three performance dimensions.

Before examining performance differences across the experimental groups, we conducted a supplementary manipulation check to address whether the personality, belief, and role attributes assigned through prompts were reflected in observable differences in the agents’ responses. Differences in Round 1 SeD across the values of each attribute dimension were examined using Kruskal–Wallis tests followed by pairwise comparisons with Holm-adjusted p-values. This analysis was intended to provide behavioral evidence of prompt-based attribute differentiation rather than to establish the full theoretical or psychometric validity of the assigned attributes.

5.2. Manipulation Check for Agent Attributes

To examine whether the personality, belief, and role attributes assigned through prompts were reflected in observable differences in the agents’ actual responses, a supplementary manipulation check was conducted using the Round 1 responses. Semantic diversity (SeD), which directly captures semantic differences among generated responses, was used as the outcome measure. Because each attribute dimension consisted of three or more categories, the sample sizes differed across attribute values, and the assumption of normality could not be confidently established, the nonparametric Kruskal–Wallis test was applied.

As shown in Table 11, statistically significant differences in SeD distributions were observed across the attribute values for all three dimensions: belief, personality, and role. These findings indicate that, across the full set of responses, the semantic characteristics of the generated responses differed according to the prompt-assigned attribute types.

Table 11. Kruskal–Wallis Test Results for SeD across Agent Attribute Values.

Attribute Dimension	H	df	p-Value	Significance
Belief	82.751	3	<0.001	***
Personality	35.226	4	<0.001	***
Role	13.179	2	0.001	**

Note. H denotes the Kruskal–Wallis test statistic. ** p < 0.01, *** p < 0.001.

Holm-adjusted post-hoc comparisons identified several significant attribute-level differences. Advocate agents showed lower SeD than Guardian, Pragmatist, and Innovator agents; Openness showed higher SeD than Neuroticism, Agreeableness, and Extraversion; and Leader agents showed higher SeD than Junior Staff and Middle Manager agents.

Overall, the personality, belief, and role attributes assigned through prompts were significantly associated with the semantic characteristics of the Round 1 responses. These results provide supplementary evidence that the prompt-based attribute differences were reflected in the generated responses. However, they do not demonstrate that the agents fully reproduced the theoretical behavioral characteristics associated with each attribute or establish the independent effect of each attribute.

5.3. Group Differences in Decision-Making Performance

Table 12 presents the mean values of the performance indicators for each experimental group, providing a descriptive comparison of performance across the diversity conditions. Table 13 reports the results of the one-way ANOVA, including the statistical significance of the differences among the diversity conditions and their corresponding effect sizes, measured using η² and ω². Statistically significant differences were observed for SeD, ICQ, CoS, ICR, and VRR, whereas the differences in AC, IV, and ECQ were not statistically significant.

Table 12. Mean values of performance indicators by experimental group.

Group	Performance Indicators
	SeD	AC	IV	ICQ	ECQ	CoS	ICR	VRR
H	0.5486	0.1806	0.0509	0.9695	0.6715	2.350	0.9372	0.2932
A-1	0.5498	0.1728	0.0487	0.9507	0.6682	2.875	0.9172	0.0535
A-2	0.5465	0.1733	0.0488	0.9340	0.6686	2.850	0.9106	0.0506
A-3	0.5585	0.1772	0.0501	0.9412	0.6648	2.800	0.9194	0.0223
B-1	0.5636	0.1867	0.0527	0.8878	0.6786	2.725	0.8356	0.3302
B-2	0.5812	0.2011	0.0567	0.8184	0.6823	2.775	0.7683	0.2202
B-3	0.5822	0.1939	0.0545	0.8433	0.6857	2.725	0.7811	0.2226
C-1	0.5524	0.1783	0.0502	0.9456	0.6540	2.600	0.9028	0.1896
C-2	0.5556	0.1761	0.0495	0.9189	0.6480	2.575	0.8900	−0.2544
C-3	0.5470	0.1672	0.0470	0.9156	0.6477	2.450	0.8922	−0.1210
D	0.6009	0.1978	0.0556	0.7477	0.6562	3.000	0.7233	−0.1411

Note. All groups had 40 independent runs. Lower CoS values indicate faster consensus, whereas higher values for the other indicators indicate better performance.

Table 13. ANOVA results for performance indicators.

Performance Dimension	Indicators	F	p-Value	Significance	η2	ω2
Idea Exploration Extent	SeD	8.683	<0.001	Significant	0.168	0.149
	AC	0.931	0.505	Not significant	0.021	0.000
	IV	0.557	0.849	Not significant	0.013	0.000
Consensus Quality	ICQ	6.219	<0.001	Significant	0.127	0.106
	ECQ	1.778	0.062	Not significant	0.040	0.017
Consensus Efficiency	CoS	4.263	<0.001	Significant	0.090	0.069
	ICR	20.885	<0.001	Significant	0.327	0.311
	VRR	5.649	<0.001	Significant	0.116	0.096

Note. η² and ω² indicate effect sizes; η² represents the proportion of observed variance explained by group differences, while ω² provides a less biased population-level estimate. Negative ω² estimates were truncated to zero.

5.3.1. Idea Exploration Extent

Idea exploration extent was evaluated using semantic diversity (SeD), agenda coverage (AC), and idea validity (IV). The analysis of ideas generated in the open-ended round showed that semantic diversity (SeD) was highest in D, the maximum-diversity group. Agenda coverage (AC) and idea validity (IV) were both highest in B-2.

As shown in Table 13, the group differences in SeD among the diversity conditions were statistically significant, with a large effect size according to Cohen’s conventional criteria [36]. This indicates that the diversity condition had a substantial effect on semantic diversity. In contrast, the group differences in AC and IV were not statistically significant. Thus, although B-2 recorded the highest mean values for AC and IV, the observed differences did not provide statistical evidence that the diversity conditions affected benchmark agenda coverage or idea validity.

The post-hoc test results for SeD showed that D, the maximum diversity group, had significantly higher semantic diversity than H, A-1, A-2, A-3, B-1, C-1, C-2, and C-3. However, its differences from B-2 and B-3 were not significant after multiple-comparison correction. B-2 and B-3 also showed significantly higher SeD than H, A-1, and A-2, whereas the A-series and C-series groups did not differ significantly from H. The statistically significant post-hoc comparisons are presented in Appendix A.

Overall, the results indicate that group diversity primarily affected semantic diversity rather than all aspects of idea exploration. While D, B-2, and B-3 expanded the semantic distance among generated ideas, no significant group differences were found for AC or IV. Thus, separation diversity and the maximum diversity condition appear to be more effective in broadening the semantic scope of exploration, but they did not significantly improve benchmark agenda coverage or idea validity.

5.3.2. Consensus Quality

Consensus quality was evaluated using internal consensus quality (ICQ) and external consensus quality (ECQ). Based on the results of the closed-ended evaluation round, the analysis of consensus quality showed that internal consensus quality (ICQ) was highest in the homogeneous group H and lowest in D. External consensus quality (ECQ) was highest in B-3. However, ECQ values were generally distributed within a narrow range of 0.65 to 0.69, indicating that the mean differences among groups were not large.

A statistically significant difference in ICQ was observed across the diversity conditions. The large effect sizes reported in Table 13 indicate that group diversity was substantially associated with the level of internal consensus achieved within the agent groups.

By contrast, the group differences in ECQ were not statistically significant. Although the mean ECQ values were relatively higher in the B-series groups, the differences did not meet the 0.05 significance level. Therefore, based on the present results, it is difficult to conclude that a particular diversity condition significantly improved the level of external alignment with the human expert benchmark.

The post-hoc test results for ICQ showed that D had significantly lower values than H, A-1, B-1, B-2, B-3, C-1, C-2, and C-3, whereas H showed significantly higher ICQ than A-1, A-2, and A-3. This indicates that the homogeneous group achieved the most stable internal consensus convergence, while the maximum diversity condition showed relatively low internal consensus quality.

For ECQ, no significant group differences were identified in the post-hoc tests. Although B-3, B-2, and B-1 showed relatively high mean ECQ values, these differences cannot be interpreted as statistically significant superiority in external consensus quality. The statistically significant post-hoc comparisons are presented in Appendix B.

Overall, the results suggest that diversity conditions affected internal consensus convergence but did not significantly change external alignment with the human expert benchmark. In particular, high levels of combined diversity appear to weaken internal consensus quality, whereas homogeneous groups tend to support more stable convergence.

5.3.3. Consensus Efficiency

Consensus efficiency was analyzed using convergence speed (CoS), initial consensus rate (ICR), and variance reduction rate (VRR). A lower CoS value indicates faster convergence to consensus, whereas higher values of ICR and VRR indicate greater consensus efficiency.

As shown in Table 12, the homogeneous group H recorded the lowest CoS and the highest ICR, indicating the fastest convergence and the highest level of initial agreement. In contrast, D showed the highest CoS and the lowest ICR. B-1 recorded the highest VRR, whereas D and several C-series groups showed negative VRR values. These negative values indicate that, in these groups, the variance of some conflict items increased rather than decreased in the final round.

The ANOVA results reported in Table 13 showed statistically significant group differences for CoS, ICR, and VRR. A comparison of effect sizes showed that ICR had the largest effect size. This indicates that the group diversity condition was most strongly reflected in differences in the initial consensus rate.

The post-hoc test results for CoS showed that D had significantly higher values than H, B-1, B-3, and C-3, indicating slower convergence to consensus. A-1 also showed significantly higher CoS than H. For ICR, D had significantly lower values than H, A-1, A-2, A-3, B-1, C-1, C-2, and C-3, while B-2 and B-3 showed significantly lower ICR than H and several A- and C-series groups. These results indicate that high separation diversity and the maximum diversity condition reduced the level of initial agreement.

For VRR, B-1 showed significantly higher variance reduction than D, A-1, A-2, A-3, and C-3, while H showed significantly higher VRR than D, A-3, and C-3. In contrast, D had significantly lower VRR than H and all B-series groups, suggesting that initial disagreements were not sufficiently reduced under the maximum diversity condition. The statistically significant post-hoc comparisons are presented in Appendix C.

Overall, the results show that the homogeneous group H achieved relatively fast consensus and a high initial consensus rate, whereas D showed the lowest initial consensus rate, the slowest convergence, and a negative variance reduction rate. These findings suggest that diversity can expand idea exploration but may also increase initial disagreement and reduce consensus efficiency during the closed-ended evaluation stage.

5.4. Trade-Off Analysis Among Performance Dimensions

The preceding analyses showed a trade-off between exploration and convergence. The maximum diversity condition D showed the highest semantic diversity but relatively low internal consensus quality and consensus efficiency, whereas the homogeneous group H showed strong consensus performance but lower idea exploration than D and the B-series groups.

To examine this trade-off, this study conducted a Pareto frontier analysis. Pareto frontier analysis identifies alternatives that are not dominated by other alternatives across multiple performance criteria [37,38]. This approach is appropriate for the present study because agent-group configurations may improve one performance dimension while reducing another. In multi-objective optimization, non-dominated sorting has been widely used to identify efficient solution sets under such conflicting objectives [39].

Each experimental group was treated as a decision-making alternative, and Idea Exploration Extent, Consensus Quality, and Consensus Efficiency were used as the three performance dimensions. The sub-indicators of each dimension were normalized to a 0–1 scale and averaged: SeD, AC, and IV for Idea Exploration Extent; ICQ and ECQ for Consensus Quality; and CoS, ICR, and VRR for Consensus Efficiency. Because lower CoS indicates faster consensus, CoS was reverse-normalized.

A group was classified as non-dominated when no other group performed at least as well across all three dimensions and better in at least one dimension. As shown in Table 14, H, B-1, B-2, B-3, and D were identified as non-dominated groups, whereas all A-series and C-series groups were classified as dominated alternatives.

Table 14. Pareto frontier analysis results based on idea exploration extent, consensus quality, and consensus efficiency.

Group	Idea Exploration Extent	Consensus Quality	Consensus Efficiency	Pareto Status	Groups Dominated by the Group	Groups Dominating the Group
H	0.2785	0.8130	0.9789	Frontier	A-1, A-2, C-1, C-2, C-3	-
A-1	0.1331	0.7275	0.5418	Dominated	-	H
A-2	0.1195	0.6944	0.5426	Dominated	-	B-1, H
A-3	0.2787	0.6612	0.5660	Dominated	C-2	B-1
B-1	0.4925	0.7218	0.6493	Frontier	A-2, A-3, C-2, C-3	-
B-2	0.8792	0.6152	0.4562	Frontier	-	-
B-3	0.7408	0.7155	0.5031	Frontier	C-2	-
C-1	0.2536	0.5284	0.7380	Dominated	C-2, C-3	H
C-2	0.2299	0.3897	0.4777	Dominated	-	A-3, B-1, B-3, C-1, H
C-3	0.0034	0.3785	0.6214	Dominated	-	B-1, C-1, H
D	0.9309	0.1118	0.0646	Frontier	-	-

H and D represented two opposite frontier positions. H was a consensus-oriented group, achieving the highest consensus quality and efficiency despite relatively low idea exploration. In contrast, D was an exploration-oriented group, achieving the highest idea exploration score but the lowest consensus quality and efficiency. This indicates that maximum diversity expanded exploration at the cost of convergence.

The B-series groups formed intermediate frontier positions. B-1 showed the most balanced performance among the frontier groups, while B-2 and B-3 achieved higher idea exploration than H while maintaining moderate levels of consensus quality. In contrast, all A-series and C-series groups were classified as dominated alternatives, suggesting that personality-based variety diversity or influence disparity alone did not provide a superior balance across the three performance dimensions.

Overall, the Pareto frontier analysis indicates that agent diversity does not have a single optimal level. Rather, different diversity structures support different decision-making goals: H is suitable for consensus-oriented tasks, D for exploration-oriented tasks, and the B-series groups for more balanced decision-making. These findings suggest that multi-agent LLM systems should be designed as a multi-objective problem in which the appropriate diversity structure depends on the purpose of the task.

5.5. Regression-Based Analysis of Diversity–Performance Relationships

This supplementary analysis conducted OLS regression by treating diversity indices (Variety Blau Index, Separation Blau Index, and ADS) as continuous predictors to complement the existing ANOVA-based between-group comparisons. The analysis yielded the following findings (Table 15). First, Separation diversity exhibited a clear trade-off structure in which it consistently enhanced idea exploration (SeD) while significantly reducing internal consensus quality (ICQ) and initial consensus rate (ICR); no inverted-U optimal point was identified for exploration within the observed range, indicating that semantic diversity expanded continuously as belief-orientation diversity increased. Second, Variety diversity imposed costs by slowing consensus speed while offering limited exploration gains, and ADS (influence disparity) weakened the effectiveness of feedback-based opinion adjustment, to the extent that at extreme levels it appeared to induce opinion polarization in some conflict items. Third, the combined model with all three dimensions entered simultaneously confirmed high explanatory power (R² = 0.954–0.977), providing empirical support for the conclusion that diversity effects operate in an additive and independent manner. These findings suggest that diversity in multi-agent system design should be treated not as a single variable but as a multidimensional structural property differentiated by dimension, and that strategically calibrating the level of Separation according to task objectives constitutes the key design variable for balancing performance outcomes.

Table 15. Subset OLS regression results.

DV	Model	Predictor	β	SE	t	p	R2	Adj. R2	F	p (Model)
SeD	Linear	Intercept	0.548	0.005	112.91	<0.001 ***	0.263	−0.106	0.714	0.487
		Variety	0.008	0.009	0.845	0.487
	Quadratic	Intercept	0.550	0.006	85.02	<0.001 ***	0.490	−0.530	0.481	0.714
		Variety	−0.019	0.041	−0.463	0.724
		Variety2	0.033	0.050	0.668	0.625
ICQ	Linear	Intercept	0.967	0.006	156.48	<0.001 ***	0.859	0.788	12.181	0.073 †
		Variety	−0.040	0.011	−3.490	0.073 †
	Quadratic	Intercept	0.970	0.006	168.84	0.004 **	0.953	0.857	10.018	0.218
		Variety	−0.090	0.037	−2.437	0.248
		Variety2	0.062	0.045	1.402	0.394
ICR	Linear	Intercept	0.932	0.008	122.47	<0.001 ***	0.608	0.412	3.106	0.220
		Variety	−0.025	0.014	−1.762	0.220
	Quadratic	Intercept	0.938	0.003	299.65	0.002 **	0.974	0.923	18.897	0.161
		Variety	−0.098	0.020	−4.879	0.129
		Variety2	0.091	0.024	3.768	0.165
SeD	Linear	Intercept	0.549	0.001	446.06	<0.001 ***	0.995	0.993	396.322	0.003 **
		Separation	0.047	0.002	19.908	0.003 **
	Quadratic	Intercept	0.548	0.002	292.32	0.002 **	0.995	0.986	107.795	0.068 †
		Separation	0.051	0.013	3.845	0.162
		Separation2	−0.005	0.017	−0.296	0.817
ICQ	Linear	Intercept	0.960	0.020	47.477	<0.001 ***	0.921	0.882	23.327	0.040 *
		Separation	−0.188	0.039	−4.830	0.040 *
	Quadratic	Intercept	0.971	0.021	45.447	0.014 *	0.965	0.895	13.847	0.187
		Separation	−0.352	0.151	−2.338	0.257
		Separation2	0.220	0.195	1.125	0.463
ICR	Linear	Intercept	0.925	0.020	46.730	<0.001 ***	0.944	0.915	33.469	0.029 *
		Separation	−0.220	0.038	−5.785	0.029 *
	Quadratic	Intercept	0.938	0.013	75.193	0.009 **	0.991	0.973	55.954	0.094 †
		Separation	−0.417	0.088	−4.749	0.132
		Separation2	0.264	0.114	2.317	0.259

The pure variety groups include H, A-1, A-2, and A-3 (n = 4), and the pure separation groups include H, B-1, B-2, and B-3 (n = 4). Superscript ² indicates the squared term of the corresponding predictor. † p < 0.10, * p < 0.05, ** p < 0.01, *** p < 0.001.

6. Discussion and Conclusions

6.1. Summary and Discussion

This study empirically analyzed how diversity in agent groups affects the performance of Delphi-based decision-making in multi-agent LLM systems. To this end, drawing on organizational diversity theory, diversity was divided into three dimensions—variety, separation, and disparity—and operationalized as personality diversity, belief and value diversity, and influence asymmetry, respectively. In addition, rather than measuring collective decision-making performance as a single outcome, this study analyzed it across three dimensions: Idea Exploration Extent, Consensus Quality, and Consensus Efficiency.

The results showed that agent-group diversity had differentiated effects on decision-making performance. First, in terms of Idea Exploration Extent, the maximum diversity group D, which combined all diversity dimensions, and the B-series groups, B-1, B-2, and B-3, which progressively manipulated the level of belief diversity, showed relatively high performance. In particular, group D exhibited the highest semantic diversity, indicating that when personality, belief, and influence structures are combined, the semantic scope of idea exploration can be substantially expanded. However, group D showed low performance in Consensus Quality and Consensus Efficiency, demonstrating that maximum diversity does not guarantee optimal outcomes across all performance dimensions.

Second, in terms of Consensus Quality and Consensus Efficiency, the homogeneous group H showed the most stable performance. Although group H showed limited performance in Idea Exploration Extent, it achieved high values in Consensus Quality and Consensus Efficiency. This suggests that when differences in perspectives among agents are relatively small, fast and stable consensus formation becomes more feasible in multi-agent system design. However, such homogeneity may restrict the generation of new ideas or the exploration of diverse perspectives. Therefore, efficient consensus does not necessarily imply qualitatively rich decision-making.

Third, the B-series groups, B-1, B-2, and B-3, which manipulated diversity in beliefs and value orientations, were all identified as non-dominated groups in the Pareto frontier analysis. This indicates that belief diversity does not improve performance in a single uniform way, but rather forms different exploration–consensus balance points depending on the level of diversity. Specifically, B-1 showed relatively balanced performance across Idea Exploration Extent, Consensus Quality, and Consensus Efficiency, and dominated several other groups. By contrast, B-2 and B-3 showed lower Consensus Efficiency than H but clear advantages in Idea Exploration Extent while maintaining Consensus Quality above a certain level. In particular, B-2 showed strength in expanding exploration with a high Idea Exploration Extent score, while B-3 simultaneously showed a high level of exploration and relatively high Consensus Quality. These results suggest that diversity in agents’ beliefs and value orientations plays an important role in expanding the scope of exploration in multi-agent systems and, at an appropriate level, can also form a balance with Consensus Quality.

Finally, the A-series groups, which manipulated personality-based variety diversity, and the C-series groups, which manipulated influence disparity, were not included in the Pareto frontier. This suggests that personality diversity or influence asymmetry alone is insufficient to improve the balance among Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. In particular, influence disparity may contribute to some improvement in efficiency, but it did not function as a superior alternative in terms of overall performance balance.

Taken together, the findings of this study demonstrate that the balance between exploration and consensus may vary depending on the type and level of agent diversity. However, the model-specific characteristics of the underlying LLMs should also be considered when interpreting these findings. Although multiple LLMs were employed to mitigate the influence of model-specific characteristics, differences across models in instruction-following capability, response consistency, and the tendency to generate semantic variation may have affected the extent to which the assigned personality and belief conditions were reflected in actual agent responses. Differences in reasoning capability and response stability may also have influenced the processes of idea exploration and consensus formation. Therefore, the observed performance differences should be interpreted with consideration of the possibility that both agent diversity and the characteristics of the underlying models contributed to the results. Nevertheless, the systematic performance differences observed across diversity conditions under the same experimental structure suggest that, in addition to the choice of the underlying model, the composition of the agent group is an important design factor affecting collective decision-making performance in multi-agent systems.

6.2. Theoretical Contributions

The theoretical contributions of this study are as follows. First, this study applied organizational diversity theory to research on LLM-based multi-agent systems, showing that the three dimensions of diversity discussed in human organization studies—variety, separation, and disparity—can be extended as design variables for artificial agent groups. Existing studies on multi-agent LLM systems have primarily focused on role allocation, discussion structures, prompt design, and self-review mechanisms. In contrast, this study is distinctive in that it systematically analyzed how the compositional diversity of agent groups themselves affects decision-making performance.

Second, this study extended the evaluation of multi-agent decision-making performance beyond single measures such as accuracy or final consensus achievement, proposing a multidimensional performance framework consisting of Idea Exploration Extent, Consensus Quality, and Consensus Efficiency. This reflects the idea that what matters in multi-agent decision-making is not merely reaching consensus quickly, but converging toward stable and efficient judgment while exploring diverse possibilities. Accordingly, this study contributes to LLM-based collective decision-making research by expanding performance evaluation from outcome-centered assessment to process- and balance-oriented assessment.

Third, this study empirically demonstrated that the relationship between diversity and decision-making performance is not linear. The maximum diversity group D maximized Idea Exploration Extent but showed substantially lower Consensus Quality and Consensus Efficiency. By contrast, the homogeneous group H showed strong Consensus Quality and Consensus Efficiency but limited exploration. These results show that the effect of diversity in multi-agent LLM systems cannot be interpreted simply as “more diversity is better.” In particular, the Pareto frontier analysis revealed that different diversity conditions produce different performance balance points, suggesting that the optimal level of diversity in multi-agent system design may vary depending on the purpose of the task.

6.3. Practical Contributions

This study provides the following practical implications for designing LLM-based multi-agent systems. First, simply assigning diverse personality personas to agents may not be sufficient to improve decision-making performance. In this study, the personality-based variety diversity conditions were not included in the Pareto frontier. This suggests that diversity in expression style or communication style alone may not be sufficient to secure substantive diversity in decision-making perspectives.

Second, diversity in beliefs and value orientations should be considered a key design variable in multi-agent systems. The finding that all B-series groups were included in the Pareto frontier indicates that assigning agents different judgment criteria and value frames can be effective in expanding idea exploration while maintaining a certain level of Consensus Quality. Therefore, in practical decision-support systems, it may be useful to combine agents that represent different value criteria, such as market orientation, stability, ethics, and innovation.

Third, the level of diversity in agent groups should be designed differently depending on the purpose of the task. For tasks where the discovery of new opportunities or strategic exploration is important, configurations with high exploration performance, such as D, B-2, or B-3, may be appropriate. By contrast, for tasks where fast and stable consensus is important, highly homogeneous configurations such as H may be more suitable. For tasks that require a balance between exploration and consensus, a low level of belief diversity, such as B-1, may be an effective alternative.

Fourth, influence asymmetry should be designed with caution. In recent LLM-based multi-agent systems, agent orchestration structures have been discussed as an important design approach, in which an orchestrator agent assigns roles to multiple sub-agents, integrates their responses, and coordinates the overall decision-making process. However, for such structures to function effectively, a balance is needed between the authority of the central coordinator and the independent judgment of individual agents. In this study, all C-series groups were classified as dominated groups, indicating that assigning high influence to a particular agent does not always lead to efficient decision-making. This means that in LLM-based multi-agent systems, when the judgment of a specific agent is excessively reflected, the advantages of collective intelligence—namely the independent presentation of diverse perspectives and mutual verification—may be weakened. Therefore, rather than allowing a single highly influential agent to strongly control the judgments of sub-agents, multi-agent systems should be designed so that the orchestrator performs procedural coordination while sufficiently incorporating dissenting opinions and alternative perspectives from individual agents.

6.4. Limitations and Future Research

This study has the following limitations. First, the experiment was conducted using a specific decision-making task within a single task domain and was evaluated based on a specific Delphi benchmark. Therefore, the observed effects of agent diversity may partly depend on the structure and characteristics of the selected task and benchmark. This limits the external validity of the findings, and the results should not be directly generalized to all types of Delphi tasks or decision-making situations. Future research should extend the experiment to diverse task domains, such as technology forecasting, policy decision-making, product planning, and risk assessment, and employ alternative benchmark tasks with different structures and levels of complexity to examine the robustness and generalizability of the observed diversity effects.

Second, this study manipulated agent diversity through prompts. Although personality, belief, and influence conditions were implemented through system prompts and response aggregation structures, it cannot be guaranteed that LLMs always internalize these instructions consistently. Future research should further examine the linguistic and behavioral characteristics of agent responses to verify the extent to which prompt-induced diversity is actually reflected in response patterns.

Third, this study used the level of alignment with a human expert benchmark as part of external consensus quality. However, given the nature of Delphi research, the consensus results of human experts should also be understood not as an absolute ground truth, but as a consensus-based criterion derived from the judgments of a specific expert group. Therefore, future research should extend the evaluation of external validity by using multiple human expert groups, independent evaluators, or actual ex post performance data.

Fourth, the Pareto frontier analysis in this study was conducted as a post-hoc analysis based on the observed results of predefined experimental groups. Therefore, this study did not automatically search for the optimal agent configuration, but rather analyzed the relative balance structures among the given diversity conditions. Future research should build on the findings of this study by treating the level of belief diversity, the number of agents, and the method of influence allocation as continuous design variables, and by using multi-objective optimization methods to explore optimal agent configurations according to task objectives.

Fifth, all experimental groups in this study were fixed at eight agents to maintain consistency in the evaluation conditions with the human expert benchmark study. Accordingly, this study did not examine the effect of group size itself on idea exploration, consensus quality, or consensus efficiency. Because increasing or decreasing the number of agents may alter the distribution of perspectives, response dispersion, and the consensus formation process within a group, future research should treat the number of agents as an independent experimental variable. Further investigation of the interaction between group size and diversity structure is also needed to identify agent-group configurations appropriate for different task objectives.