RAE: A Role-Based Adaptive Framework for Evaluating Automatically Generated Public Opinion Reports

Abstract

Public Opinion Reports are essential tools for crisis management, yet their evaluation remains a critical bottleneck that often delays response actions. Recently, dominant Large Language Model (LLM)-based evaluators often overlook a critical challenge: highly open-ended dimensions such as “innovation” and “feasibility” require synthesizing diverse stakeholder perspectives, as different groups judge these qualities from fundamentally different perspectives. Motivated by this, we propose the Role-based Adaptive Evaluation (RAE) framework. This framework employs an adaptive mechanism leveraging multi-perspective evaluation insights through role-based analysis, and further introduces dynamically generated roles tailored to specific contexts for these dimensions. RAE further incorporates multi-role reasoning aggregation to minimize individual biases and enhance evaluation robustness. Extensive experiments demonstrate that RAE significantly improves alignment with human expert judgments, especially on challenging highly open-ended dimensions.

1. Introduction

Public Opinion Reports consolidate information from news and social media to provide critical decision support for crisis management [1]. While Large Language Models (LLMs) have made automated report generation technically feasible, the evaluation of these reports remains a significant bottleneck: manual evaluation is time-consuming and often causes delays that lead decision-makers to miss optimal response windows, potentially worsening the crisis [2].

The open-ended and long-form nature of Public Opinion Reports presents a significant challenge for quality evaluation [3,4]. Traditional n-gram-based metrics such as ROUGE and BLEU prove inadequate as they rely on surface-level text overlap and fail to measure deeper qualities like semantic coherence [5]. Consequently, the research community has increasingly turned to LLM-based evaluators, which provide a more comprehensive evaluation that better aligns with human judgment [6,7].

Despite these advancements, a significant research gap remains because current LLM-based evaluators primarily operate from a singular and static perspective. Specifically, single-perspective evaluators [8,9] rely on a single LLM judgment, which fails to capture the multi-stakeholder nature of crisis evaluation. Furthermore, fixed-role evaluators [10] assign predefined personas but lack the flexibility to adapt to diverse crisis contexts. Such limitations prevent these systems from capturing the multi-faceted nature of public opinion analysis as these reports inherently consolidate varied and even conflicting sentiments and viewpoints from diverse groups, including the public, media, and official institutions. This inherent diversity of perspectives creates a fundamental problem for automated evaluation since open-ended qualities like innovation or feasibility are not absolute and instead vary according to the specific perspective of each stakeholder. For instance, a report recommendation to use a city stadium as a mass shelter might be rated highly feasible by a government official focusing on public space availability but deemed not feasible by a public health expert who anticipates significant risks of disease transmission. Single-perspective evaluators cannot synthesize these diverse stakeholder concerns, which leads to misalignment with human expert judgments that naturally integrate multiple viewpoints. As shown in Figure 1, multi-perspective evaluation achieves stronger alignment with expert judgment than single-perspective approaches by explicitly integrating diverse stakeholder viewpoints.

To address this research gap, we propose the Role-based Adaptive Evaluation (RAE) framework, which enhances alignment with human expert judgment by capturing multiple perspectives. This framework centers on how to design an evaluation approach that effectively captures diverse stakeholder perspectives for public opinion reports, particularly on highly open-ended dimensions. RAE operates through an adaptive role-play mechanism where the central innovation involves tailoring the role composition strategy to the specific nature of each evaluation dimension. For dimensions with reference information (e.g., Title, Causes), RAE employs representative stakeholder roles to ensure consistency while capturing diverse concerns. Since these dimensions rely on verifiable facts, representative roles are sufficient for providing necessary perspectives. For highly open-ended dimensions (e.g., Innovation, Feasibility), RAE dynamically generates context-specific roles to reflect varying stakeholder viewpoints. This dynamic role generation is essential because stakeholder priorities and expertise vary significantly across different crisis contexts. Moreover, RAE incorporates multi-role reasoning aggregation, which leverages insights from multiple roles to minimize biases and improve the overall robustness of the evaluation process.

This work makes the following key contributions:

• We propose a multi-perspective evaluation framework that addresses open-ended dimensions evaluation challenges by dynamically generating roles to capture diverse, context-specific stakeholder viewpoints.
• We introduce a multi-role reasoning aggregation mechanism that, for each evaluation dimension, synthesizes perspectives from multiple roles into a comprehensive dimension-specific score.
• Comprehensive experiments demonstrate that RAE achieves strong alignment with human expert judgments, with particularly notable improvements on the challenging highly open-ended dimensions.

2. Related Work

While human evaluation is the gold standard for open-ended text generation [4], it suffers from being expensive, time-consuming, and unreliable [2]. Given the increasing capabilities of LLMs, LLM-as-a-Judge frameworks serve as a scalable and cost-efficient alternative [5,6]. Zero-shot and few-shot LLM-based evaluators show higher agreement with human annotators than traditional metrics across NLG tasks [11,12,13], sometimes surpassing crowdworkers when compared against expert annotations [14]. Recent research extends this approach by developing customizable judges [8,9,15] that can adopt diverse societal perspectives, ranging from the general public to domain experts [16,17,18].

However, relying on a single evaluator, whether human or LLM, can introduce bias and instability into the evaluation results [19,20,21]. Recent multi-agent evaluation systems explore diverse collaboration strategies [22,23]. Some strategies focus on interaction mechanisms [24], such as discussion with weighted voting [25], interactive querying [26], and structured committees or reviewer-chair hierarchies [27,28]. Others focus on the composition of the agent group, for example by simulating multiple roles within a single model [29] or using multiple smaller models for cost-effectiveness [30]. Collectively, these collaborative methods have shown improvement in response factuality [31], enhance performance on complex tasks [10], and mitigate the Degeneration-of-Thought (DOT) problem [32].

Despite these advances, existing approaches overlook a fundamental challenge in crisis-related public opinion evaluation [8,9,10]. Crisis scenarios are inherently multifaceted, and highly open-ended dimensions require synthesizing diverse stakeholder perspectives because different groups judge qualities like “innovation” or “feasibility” from fundamentally different viewpoints that vary by context. To address this, we introduce an adaptive mechanism that dynamically generates context-tailored evaluator roles for these dimensions, combined with a reasoning aggregation mechanism that synthesizes independent role-specific judgments into comprehensive and robust final assessment.

3. Preliminaries: The Structure of a Public Opinion Report

A Public Opinion Report comprises five key components, each designed to provide a comprehensive overview of a crisis event.

3.1. Event Title

The Event Title serves as a concise identifier that allows readers to immediately grasp the event’s essence. A well-formed title conveys the crisis name, type, and time, facilitating efficient storage and retrieval.

3.2. Event Summary

The Event Summary offers a structured overview covering six key aspects: Crisis Name (What), Location (Where), Time (When), Cause (Why/How), Involved Parties (Who), and Impact (the event’s consequences and severity).

3.3. Event Timeline

The Event Timeline traces public opinion evolution through three phases: Incubation Period (initial low-activity phase), Peak Period (rapid attention growth with widespread discussion), and Decline Period (diminishing interest phase).

3.4. Event Focus

The Event Focus analyzes public opinion from two participant groups—Netizens and Authoritative Institutions—extracting three dimensions for each: Core Topics, Sentiment Stance, and Key Viewpoints.

3.5. Event Suggestions

The Event Suggestions section provides actionable recommendations based on the preceding analysis. These may include communication strategies, public relations actions, or policy adjustments.

4. Methodology

We propose the RAE framework (Figure 2), a role-playing-based approach for multi-perspective evaluation. Compared to existing single-perspective evaluators [8,9] and fixed-role systems [10], RAE is designed to better reflect the multi-stakeholder nature of crisis evaluation by combining an adaptive role-play mechanism for dimension-specific evaluation with multi-role reasoning aggregation for robust score synthesis.

4.1. Task Definition and Research Hypotheses

Given a crisis event E and its corresponding generated public opinion report R, our evaluation task aims to assess the quality of R across a set of evaluation dimensions $\mathcal{D}=\{d_1,d_2,\dots ,d_n\}$, where n represents the total number of dimensions. In our framework, $n=15$ dimensions span five major components including Event Title, Event Summary, Event Timeline, Event Focus, and Event Suggestions. Each dimension $d_i\in \mathcal{D}$ evaluates a specific quality aspect with scores ranging from 1 (unacceptable) to 5 (excellent).

The evaluation process can be formalized as a function:

$$S_d=f_{\mathrm{eval}}(r,R,E,d),S_d\in \{1,2,3,4,5\},$$

(1)

where r denotes the evaluator role (representing a specific stakeholder perspective), R is the generated report, E contains source materials and reference information, d is the dimension being evaluated, and $S_d$ is the resulting quality score.

Our goal is to construct a framework that maximizes the alignment between the evaluated scores $\mathcal{S}$ and the human expert consensus $S_{\mathrm{human}}$ as follows:

$${\mathcal{S}}^{\ast }=\underset{f_{\mathrm{eval}}}{\arg \max }\mathrm{Alignment}(S,S_{\mathrm{human}}),$$

(2)

where $\mathcal{S}=\{S_{d_1},S_{d_2},\dots ,S_{d_n}\}$ represents the set of scores across all dimensions, and $\mathrm{Alignment}\left(·\right)$ measures the degree of agreement through correlation metrics (Pearson’s $\rho$, Kendall’s $\tau$) and error metrics (MAE).

To investigate and validate the effectiveness of the proposed RAE framework, we formulate the following four research hypotheses based on the core design of our system:

• H1: What aggregation mechanism optimally synthesizes multi-role evaluations into reliable scores?
• H2: Does increasing the number of dynamic roles continuously improve evaluation performance, or is there an optimal threshold?
• H3: Why employ adaptive role composition rather than using only dynamically generated roles for all dimensions?
• H4: What underlying mechanisms account for RAE’s improved alignment with human expert judgments in complex evaluation scenarios?

4.2. Adaptive Role-Play Mechanism

The central innovation of RAE is its adaptive role-play mechanism, which tailors the role composition strategy based on the nature of evaluation dimensions. All dimensions in RAE employ multi-perspective evaluation through multiple evaluator roles, but the approach to composing these roles differs based on whether the dimension relies on reference information or is highly open-ended.

4.2.1. Predefined Roles for Dimensions with Reference Information

For dimensions grounded by reference information (e.g., Title, Causes), evaluations revolve around verifiable facts. However, evaluating the quality of how the report writes and presents these facts (e.g., whether a Title is concise, or whether the Cause captures the core issue) still requires synthesizing multiple viewpoints. To achieve a stable, consistent, and representative multi-perspective evaluation for these dimensions, RAE employs a set of predefined, representative roles (e.g., “Netizen”, “Journalist”). Each role follows the schema: ⟨Role Type, Role Description⟩, where the Role Description specifies the evaluation focus for that role. By leveraging this fixed set of specialized roles, RAE ensures the evaluation of these core components is repeatable and robust, establishing a foundational multi-perspective evaluation layer for the framework.

4.2.2. Dynamic Roles for Highly Open-Ended Dimensions

Highly open-ended dimensions, such as evaluating the Innovation or Feasibility of suggestions, lack fixed standards and assess long-form content where quality judgments vary dramatically by context. To address this, RAE dynamically generates context-specific evaluator roles, capturing the diverse, situation-dependent viewpoints needed for comprehensive evaluation. Each dynamic role follows the schema: $\langle \mathrm{Role\; Type},\mathrm{Role\; Description}\rangle$, where the description captures the stakeholder’s specific concerns and evaluation priorities. Given a crisis event E and its report R, this generation process can be formalized as:

$$P\left(\mathit{roles}\right|E,R)=\mathrm{LLM}(P_{\mathrm{dynamic}},E,R)$$

(3)

Dynamic generation may produce semantically overlapping roles. To ensure diverse and representative evaluators while maintaining efficiency, RAE implements a two-stage deduplication process. First, we apply MinHash with n-gram features to efficiently remove roles with high surface-level similarity. Second, we perform semantic clustering to identify conceptually similar roles. A sentence embedding model encodes each role description into a vector representation, and the k-means algorithm then clusters these vectors based on similarity. Finally, only the role closest to each cluster centroid proceeds to the evaluation stage.

4.3. Multi-Role Reasoning Aggregation

The final stage of RAE consolidates diverse evaluations from multiple roles into a single, reliable score for each dimension. To identify the aggregation strategy that best aligns with human expert judgments, we investigate three alternative approaches:

(1) Voting Aggregation: This approach employs majority voting to aggregate scores from all evaluating roles. For a given evaluation dimension d, the final score ${\widehat{S}}_d$ is the score (on a 1–5 scale) that is most frequently assigned by the set of all evaluating roles R:

$${\widehat{S}}_d=\underset{s\in \{1,\dots ,5\}}{\mathrm{argmax}}{\displaystyle \sum _{r=1}^{\left|R\right|}}\mathbf{1}(S_{r,d}=s)$$

(4)

where $\left|R\right|$ is the total number of roles, $S_{r,d}$ is the score from role r, and $\mathbf{1}\left(·\right)$ is the indicator function.

(2) Process Aggregation: This reasoning-centric approach employs a judge LLM. For a given dimension d, the judge receives only the textual justifications, excluding their numerical scores. The judge analyzes these arguments for logical soundness and then generates a final score ${\widehat{S}}_d$ based on this qualitative evaluation:

$${\widehat{S}}_d={\mathrm{LLM}}_{\mathrm{judge}}\left(\{J_{1,d},J_{2,d},\dots ,J_{\left|R\right|,d}\}\right)$$

(5)

where $J_{r,d}$ represents role r’s justification for dimension d.

(3) Comprehensive Aggregation: This approach employs a judge LLM to holistically evaluate both the numerical scores and the textual justifications. For a given dimension d, the judge LLM receives these dual inputs to determine the most appropriate final score ${\widehat{S}}_d$:

$${\widehat{S}}_d={\mathrm{LLM}}_{\mathrm{judge}}\left(\{S_{1,d},S_{2,d},\dots ,S_{\left|R\right|,d}\},\{J_{1,d},J_{2,d},\dots ,J_{\left|R\right|,d}\}\right)$$

(6)

where $S_{r,d}$ represents role r’s score for dimension d.

5. Experiments

5.1. Datasets

OPOR-Bench [33] is an event-centric dataset specifically designed for public opinion report, covering 463 crisis events spanning from 2012 to 2025.

These events span diverse crisis types including natural disasters (e.g., cyclones, floods) and human-caused crises (e.g., traffic accidents, industrial incidents). Each crisis event in the dataset includes a collection of multi-source documents, specifically news articles and social media posts, along with a structured reference summary containing key event information (location, time, cause, impact, etc.) and a set of reports generated by LLMs.

5.2. Experimental Setup

5.2.1. Baseline

We compare RAE against OPOR-Eval, an agentic evaluation pipeline that orchestrates three predefined tools (Fact-Checker, Opinion-Miner, and Solution-Counselor) for section-specific evaluation of public opinion reports [33].

5.2.2. Model Configuration

To evaluate the performance of RAE, we establish a comprehensive experimental framework encompassing both model configurations and statistical metrics. For the automated evaluation phase, we employ GPT-4o and DeepSeek-V3 to score reports across 15 distinct dimensions. During this process, the model temperature is set to 0.3 and all other hyperparameters remain at their default values as specified in the evaluation guidelines in Appendix A.

5.2.3. Evaluation Metrics

To ground these automatic results, we also conduct a human evaluation and calculate the Intraclass Correlation Coefficient (ICC) to assess the inter-rater reliability among human experts. The alignment between automatic metrics and human judgments is subsequently quantified through three key indicators including Spearman’s rank correlation ($\rho$), Kendall’s tau ($\tau$), and Mean Absolute Error (MAE).

5.3. Evaluation Framework Validation

5.3.1. Human Evaluation Protocol

To validate the RAE framework, we conduct a rigorous human evaluation with three domain experts who co-design the scoring criteria. The entire process is blind, with experts independently scoring reports via a dedicated annotation interface. The protocol divides into two sequential phases:

• Calibration: Experts first rate a subset of reports. The goal is to establish a consistent understanding of the scoring criteria, with agreement formally measured using the Intraclass Correlation Coefficient (ICC).
• Formal Evaluation: After achieving a high level of agreement in the calibration phase, the experts proceed to score the entire corpus. This phase yields the three complete sets of ratings used for our human-agent agreement analysis.

5.3.2. Agreement Analysis

We analyze the human evaluation results to address two critical questions: (1) How reliable are human experts? (2) How strong does RAE align with human judgments?

Answer 1: Our analysis confirms a high degree of inter-rater reliability among human experts. Established guidelines categorize ICC values indicate poor (<0.50), moderate (0.50–0.75), good (0.75–0.90), and excellent (>0.90) agreement [34]. In light of these standards, our results in

Table 1. The ICC scores confirm high inter-rater reliability among human experts.

Dimension	ICC3
Event Title	0.843
Event Summary
Event Nature	0.868
Time & Loc.	0.860
Involved Parties	0.856
Causes	0.879
Impact	0.887
Event Timeline
Date Acc.	0.839
Sub Events	0.793
Event Focus
Contro. Topic	0.894
Repr. Stmt.	0.877
Emo. Anal.	0.893
Event Suggestions
Rel.	0.855
Feas.	0.841
Emo. Guide.	0.850
Innov.	0.861

reveal consistently high reliability across all dimensions, ranging from objective ones like Date Accuracy to more subjective ones such as Innovation. The consistently strong ICC scores validate the human ratings as a reliable standard for evaluating the RAE framework.

Answer 2: RAE achieves strong alignment with human judgments. Established guidelines consider correlations (Spearman’s $\rho$, Kendall’s $\tau$) above 0.70 are considered strong [35], and lower MAE indicates better alignment [36]. As detailed in

Table 2. Human-agent agreement analysis on RAE aggregation methods (VOTE, PROC, COMP) compared with the Direct (no role-play) baseline and the OPOR-Eval framework. Best results in bold. Higher correlation ($\rho$, $\tau$, ↑) and lower MAE (↓) indicate better performance.

Method	Spearman’s $\mathit{\rho }\uparrow$	Kendall’s $\mathit{\tau }\uparrow$	MAE ↓
GPT-4o
VOTE	0.873	0.864	0.335
PROC	0.862	0.852	0.346
COMP	0.844	0.834	0.382
Direct	0.695	0.605	0.534
OPOR-Eval [33]	0.721	0.633	0.412
DeepSeek-v3
VOTE	0.868	0.861	0.414
PROC	0.855	0.843	0.443
COMP	0.825	0.819	0.488
Direct	0.285	0.240	0.885
OPOR-Eval [33]	0.312	0.288	0.761

, the RAE framework demonstrates consistently high performance, with the majority of results surpassing the “strong” correlation threshold. RAE also outperforms the OPOR-Eval baseline on both base models, achieving higher correlation and lower MAE on GPT-4o and DeepSeek-v3. This robust performance confirms that RAE reliably captures human evaluation patterns. The consistency holds across different aggregation methods, role counts, and base models. We further analyze the impact of specific design choices in the following sections.

6. Discussions

This section provides a hypothesis-driven discussion of our experimental results. We validate the research hypotheses defined in Section 4.1 through controlled comparisons, ablations, and a case study, and we summarize the practical implications of each finding.

6.1. Validation of H1: Voting Aggregation Achieves Optimal Alignment with Human Judgments

Our results confirm H1, which posits that a democratic synthesis of multi-role perspectives is superior to centralized judgment. The Voting strategy demonstrates a clear performance advantage, achieving an average Spearman’s $\rho$ of 0.835 that consistently outperforms both the Process (0.818) and Comprehensive (0.798) methods. This superiority is robust, holding true across different base models and numbers of roles, and is further corroborated by its consistently lower MAE, as detailed in Table 2. This hierarchy reveals a compelling insight about aggregation mechanism design. The success of voting suggests that it leverages the “wisdom of crowds” effect, where diverse, independent judgments converge toward the ground truth by canceling out individual biases and random errors. Conversely, introducing a single judge LLM creates a potential single point of failure that may amplify rather than mitigate subjective biases. Most notably, the Comprehensive approach (providing both scores and justifications) paradoxically degrades performance. This stems from its centralized design, which concentrates decision-making in a single judge LLM. This centralization is vulnerable to two sources of bias: (1) while voting distributes power across roles (allowing biases to cancel out), the judge’s own preferences can override the statistical consensus; and (2) the justifications introduce noise, allowing a single eloquent outlier (e.g., a low score with strong reasoning) to unduly sway the judge. In either case, this replaces distributed collective wisdom with a potentially flawed individual judgment. In contrast, the straightforward majority voting mechanism preserves the robustness of multi-perspective evaluation through direct numerical aggregation, proving to be the most reliable aggregation strategy for the RAE framework.

6.2. Validation of H2: Five Dynamic Roles Optimally Balance Diversity and Relevance

To determine the optimal number of dynamic roles, we vary the number of roles (k) retained after clustering from 3 to 7. Given that Voting aggregation achieves the strongest alignment, we report its performance across different base models for each k value. As shown in Figure 3, the results reveal a clear non-linear trend peaking at $k=5$, which validates H2. Performance improves as k increases from 3 to 5, peaking at k = 5 with the strongest alignment, but then declines at k = 6. This pattern highlights a critical trade-off between perspective diversity and domain relevance. Initially, adding more roles (from 3 to 5) is beneficial as it captures essential stakeholder perspectives. For a flooding crisis, increasing from 3 to 5 roles might add critical roles such as “Urban Infrastructure Planner” and “Public Health Specialist” alongside core roles like “Hydraulic Engineer” and “Emergency Coordinator”, ensuring comprehensive coverage of the crisis’s multifaceted dimensions. However, excessive role generation (at k = 6 or beyond) begins to introduce marginally relevant perspectives that weaken evaluation quality. For instance, generating a sixth role like “Agricultural Policy Expert” for the same flooding crisis may provide limited insight into urban flood response evaluation, introducing noise rather than meaningful perspective diversity. The system trades comprehensive domain coverage for increasingly less central viewpoints. This finding provides practical guidance for deploying RAE by suggesting the generation of a diverse pool of candidate roles, followed by clustering and retaining roughly five key representatives to maximize both evaluation coverage and precision.

6.3. Validation of H3: Dynamic Role Generation Is Critical, While Adaptive Composition Improves Efficiency

To analyze the role of each component and validate the adaptive composition strategy, we conduct a comparative analysis on GPT-4o, using the Voting-based aggregation configuration with five dynamically generated roles. We design four variants: (1) w/o Predefined: Removes the predefined representative roles. It evaluates the dimensions with reference information directly without role-play, but keeps dynamic roles for open-ended dimensions; (2) w/o Dynamic: Removes the dynamically generated roles. It evaluates the highly open-ended dimensions directly without role-play, but keeps predefined roles for dimensions with reference information; (3) All Predefined: Applies the predefined representative roles to all dimensions; and (4) All Dynamic: Applies dynamically generated roles to all dimensions.

The ablation study in

Table 3. Component analysis and strategy comparison of RAE on GPT-4o.

Method	Title	Summary					Timeline		Focus			Suggestions				AVG
	Tit.	Nat.	T&L.	Par.	Cau.	Imp.	Dat.	Sub.	Top.	Sta.	Emo.	Rel.	Fea.	Gui.	Inn.
RAE	0.88	0.87	0.89	0.88	0.86	0.88	0.86	0.89	0.86	0.89	0.87	0.88	0.87	0.87	0.88	0.88
w/o Predefined	0.81 †	0.83 †	0.83 †	0.84 †	0.77 †	0.79 †	0.83 †	0.87	0.85 †	0.89 †	0.85 †	0.89 †	0.87	0.88 †	0.86 †	0.84 (−0.04)
w/o Dynamic	0.83 †	0.84 †	0.86 †	0.85	0.86 †	0.87 †	0.82 †	0.78 †	0.81 †	0.76 †	0.82 †	0.78	0.75 †	0.75	0.77 †	0.81 (−0.07)
all Predefined	0.86 †	0.84 †	0.87	0.86 †	0.83 †	0.85	0.84 †	0.81 †	0.82 †	0.77	0.83 †	0.79 †	0.76 †	0.77 †	0.78 †	0.82 (−0.06)
all Dynamic	0.90 †	0.87 †	0.88 †	0.86 †	0.89 †	0.90 †	0.87 †	0.87 †	0.86 †	0.89 †	0.87 †	0.87 †	0.88 †	0.86 †	0.86 †	0.88 (0.00)

Notes: AVG shows the average Pearson correlation. Drops from the full RAE model are in red italics. ^† denotes statistically significant drop (p < 0.05). Abbreviations: Tit. = Title, Nat. = Nature, T&L. = Time & Location, Par. = Parties, Cau. = Causes, Imp. = Impact, Dat. = Date Accuracy, Sub. = Sub Events Coverage, Top. = Controversy Topic, Sta. = Representative Statement, Emo. = Emotional Analysis, Rel. = Relevance, Fea. = Feasibility, Gui. = Guidance, Inn. = Innovation.

provides strong support for H3. First, dynamic role generation is the most critical component for high performance. This is strongly supported by the substantial performance drops in the two variants lacking dynamic roles for open-ended evaluation: w/o Dynamic drops to 0.81 (−0.07) and All Predefined drops to 0.82 (−0.06). The substantial performance drops in the two variants lacking dynamic roles for open-ended evaluation (w/o Dynamic drops to 0.81 (−0.07) and All Predefined drops to 0.82 (−0.06)) strongly support this conclusion. Both demonstrate that a fixed, predefined role strategy cannot adequately capture the context-specific expertise required for highly open-ended dimensions. Second, dynamic generation proves universally effective. Surprisingly, All Dynamic achieves performance (0.88) identical to the full RAE. This suggests that dynamic role generation, while specifically designed for open-ended dimensions, does not harm performance on dimensions with reference information. It is likely capable of generating domain-specific roles that prove equally effective as our predefined ones. Finally, these results validate the practical efficiency of RAE’s adaptive strategy. The similarly poor performance of All Predefined (0.82) and w/o Dynamic (0.81) confirms that generic representative roles cannot substitute for context-specific expertise. Performance drops concentrate in open-ended dimensions: Innovation (0.78 vs. 0.88), Feasibility (0.76 vs. 0.87), and Guidance (0.77 vs. 0.87). While All Dynamic performs well, full RAE achieves equivalent performance with greater efficiency by using predefined roles for reference-informed dimensions, avoiding unnecessary dynamic generation costs. This adaptive strategy represents an optimal balance between performance and computational efficiency.

6.4. Validation of H4: Case Evidence Illustrates Why Multi-Perspective Evaluation Improves Expert Alignment

To provide intuition into the mechanics of RAE, we present a case study on a report about the “2021 Black Sea incident”, comparing RAE’s evaluation of the “Innovation” dimension against a vanilla LLM baseline, as depicted in Figure 4. The vanilla LLM baseline, lacking a specific viewpoint, offers a generic 4/5 score, praising the suggestions as a “fresh approach”. In contrast, RAE’s five dynamically generated roles produce domain-specific disagreements. Some roles, such as the Publicist and Diplomat, recognize innovation in transparency initiatives and multilateral dialogues within their respective domains, assigning scores of 4/5. However, the Military Strategist, International Lawyer, and Media Journalist assign a more critical 3/5, converging on the view that while the suggestions are solid, they remain “conventional in nature” and do not introduce fundamentally new concepts in their respective domains. With three of the five roles assigning a score of 3, the voting mechanism converges on a final, more grounded score of 3/5. This case vividly demonstrates how voting aggregation avoids being dominated by a few positive opinions and instead captures the cautious consensus. It reveals a nuanced reality that the baseline misses: what appears innovative in one domain can be conventional in others. This context-aware perspective leads to a more robust and well-grounded evaluation.

7. Conclusions

In this work, we propose the Role-based Adaptive Evaluation (RAE) framework for public opinion report evaluation in crisis contexts. RAE combines an adaptive role-play mechanism with multi-role reasoning aggregation to better reflect the multi-stakeholder nature of crisis assessment and to improve alignment with human expert judgments, especially on highly open-ended dimensions. Our findings show that RAE improves reliability over static single-agent evaluation by integrating diverse stakeholder viewpoints and synthesizing them with robust aggregation. Specifically, multi-perspective role-play is crucial for open-ended dimensions, where perceived innovation and feasibility depend strongly on context-specific stakeholder concerns. Moreover, the adaptive role-play mechanism balances evaluation quality and computational cost. In particular, voting-based aggregation provides the most reliable synthesis of role-specific judgments, with five dynamic roles offering an effective trade-off between perspective diversity and relevance. Furthermore, RAE provides practical value by informing stakeholder-oriented messaging [37], supporting multi-stakeholder ethical engagement [38], and guiding context-sensitive deployment in complex public opinion environments [39].

7.1. Limitations

Despite its effectiveness, RAE has several limitations. First, the computational cost of multi-role aggregation is higher than single-agent systems due to multiple LLM calls, which may be prohibitive for resource-constrained or real-time large-scale deployments. Second, performance is strictly dependent on the base LLM’s capability to simulate roles, and inherent model biases or limited domain knowledge could potentially affect evaluation quality. Furthermore, our current scope is primarily limited to text-based, English-language crisis reports, which may affect its generalization to multimodal data (e.g., images/videos) or other languages and domains.

7.2. Future Work

Future work will focus on developing parameter-efficient aggregation mechanisms and exploring model distillation to reduce computational overhead and latency. We will also explore stronger role simulation and calibration methods, including domain-aware role constraints and automatic checks for role relevance and consistency, to mitigate model bias and knowledge gaps. Finally, we will extend RAE to additional languages and domains and explore multimodal extensions for more realistic crisis communication settings.