Distilling the Complexity of Agent-Based Simulations into Textual Explanations via Large Language Models

Abstract

Communicating the design and results of agent-based models (ABMs) to subject matter experts is challenging, which hinders participation and limits trust in simulation-based decision support. Large language models (LLMs) can communicate ABMs as textual summaries, thus complementing traditional disclosure through statistical and visualization techniques. While prior work translated the structure of conceptual models into narratives via LLMs, our extension covers the dynamics of simulation models via an automated simulation-to-text method that extracts contextual information from NetLogo ABMs, performs repeated simulations, and generates narrative descriptions (including the model’s purpose, parameters, and simulation dynamics) using mutimodal LLMs. Furthermore, four summarization algorithms spanning abstractive and extractive methods provide shorter reports. Using Design-of-Experiments methods over three peer-reviewed ABMs, state-of-the-art multimodal LLMs from 2026 (Gemini 3.1 Pro, Qwen 3.5, Kimi K2.5, Claude Opus 4.6) and different prompt elements (e.g., roles, examples, generating insights, statistical analyses), we compare our results with several reference reports (e.g., from associate professors). We find that report quality is determined mainly (i.e., up to 34% of the variance) by the summarization algorithm and its interaction with the LLM, with abstractive summarizers (BART, T5) producing more coherent and readable reports, while Claude Opus 4.6 is the most robust LLM.

1. Introduction

Explainability and transparency are important considerations in Modeling & Simulation (M&S) as they build trust and confidence in the validity of models, thus supporting the translation from a model’s insights into practice. The creation of understandable narratives to accompany models is particularly beneficial for end users who may not have the technical expertise to interpret raw simulation data directly or appreciate the limitations of specific model-building processes [1]. Explainability and transparency are multifaceted, as they can be conducted at the local (e.g., sensitivity to changes in individual parameters [2]) or global level (e.g., comparing point estimates from a simulation with ground truth), and they cover the production of a model [3] (how was it created?) as well as the intended use of the final product (Figure 1). When conceptual models strive to be comprehensive, they are at risk of becoming “spaghetti diagrams” [4], thus turning into unintended illustrations of complexity that offer little support for global explanations of the model and hinder translations into practices. The Foresight Obesity Map is a case in point, as it was depicted by researchers as an “almost incomprehensible web of interconnectedness [in which] the scale and number of interactions make it difficult to see how one might use it in any practical way to develop systemic approaches” [5] with others stating that the model “looks more like a spilled plate of spaghetti than anything of use to policymakers” [6]. The solution is not necessarily to force a model into a reduced scope or to omit mediating effects and multiple consequences [7], as over-simplified models may no longer be fit for purpose [8]. For example, public health models require the integration of biomedical, social, and environmental variables [9], so these models often need to be comprehensive and clearly reflect unintended consequences in order to serve their purpose, even if that initially appears to come at the expense of explainability and transparency.

Given the necessity of keeping models fit for purpose while addressing explainability and transparency, a possible solution could be that modelers provide all necessary descriptions. However, analyses of descriptions surrounding conceptual models and their production repeatedly showed that modelers do not systematically describe the model development process [3,10] or the sources of model elements such as relationships between variables [3]. The realization that the burden cannot solely be placed on modelers to provide more descriptions has resulted in a renewed interest for automation via Natural Language Generation. This is not merely a matter of generating (any) text, as simple templates could produce strings that describe a conceptual model such as “A increases B. B decreases C.” Advances in cognitive sciences show that certain representations of events and causality support language processing tasks in readers [11]; thus, we seek textual descriptions that allow readers to form correct views about the model being described. Generative Artificial Intelligence (GenAI) solutions such as GPT have been used to translate large conceptual models into descriptions at the local level (e.g., model components such as relationships [12,13]) or at the global level (e.g., decomposing a model into blocks [14] or summarizing its comparison with ground-truth data [15]) that have satisfactory readability levels and remain factually correct. This flurry of research on model-to-text transformation complements the significant research growth on text-to-model transformation [16,17,18,19].

Figure 1. Conceptual models (consisting of labeled nodes for concepts and directed edges for causal relationships) have been translated into text using GenAI solutions such as OpenAI’s GPT (dotted arrows), often with a focus on local elements [12,13] and occasionally by using summarization techniques to provide a sense of the entire model. In this paper, we generate text for simulation models, which adds dynamic scenarios.

Figure 1. Conceptual models (consisting of labeled nodes for concepts and directed edges for causal relationships) have been translated into text using GenAI solutions such as OpenAI’s GPT (dotted arrows), often with a focus on local elements [12,13] and occasionally by using summarization techniques to provide a sense of the entire model. In this paper, we generate text for simulation models, which adds dynamic scenarios.

As significant advances have been made on automatically producing narratives for conceptual models by using GenAI, the next frontier is to create descriptions for simulation models. The simulation-to-text task is more challenging than a model-to-text translation because simulations are dynamic systems; thus, we have to convey both the rationale and structure of a model (as in model-to-text transformation) and its dynamics in response to various stimuli or input scenarios (Figure 1). To date, there is a paucity of works in this domain. Notable exceptions include the study of Verduzco et al., who used GPT-4 to describe input files for the Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS), thus demonstrating the feasibility of employing GenAI to address lacking descriptions of simulation methods [20]. While GenAI can create narrations by interacting with agents or simulation logs [21], its primary goal was to generate compelling stories for the game industry (e.g., the hero, warrior, wizard, and dragon from Aoki et al. [22]) rather than to increase the explainability or transparency of a simulation serving as a decision-support tool. There is thus a demonstrated need to automatically produce descriptions that alleviate, at least partly, the challenges of conveying simulation models to a target audience.

In this paper, we focus on translating agent-based models (ABMs) into text. ABMs simulate interactions between individual entities (‘agents’) and their environment with successful applications across many domains including ecological and socio-ecological systems [23,24], sociology [25], or political decision making [26]. While ABMs are powerful tools to explore complex systems, their results are notoriously challenging to communicate due to the difficulties in analyzing and interpreting the multidimensional output data [27]. This difficulty is exacerbated when trying to communicate findings [28], particularly to non-experts who need to make informed decisions based on these outcomes [29]. Furthermore, the amount of data generated can be significant for detailed simulation models or those with large search spaces, but readers have a limited cognitive capacity. In other words, trying to solve the challenge of explainability by translating every aspect of a simulation model into an enormous phone book would just displace the problem. ABMs thus require distilling information into concise yet comprehensive insights, requiring sophisticated analytical approaches [27]. Note that we use the term agent exclusively in the context of ABMs to refer to simulated entities. Large language models are treated as external reporting components and are not conceptualized as autonomous or “agentic” systems within this paper.

Our overarching contribution is to generate concise executive-style reports from agent-based model (ABM) simulations with readability levels that suggest potential accessibility for non-modeling experts [30], while acknowledging that dedicated stakeholder studies are needed to fully assess how such reports support understanding and decision making [31]. This overarching aim is realized through three specific contributions:

• We design, implement, and evaluate a staged design-of-experiments (DoE) methodology to analyze and optimize multi-component LLM-based reporting systems. By combining aggregated ANOVA screening, factorial variance decomposition, and direct level-dominance comparisons, the approach attributes performance variability to individual factors and interaction effects, thereby providing a systematic and generalizable workflow for configuring simulation-to-text methods.
• We analyze how summarization strategies, prompt elements, and the identity of the large language model jointly influence the quality of generated reports, showing in particular that interactions between summarization algorithms and LLMs explain a substantial share of performance variability.
• We perform a comparative empirical evaluation across multiple ABMs using four recent large language models (Google’s Gemini 3.1 Pro Preview, Alibaba’s Qwen 3.5 27B, MoonshotAI’s Kimi K2.5, and Anthropic’s Claude Opus 4.6), providing practical guidance on evidence presentation, summarization choices, and model selection for simulation reporting tasks.

The remainder of this paper is organized as follows. In Section 2, we provide background information to keep the manuscript self-contained with special emphasis on techniques from Generative AI and their intersection with simulations. Building on the technical foundations summarized in our background, we present our methodology in Section 3, including the processes of data extraction, report generation, and our evaluation methods. Our results are presented in Section 4, which are followed by a discussion of our findings in Section 5. Finally, we provide concluding remarks in Section 6, and we support replicability by providing access to our code along with all results, ground truth datasets, and verification of the implementation.

2. Background

Our paper is motivated by the realization that simulation models such as ABMs can be complex and produce large amounts of data, which need to be succinctly conveyed to a variety of end users (Section 2.1). We propose using large language models (LLMs), which have already been employed in several simulation tasks (Section 2.2). We focus on using LLMs to generate reports, which can be evaluated (Section 2.3) in response to different prompt design techniques (Section 2.4).

2.1. Conveying Agent-Based Models to End-Users

ABMs can capture interactions between multiple agents (characterized by a number of dynamic attributes), over several steps (or ‘ticks’), through a variety of scenarios that can be performed multiple times to account for stochasticity. The sheer volume and complexity of the data produced by ABMs can overwhelm users, making it difficult to extract actionable insights directly from the raw outputs [30]. In addition, the design of the ABM and simulation can become complex, as it involves numerous agent attributes, several data- and theory-driven rules, along with various scenarios consisting of combinations of parameter values. This information can be comprehensively captured through standardized approaches to describe ABMs, such as the ODD (Overview, Design concepts, Details) protocol [32] and its extensions [33,34], and some aspects can be covered through a UML diagram (emphasizing software design) or reporting standards for data use in ABMs specifically [35] or the STRESS guidelines for simulations in general [36]. However, these standards and guidelines are not systematically used by the modeling community [37], and they can be challenging for stakeholders and model commissioners, who are not always modelers but need to attain a sense of what the model does and how to use it without parsing large volumes of technical information.

Several works have sought to improve communication for the design of ABMs and/or their simulation results, particularly for non-technical audiences. For example, Hall and Virrantaus used diagrams to convey the rules of ABMs [38], which was more recently echoed by Szangolies and colleagues in creating a visual version of the standardized ODD [39]. Visualizations have been an important tool and research area to communicate ABM for over a decade, across all phases of the modeling process [40], for instance using the popular NetLogo software [41]. While statistical techniques are commonplace in simulations (e.g., choosing a number of repeats for a target confidence interval), they often support modelers [42]. On occasions, statistics have been employed to convey ABMs to practitioners beyond M&S such as statisticians [43] or a broader audience [27]. As shown in the next subsection, advances in GenAI have provided new opportunities to explain ABMs as text, thus complementing studies on visualizations and statistics.

2.2. LLMs and Simulation

LLMs have intersected with simulations in several ways. While only one task (simulation-to-text) is directly relevant to this paper, we briefly discuss the names and techniques involved in three related areas to avoid confusion. First, in generative agent-based modeling or LLM-augmented social simulations [44], the decision-making modules of the agents are replaced or extended with LLMs (Figure 2). This allows agents to react to messages and to explain their reasoning [45,46], and it also transforms the challenge of crafting behavioral rules into a prompt engineering task that accounts for the agents’ attributes [47]. As pointed out by Larooij and Tornberg, “generative ABMs are still in a stage of early experimentation” [48] with prototypes emerging in late 2024 and early 2025. All ABMs used to generate simulations in this paper are ‘traditional’ as they use sets of rules to express behavior rather than LLMs. Second, in Agentic AI, the emphasis is on using a set of specialized agents to interpret a textual request [49]. For example, “approach potential clients” can trigger a market discovery agent that looks for suitable companies, then another agent extracts email addresses, and a third agent crafts tailored emails. In other words, agentic AI decomposes a problem so that it can be solved by a set of specialized, autonomous entities. The goals of the traditional ABMs studied in this paper are different, since they provide simulations as decision-support tools so that end users can estimate the effects of scenarios defined through input parameters. Third, there is a wealth of studies on generating simulations from text (Figure 2), drawing from earlier works on extracting ABM concepts from narratives [50,51], and then using questions and answering systems to automate the extraction [52,53] and finally generating simulation code in a target ABM platform such as NetLogo [54,55]. In this paper, we study the inverse transformation: instead of building an ABM from text, we use existing ABMs to generate a textual description.

As tools such as GPT make it possible to translate simulations into text, the idea of using GPT to explain simulations was proposed [15,56] and realized in a few studies. In particular, Jackson and Rolf noted the technical complexity of simulations and translated them into natural language for domain experts by translating the Python source of inventory management systems into English. While the results use a repetitive style that does not match the readability level expected of a report, this proof-of-concept demonstrated that GenAI could know enough about a simulation model in order to depict it with factually correct sentences [57]. Gürcan also noted that agents equipped with LLMs (i.e., generative ABM) could help with explainability, as the agents “can generate natural language explanations for their actions, decisions, and the underlying mechanics of the simulation” [44].

Note that both approaches can generate a large amount of low-level information either by translating the whole simulation code to text [57] or by asking each agent to explain how it decided and acted [44]. There is thus a need to distill the information, which we accomplish in this paper by using summarization techniques as presented in the next subsection.

2.3. Evaluating LLM-Generated Reports

When the output from LLMs is too verbose for practical use, we can further request that the LLM summarizes it. Several systematic reviews on text summarization cover the variety of methods employed in this field, as shown by Table 2 in [58] or Table 2 from an earlier review in [59]. Among several ways to organize these algorithms (e.g., statistical, deep-learning, LLMs), we consider two broad categories based on how they operate: extractive summarization copy–pastes the most relevant parts of the original text, while abstractive summarization synthesizes information by paraphrasing the text. By design, extractive summarization only provides factual content: it cannot hallucinate sentences that were not present in the input. It is also computationally efficient, and it has long been employed to summarize informative texts. These algorithms are designed to evaluate the importance of sentences (e.g., by computing a score) and assemble them without any changes, which risks creating a fragmented summary with abrupt transitions between sentences. BERT summarizer [60] (Bidirectional Encoder Representations from Transformers) is one of the most well-known extractive summarizers, but its standard context window size is small so it needs a sliding window to process inputs in chunks [61]. LongformerExt overcomes the input length limitation of models by using a sparse attention mechanism [62] that allows it to deal with long documents, which is relevant in the context of this paper.

In contrast, abstractive summarization is closer to the way that people write summaries: we want a text that is about the input material, and we may need to use new words or expressions to retain the essence of the original material. Transfer learning is a common component of abstractive summarization models, since they are pre-trained to generate tokens beyond the original text in order to remain cohesive [63]. As noted by Koh and colleagues, “Bidirectional and Auto-Regressive Transformers (BART) and Text-to-Text Transfer Transformer (T5) are the two most widely used sequence-to-sequence pre-trained models” for abstractive summarization [59]. As general domain encoders, T5 and BART (for abstractive summarization) along with Longformer and variants of BERT (for extractive summarization) remain key elements of current applied studies in summarization [64,65].

Since summarizing is about communicating, it is important to identify the audience and the format [66,67]: do we seek one paragraph for an email to a modeler, a five-minute elevator pitch to a potential model commissioner, or a written overview for subject-matter experts? Executive summaries are a common format that is found across application areas from military simulations [68] to behavioral modeling [69]. Executive summaries typically require concise and focused content yet remain powerful for the readers to take action [70], often coming down to one or two pages with no more than 1000 words [71].

Evaluating the results produced by LLMs involves both quantitative (our focus) and qualitative methods [72,73], which may employ human readers and/or LLMs to evaluate items on a Likert scale [74,75] along with measures of inter-rater agreement [76,77] as further described in Supplementary Material S1. To effectively evaluate LLMs and avoid common pitfalls, it is essential to develop a comprehensive suite of metrics rather than relying on a single metric [78]. Quantitative evaluation often uses metrics such as BLEU, METEOR, and ROUGE, which are standard in summarization tasks, to compare the generated content against a ground truth [79,80]. BLEU (Bilingual Evaluation Understudy) measures the precision of n-grams in the generated text compared to a reference text, using a scale from 0 to 1, where higher scores indicate better quality. Prior studies showed performances reaching 24.58% for this metric on summaries from BART [81] and 63% from BERT [82]. METEOR (Metric for Evaluation of Translation with Explicit ORdering) evaluates translation quality based on precision, recall, and alignment, producing scores ranging from 0 to 1, with higher scores representing better quality. Researchers have reported results on summaries using BART (73.72%) [81] along with BERT (33%) [83]. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures recall by comparing the overlap of n-grams between the generated text and reference texts. It includes several variants, such as ROUGE-N (measuring n-gram overlap), ROUGE-L (measuring longest common subsequence), and ROUGE-W (measuring weighted longest common subsequence), with scores typically ranging from 0 to 1, where higher scores indicate better performance. Prior works display results for ROUGE-1 that are approximately 26.60 for BERT summaries and 52.53 for BART summaries [84]; for ROUGE-2, results approach 11.11 for BERT summaries and 21.83 for BART summaries [84], and lastly, for ROUGE-L, we have 24.29 for BERT and 49.75 for BART [84]. In addition, the Flesch Reading Ease [85] assesses the ease of reading a text based on sentence length and word syllable count with scores ranging from 0 to 100. Higher scores indicate easier readability with 90–100 being easily understandable by an average 11-year-old student and 0–30 being best understood by university graduates. Previous studies on LLMs have shown a Flesch Reading Ease between 24.88 and 58.01 [86].

2.4. Prompt Engineering and Experimental Design

Effective prompt engineering is critical for leveraging the capabilities of LLMs in generating high-quality explanations [87]. Rather than relying on random attempts, prompts must be carefully designed to include essential elements such as assigning roles, providing examples, or requesting insights. This structured approach ensures that the generated content is relevant and informative [88]. The design of LLMs can be articulated along several dimensions [89], which can be binary (e.g., whether to define a role and style) or multi-valued (e.g., whether the goal is expressed as looking-up information or creating). A notable challenge is that having more prompt elements does not guarantee better results [90]. Intuitively, ‘throwing the kitchen sink’ at the LLM may confuse it rather than guide it. Since we do not know a priori which elements to include, several prompts can be designed and their effects evaluated empirically.

The M&S studies that used prompt engineering have used several strategies to design their experiments. Several works have used the experimental design of varying one factor at a time [91,92,93], which involves systematically changing a single variable while keeping all other factors constant to isolate its individual effect on the outcome. This simple design avoids the complexity and costs of testing all possible combinations of multiple factors at the expense of missing how prompt choices influence each other. Other studies have used a $2^k$ factorial design with k binary prompt elements, which allows researchers to study both individual effects and interactions between factors efficiently at the expense of generating more prompts. For example, Zhao and colleagues tested a flat vs. hierarchical word representation (one binary factor) and whether to use a zero-shot approach that does not give information or to provide metaproperties documentation (another binary factor), resulting in a $2^2$ design [94]. To date, no study has used a $2^{k}r$ factorial design, that is, an experimental setup that also accounts for randomness in the output by performing r repeats of each experiment.

In a $2^k$ experiment, the analysis involves decomposing the total variance in the outcome (e.g., the BLEU or ROUGE scores) into contributions from main effects (1st order, individual factors) and interactions (2nd order for pairs of factors, higher order for combinations). For example, in the $2^2$ design above, results can be decomposed into the individual effect of word representation and documentation and their joint effect. Although the use of a factorial design is scarce to evaluate prompt elements in GenAI, this design of experiments is very well established in the simulation literature [95,96].

3. Materials and Methods

3.1. Overview

Our methods are divided into three main parts (Figure 3). First, we obtain data about the ABM (Section 3.2). Given a set of parameters, we run several simulations to obtain results from the model. We also extract data from the simulation file: we parse the documentation and the Graphical User Interface (GUI). Then, we generate a report using this information (Section 3.3). We transform simulation results into images and statistical summaries, thus leveraging the multimodal ability of the LLMs (Google’s Gemini 3.1 Pro Preview, Alibaba’s Qwen 3.5 27B, MoonshotAI’s Kimi K2.5, Anthropic’s Claude Opus 4.6) to generate a context and several trend analyses. A summarization algorithm reduces the number of paragraphs to the length suitable for an executive report. We optimize the prompt engineering process using a systematic Design of Experiments (DoE) to ensure the generated reports are both accurate and informative. Finally, we evaluate the report (Section 3.4) against full-length reports and a variety of summaries.

3.2. Data Extraction

An ABM that adheres to professional standards may include some documentation (e.g., at least a README file) and a minimal visual interface (i.e., the model is not solely driven from the command line). This provides potentially valuable information about an ABM, as it can explain its purpose or give context to its parameters. We thus extract this static data in addition to running the model to obtain dynamic simulation results. While our implementation uses ABMs coded in NetLogo, the automatic extraction process described below can be modified to accommodate other platforms.

For the static data, we extracted two components (Figure 4). First, we parse the documentation, ensuring that we ignore any default content (e.g., default headers from NetLogo) so that the information extracted was written by modelers specifically for their model. We also ignore URLs, as they may lead LLMs to retrieve information from webpages that create hallucinations. Second, we parse the GUI to identify modifiable elements and their range of values (e.g., sliders) as well as default values. These settings tell us how to interact with the model to generate simulations. The extracted static data provide essential context for the ABM, and we organize it as a JSON file, since the hierarchical format can organize diverse elements like documentation and GUI configurations.

The dynamic data consist of simulations that we run. Thanks to information provided from the model, we identify initial parameter values, the metrics to follow through the experiment, and the halting condition. For example, a parameter is the ‘number of agents: 1000’, a metric is to ‘track the average quantity of milk consumed by the agents’, and a halting condition is to ‘run for 73,000 time steps’. We run simulations by piloting a model externally (i.e., from Python 3.12.2), and we save the results into a CSV file, as it provides a standardized format for storing and analyzing structured data.

3.3. Report Generation by Prompt Engineering

The report is generated in three steps. First, we generate the context (Prompt 1): the purpose of the model, key parameters, and experimental conditions. Second, we describe each of the simulation runs via Prompt 2; that is, the prompt is called repeatedly to transform each dataset into a narrative. Note that these two parts do not have a one-to-one mapping with the static and dynamic features discussed in the previous subsection: the context tells readers about the parameters (static) and how they have been set for simulations (dynamic), while the results (dynamic) are interpreted in light of the model’s structure and aims (static). Third, the descriptions produced by the two prompts are unrestricted in length, but an executive summary needs to be concise. We thus perform post-processing by considering both abstractive and extractive summarization (as well as a case without summarization to generate full length reports) along with cleaning steps for punctuations, repeats, and other symbols. The three parts are detailed in this section.

3.3.1. Prompts and $2^3$ Factorial Design

The prompts can be formulated in several ways or include different types of content to support the LLMs in their task. The best options are not all known beforehand; thus, they need to be assessed experimentally (see Section 2.4). To this end, we systematically explored the search space of potential prompt elements by accounting for individual elements as well as their combinations. Our Design of Experiments includes two parts. First, our prompts contain three binary elements: we have two different choices for the role, example, and insight generation. The combinations are captured by a $2^k$ factorial design, where $k=3$ covers the three binary prompt elements. Second, our prompts contain one element with three levels rather than a binary: we can provide simulation outputs to the LLM as visuals, as statistics, or using both. We thus have a $2^k\times 3$ design where the three binary prompt elements are crossed with a three-level evidence factor. By recording the results for each of these combinations, we assess the effect of the features both independently and in interactions to (i) find the best prompt and (ii) assess which features contribute to prompt effectiveness.

In the following prompts, we use the format «element» to indicate a dynamic element whose value is populated based on the corresponding variable in the experimental design. For example, «Role» would be populated by one of two values for the binary role when the prompt is created. We use the notation «a|b|c» to indicate short options. The prompts are also populated based on the extracted data, which are shown as [data]. Supplementary Material S1 provides a sample instance for the prompts, illustrating how the dynamic elements are replaced by specific instructions.

Prompt 1 sets the task. The only prompt element changed by the experimental design is the role, which is also known as ‘persona’ [88,97]. By telling the LLM that it is an expert in a specific domain, we guide its tone, style, and type of response. We noticed that LLMs tended to go beyond the task of providing a context as they included summaries or conclusions, which were not only unnecessary but also occasionally erroneous since they did not account for simulation results. To prevent LLMs from making broad generalizations just based on the goal of a model and its parameters, we explicitly request not to write any summary or conclusion.

Prompt 2 describes the data from the simulation, one variable at a time, which is the appropriate place to make conclusions based on evidence. In addition to the role, this prompt has three dynamic elements. The example illustrates the style of an executive report for simulation models; a sample is provided in our Supplementary Material (page 8, “Here is an example of the style of the report:”). The insight adds “When summarizing trends, provide brief insights about their implications for decision makers.” In principle, a summary tells the reader what is there while insights explain why it matters. However, requesting insights might lead the LLM to hallucinate by seeing causality where the data does not support it (since LLMs are based on patterns in data rather than genuine comprehension [98]), or their tendency to prioritize interesting findings may give the summary a suddenly dramatic flair. Finally, we specify whether data are provided as visuals and/or as statistics, as explained below. Note that the literature has demonstrated the potential of generating insights via LLM when the data consist of a single table, the goals are clear, and the data are clean—as is the case in this paper. If the LLM is intended to navigate a database (e.g., multi-hop reasoning that joins databases), goals are unknown, and the data need fixing (e.g., missing values), then our workflow would need to be revised with an additional processing step. For example, a hypothesis generator can create potential insights [99], which are tested against the data by a query agent [100].

Simulation results were extracted as structured data (see Section 3.2), which leaves three options to pass them to the LLM: (i) provide the raw CSV files; (ii) provide visuals as time series; or (iii) provide statistical summaries. While LLMs can interpret results and generate insights, they are not designed for statistically rigorous analyses of numerical accuracy at scale, as a large amount of data may exceed their context size [101], and researchers continue to report issues in directly processing structured data [102]. Having ruled out a direct upload of simulation outputs per option (i), we focused on supporting the next two options. Note that they are not mutually exclusive: we can provide visuals, or we can provide statistics, or both. Uploading time-series images to an LLM can quickly extract high-level insights, especially when trends and anomalies are visually clear. While not as precise as statistical tools, the vision ability of LLMs can identify patterns and generate hypotheses efficiently, which addresses the need to communicate findings in an executive report. We thus supported option (ii) by automatically generating plots from the simulation data and providing them along with the prompt (Figure 5).

To support option (iii), we extracted seven categories of statistics as listed in

Table 1. Summary of statistical features extracted from the simulation time series.

Feature	Purpose	Method	Library	Output
Descriptive statistics	Summarize global properties of the signal	Direct statistical computation	NumPy/Python	Mean, median, variance, standard deviation, min/max values and indices, start/end value, counts of valid/dropped samples
Peaks (local maxima)	Identify salient positive events	Peak detection using neighbor comparison and prominence estimation	scipy.signal	Indices and values of peaks, with prominence
Valleys (local minima)	Identify salient negative events	Peak detection applied to the negated signal	scipy.signal	Indices and values of valleys, with prominence
Inflection points	Detect changes in curvature of the signal	Savitzky–Golay filtering and second-derivative sign analysis	scipy.signal	Indices where the second derivative changes sign
Local trends	Determine monotonic behavior in short segments	Rolling Mann–Kendall trend test with Sen’s slope estimator	pymannkendall	Start/end, Trend label, Kendall’s $\tau$, Sen’s slope, p-value, significance flag
Change points	Detect structural regime shifts in the signal	PELT change-point detection algorithm	ruptures	Indices marking boundaries between statistical regimes
Oscillatory regimes	Identify periodic behavior and dominant cycles	Continuous Wavelet Transform (Morlet wavelet)	pywt (PyWavelets)	Dominant oscillation period and indices where oscillation occurs

. The output of each simulation run is represented as a one-dimensional time series, where each entry corresponds to the value of the system state at a given discrete simulation tick. The signals from multiple simulations are aggregated by computing the tick-wise mean across runs; then, we analyze the aggregated signal using complementary methods drawn from signal processing, non-parametric statistics, and time-series segmentation in order to identify significant structural features such as extrema, inflection points, trends, regime changes, and oscillatory behavior. Basic descriptive statistics include the number of samples, mean, median, variance, standard deviation, and the locations and values of the global minimum and maximum. These summary statistics provide a coarse characterization of the signal’s scale and dispersion. Then, the peak detection algorithm evaluates each point relative to its neighbors and returns the indices of candidate maxima. Additionally, the peak prominence characterizes the magnitude of each peak. Valleys are detected by applying the same algorithm to the negated signal, which effectively transforms minima into maxima. These extrema provide a discrete representation of the signal’s most salient events. We also identify inflection points, which correspond to changes in the curvature of the signal, that is, when the second derivative of the signal changes sign, marking transitions between concave and convex behavior. These locations often correspond to turning points in the system dynamics. Since numerical differentiation tends to amplify high-frequency noise, we first smooth the signal using a Savitzky–Golay filter (which preserves information unlike smoothing by averaging), which performs a local least-squares polynomial fit within a sliding window (set to 10% of the data) and evaluates the fitted polynomial at the center of the window. We set the window size to 10% of the data with three constraints: we need at least 5 data points (to avoid windows that are too small to perform smoothing), we need windows with odd lengths (required by the Savitzky–Golay filter), and we clip to ensure that we stay within the data (e.g., the first two constraints could otherwise request more data points than are available). Because the polynomial representation is differentiable, the filter can compute derivatives directly. That is, the second derivative of the smoothed signal is computed directly from the fitted polynomial, and inflection points are detected by identifying locations where the sign of the second derivative changes.

To characterize directional changes in the signal, we use the non-parametric Mann–Kendall statistical test that evaluates whether a time series exhibits a monotonic upward or downward trend. The statistic is normalized to produce Kendall’s $\tau$, which measures the strength of the monotonic association between time and the signal value. To estimate the magnitude of the trend, the test also computes Sen’s slope estimator, which is a non-parametric estimate of the slope that is robust to outliers and does not assume normally distributed residuals. Since we are not interested in determining whether the entire time series follows a single global trend, but rather to identify local trend regimes, we apply the Mann–Kendall test in non-overlapping windows spanning 10% of the data. Again, the window sizes account for constraints such as having at least 5 data points. Windows containing constant values may lead to undefined statistical quantities in the Mann–Kendall test, so we detect and skip cases with fewer than two unique values, thus ensuring that the analysis does not produce misleading trend estimates. To identify abrupt transitions in the statistical structure of the signal, we perform change-point detection with the PELT (Pruned Exact Linear Time) optimization method. This method combines dynamic programming with pruning rules to discard sub-optimal solutions, which allows it to scale well with large datasets. Finally, we identify oscillatory regimes using Continuous Wavelet Transform (CWT). Traditional spectral methods such as the Fourier transform assume stationarity and therefore cannot localize oscillations in time, so we use wavelet analysis to project the signal onto a family of localized wavelet functions that are scaled and translated across the time axis. In this paper, a Morlet wavelet is used because its shape resembles a localized sinusoid and is therefore well suited for detecting periodic structures. An example of a plot and derived statistics is provided in Figure 6. Statistics were provided as a list of key:value pairs with one statistical information per line.

3.3.2. Post-Processing

In line with best practices, we ask the LLM to first generate a full response (i.e., without constraints on lengths); then, we summarize it. The responses can be particularly long, since each variable is either depicted as a visual and/or conveyed as a statistical summary, which are transformed into text. For example, if an ABM has 15 variables, then we create 15 CSV files (each comprising several runs), leading to 15 complete descriptions through Prompt 2. Our process thus consists of generating one output for the context, one output for the trend of each variable (e.g., from 10 to 14 trends depending on the ABM), concatenating all these outputs to create a full length report, and then using a summarization algorithm to produce the summary.

We considered four summarization algorithms, which cover the two main types of techniques: T5 and BART for abstractive summarization and Longformer and BERT for extractive summarization (see Section 2.3). Because of its extractive approach, BERT may combine ‘fragments’ to form sentences. This required further automated corrections including punctuation, spaces in parentheses, hyphens, or setting the first letter of each sentence to upper case. We also identified cases in which a summarization algorithm (e.g., LongFormerExt) produced loops of text, such as “This uniformity in the rate of choices made implies that the agents are continuously evaluating and possibly reevaluating their milk consumption choices, which are influenced by their perceptions, habits, and social interactions. The rate of change in the rate of change in the rate of change […]”. Our final step of post-processing was thus to remove repetitions, which were identified as consecutive identical sentence fragments.

3.4. Evaluation: Comparing LLM Outputs with Reports and Quantifying Readability

We evaluated the output of an LLM using six metrics. The Flesch Reading Ease computes the readability of the report [85], and it does not need a reference report. Reference reports are used to compute five scores (Section 2.3): BLEU [79,103], METEOR, ROUGE-1, ROUGE-2, and ROUGE-L [80,103]. We created four reference reports for each ABM, and all reports are provided for full disclosure in Supplementary Material S2:

• A summary was jointly written by the authors for each ABM to serve as a ground truth reference. We wrote the summary given the same information as was available to the LLMs, including the time series from each simulation run. These summaries were all written prior to generating text via LLMs, thus ensuring that the ground truth reference provided by our summary was not ‘contaminated’ by sentences that we may read in the LLM output. We refer to these summaries as ‘authors’ in our results and on the files provided in Supplementary Material S2.
• A summary was written for each ABM by independent associate professors with several peer-reviewed publications on agent-based modeling. That is, the summary for each ABM was written by a different modeler. To ensure consistency in the task, we created the same onboarding form for the modelers, varying only the content of the ABM. From hereon, these summaries are labeled as ‘modelers’ in our results and in Supplementary Material S2, which also contains the onboarding forms.
• We generated a full report and a summary as follows. We adapt the prompts shown in prior sections to rely on articles instead of the ABM documentation, and we use a state-of-the-art LLM that we did not use in our experimental setup (to avoid the output of an LLM being compared to itself) to generate the description of each plot. As a means to also demonstrate that reports could be produced without documentation from the ComSES.net repository or the NetLogo code, we used contextual augmentation by providing the peer-reviewed article for each ABM in the prompt. We corrected the descriptions of each plot to ensure factuality. Then, these descriptions were combined into a long report (note that it is the only material for comparison that is not a summary), which we call ‘long LLM’ from hereon. Finally, these reports were summarized into a short form using the same state-of-the-art LLM, resulting in the document named ‘short LLM’. We used OpenAI’s GPT 5.2 Thinking (released in December 2025) for this task. Our prompts and the LLM’s raw responses are shown in Supplementary Material S2, where our manual corrections are highlighted. For the fauna model, we made 6 corrections to the values reported in the curve, and we added two sentences on noticeable trends (out of 14 plots). For the milk model, we expanded two sentences and corrected fifteen values (out of 12 plots). No changes were needed for the grazing model.

3.5. Screening and Optimization Design

A commonly used Design of Experiments proceeds in two stages. First, a screening design can perform a large number of experiments (e.g., a $2^k$ factorial analysis with large k) to identify potentially important factors and interactions using a cost-efficient estimator. As a result, we may learn which factors can be fixed or dropped. For example, if we find that the summarization algorithm has a very large impact and that some algorithms offer clearly more compelling design choices, then we can set them accordingly. In the second stage, the optimization design (or ‘refinement stage’) then proceeds with a smaller search space by focusing on the remaining active factors, using a higher-cost estimator so that predictions are closely aligned with performances in deployment. Intuitively, the early stage offers a broad exploratory design that is only afforded through low-cost estimators, while the later stage focuses on a narrower search space with higher fidelity estimators. Our approach proceeds with these two stages as follows.

For the screening stage, we comprehensively examine a broad range of factors and their interactions using leading open-weight models (a large language model and a smaller model) as cost-efficient estimators. Each of these two models serves to examine 360 combinations of experiments corresponding to $2^3$ combinations of prompt designs (the binary prompt elements regarding role, insight, and example) across five summarization approaches (none, BART, BERT, T5, LongFormerExt) and three approaches to present evidence (as a plot only, as a statistical table only, or both) for the three ABMs. Each experiment is performed three times to account for stochasticity, that is, we have an experimental design with replications. Note that each run of an experiment further involves several calls to a model: the LLM is prompted once for the context and then once per plot over 10 to 14 plots depending on the ABM. Across the two models, we thus generate approximately $2\times 360\times 3\times 12\approx$ 25,920 prompts, where the 12 indicates the average number of variables encountered across ABMs in our case studies (see Section 4.1). As a result of the screening stage, we can comprehensively examine the influence of prompting elements, summarization approaches, and approaches to present evidence. Based on the experimental results, we thus set some of these choices to proceed with the second stage, in which we use the best available LLM based on current benchmarks to provide estimates of overall performance.

Note that our approach for a multi-step cost-efficient design echoes several practices in the LLM literature. Recent studies have indeed demonstrated the benefits of multi-stage cost-efficient optimization strategies. For example, we can start by prototyping and eventually culminate in using state-of-the-art models for deployment [104]. This idea is echoed by FrugalGPT [105] and LLM cascades, where the authors preserved performances but reduced costs by 60% by using cost-efficient LLMs to handle queries and only escalating them to a higher cost model when needed [106].

4. Results

4.1. Experimental Setup

We used four LLMs and three ABMs. The three ABMs were sourced from peer-reviewed publications created by different research groups and covering various topics in order to evaluate the ability of our approach across application domains. The ABMs had to be adequately documented in order to provide sufficient context for our LLMs to operate. Given these criteria, we used three ABMs from the ComSES.net repository:

• The Megafaunal Hunting Pressure Model (MHPM) by Ullah and colleagues (Figure 4), which was released in late 2024 to “understand how environmental dynamics and human predation preferences interact with ungulate life history characteristics to affect ungulate population dynamics over time” [107].
• The Milk Adoption ABM from Gibson et al. [108], which was designed to evaluate different mechanisms of consumer decision making in milk choice substitution to better understand and replicate historical trends in British milk consumption as a step toward modeling sustainable dietary shifts.
• The extended RAGE model [109], which explores how different types of learning (learning-by-doing and social learning) and varying attitudes toward risk influence smallholder farmers’ decisions about livestock management and the resulting outcomes for their livelihoods and the environment.

We chose four multimodal LLMs based on their state-of-the-art performances in their respective categories: proprietary LLMs, open-weight LLMs, and a smaller model. Gemini 3.1 Pro Preview was introduced by Google in February 2026. The Arena text and vision leaderboards (as of 4 March 2026) showed that Gemini 3.1 Pro Preview significantly improved over its predecessor, Gemini 3 Pro, as well as over competing models such as GPT-5.2 [110]; this trend was corroborated by other benchmarks [111]. It was considered to be the leading multimodal LLM at the time of its release on several benchmarks [112]. On the same benchmarks, Moonshot AI’s Kimi K2.5 (released in late January 2026) was the leading open-weight multimodal LLM [113]. Anthropic released Claude Opus 4.6 in February 2026 and also reported very strong performances across benchmarks [114]. Finally, Alibaba released the open-weight Qwen 3.5 in early 2026 [115], which quickly became a popular small language model in its 27B version, outperforming GPT 5 Mini [115]. All LLMs used the hyper-parameter settings recommended by the provider; e.g., we set the temperature to 1. Our experimental setup runs all LLMs through https://openrouter.ai/. For replicability, the libraries used in each part are summarized in

Table 2. Python libraries used including their role in our method.

Part	Use	Python Library
Data Extraction	Run simulation; export repeated runs	pynetlogo (0.5.2), pandas (2.2.3)
	Extract parameters, docs, and code from NetLogo files	re, json, xml.etree.ElementTree (3.12.2)
Report Generation	Plot repeated-simulation data	pandas (2.2.3), matplotlib (3.10.5)
	Build statistical table evidence	scipy (1.15.2), PyWavelets (1.8.0)
	Generate explanations	openai (2.15.0)
	Convert images to base64	base64 (3.12.2)
	Combine prompt features and sweep settings	itertools (3.12.2)
	Summarize source text	transformers (4.49.0), torch (2.10.0), sentencepiece (0.2.1), bert-extractive-summarizer (0.10.1)
Evaluation	Text/reference scoring	nltk (3.8.1), rouge_score (0.1.2), textstat (0.7.7)
	DoE/ANOVA analysis	pandas (2.2.3), scipy (1.15.2), statsmodels (0.14.2)

.

4.2. Screening Stage: Moonshot AI’s Kimi K2.5 and Alibaba’s Qwen 3.5 27B

4.2.1. Impact of Experimental Design Factors

Table 3. p-value from a one-way ANOVA between our six experimental variables (including the three dynamic prompt elements in the bottom rows) and our six quantitative metrics, where ‘R’ denotes ROUGE. Statistically significant p-values are shown in green; others are shown in red.

Reference Family	Variable/Metric	BLEU	METEOR	R-1	R-2	R-L	Reading Ease
author	Agent-based model	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
author	Summarization algorithm	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
author	Simulation evidence	0.05	0.71	<0.01	<0.01	<0.01	<0.01
author	LLM	<0.01	<0.01	0.02	<0.01	<0.01	<0.01
author	Use of roles	<0.01	<0.01	<0.01	<0.01	<0.01	0.74
author	Generating insights	0.68	0.48	0.02	<0.01	<0.01	<0.01
author	Providing examples	0.43	0.05	0.11	0.18	0.65	<0.01
long LLM	Agent-based model	<0.01	0.74	<0.01	<0.01	<0.01	<0.01
long LLM	Summarization algorithm	N/A	N/A	N/A	N/A	N/A	N/A
long LLM	Simulation evidence	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
long LLM	LLM	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
long LLM	Use of roles	<0.01	<0.01	<0.01	<0.01	<0.01	0.94
long LLM	Generating insights	0.01	0.07	<0.01	<0.01	<0.01	<0.01
long LLM	Providing examples	0.84	<0.01	0.27	0.56	0.80	<0.01
short LLM	Agent-based model	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
short LLM	Summarization algorithm	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
short LLM	Simulation evidence	0.84	0.64	<0.01	<0.01	<0.01	<0.01
short LLM	LLM	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
short LLM	Use of roles	<0.01	0.22	<0.01	<0.01	<0.01	0.74
short LLM	Generating insights	0.58	0.05	0.80	<0.01	<0.01	<0.01
short LLM	Providing examples	0.72	0.06	0.02	0.51	0.15	<0.01
modeler	Agent-based model	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
modeler	Summarization algorithm	<0.01	<0.01	<0.01	<0.01	<0.01	<0.01
modeler	Simulation evidence	0.08	0.30	0.16	<0.01	<0.01	<0.01
modeler	LLM	<0.01	<0.01	<0.01	0.52	0.08	<0.01
modeler	Use of roles	<0.01	0.77	0.02	<0.01	<0.01	0.74
modeler	Generating insights	0.91	0.43	0.13	<0.01	<0.01	<0.01
modeler	Providing examples	0.90	0.20	0.28	0.55	0.90	<0.01

reports the statistically significant effects of experimental variables across the evaluation metrics. Among all factors, the choice of summarization algorithm consistently exhibits statistically significant effects across every metric. This result is expected because except for the ‘long LLM’ baseline (which evaluates an unsummarized report against a full-length reference), the final text used for evaluation is the summary. Consequently, the summarization step directly determines the lexical structure, sentence selection, and level of compression of the evaluated output, which naturally affects metrics based on lexical overlap (BLEU, METEOR, ROUGE) as well as readability. The ABM and the LLM also frequently show statistically significant effects. This indicates that the results depend partly on the characteristics of the application domain and on stylistic differences between language models. In particular, the three ABMs differ in the number of variables, the complexity of their dynamics, and the domain vocabulary involved, which may influence how easily trends can be described and summarized. Likewise, different LLMs may vary in writing style, verbosity, and the interpretation of prompts, which can affect both lexical overlap metrics and readability scores. The type of simulation evidence provided to the model (statistics, plots, or both) has limited impact on BLEU and METEOR when operating on summaries, but it shows statistically significant effects on the other metrics. This suggests that while the form of evidence may not substantially alter the specific n-grams used in the summaries, it may influence how the LLM structures the explanation of trends and emphasizes particular aspects of the data, thereby affecting other evaluation measures.

In contrast, the prompt engineering factors tend to exhibit weaker or less consistent effects. Providing examples appears to be the least significant construct by itself. However, these results should be interpreted cautiously, as prompt variations may interact with other factors or influence aspects such as tone, framing, or narrative structure that are not fully captured by lexical overlap metrics. Interestingly, when comparing these results with Table S2 in Supplementary Material S1, we observe that newer LLMs appear to be sensitive to a larger number of prompt elements. In the earlier experiments conducted with GPT-4o, Claude 3.5 Sonnet, and DeepSeek R1 7B, none of the three prompt elements had statistically significant effects except on reading ease. In contrast, the newer LLMs used in this paper show significant effects for roles and insights on several ROUGE metrics. This suggests that recent models may be more responsive to prompt design, potentially reflecting improvements in instruction-following capabilities.

While the ANOVA analysis identifies which experimental variables have statistically significant effects on the evaluation metrics, it does not fully characterize the structure of these effects. In particular, statistical significance alone does not indicate the relative magnitude of each factor’s contribution to the variability of the results nor does it capture how factors interact with each other. This distinction is important in our setting because several components of the pipeline (e.g., prompt design, summarization algorithms, choice of LLM) may influence the final report jointly rather than independently. For example, the impact of a prompt element may depend on the summarization method used or on the LLM generating the text. To better understand these relationships, we therefore analyze the experimental design using a factorial decomposition of variance (Figure 7 and Figure 8). This approach quantifies the proportion of variance explained by each factor and by pairwise interactions between factors.

Given the results from the variance decomposition, we have four observations. First, the summarization algorithm is the most influential individual factor for lexical-overlap metrics (Figure 7), typically explaining around 9–15% of the variance depending on the reference material. This reflects the central role of the summarization step in shaping the final text evaluated by BLEU, METEOR, and ROUGE. Second, the largest effect overall arises from the interaction between the LLM and the summarization algorithm, which explains up to one third of the variance depending on the metric. This indicates that summarization performance depends strongly on the textual characteristics of the underlying LLM output, such as verbosity and narrative structure. Third, the individual prompt elements (examples, insights, and role) contribute relatively small proportions of variance, typically below 5% for lexical metrics, although their interaction terms are more visible for readability (Figure 8). Finally, the representation of simulation evidence and the identity of the LLM contribute moderate yet consistent effects, particularly for readability metrics, reflecting stylistic differences between models and variations in how simulation outputs are interpreted. Overall, these patterns are consistent across the different materials used as ground truth. These results indicate that the pairing of LLMs and summarization is essential in determining report quality.

4.2.2. Optimal Performances

What is best depends on how we look at it. We performed a one-way ANOVA and a Kruskal–Wallis test comparing the distributions of evaluation scores across the four types of reference materials (summaries by authors, modelers, or GPT; or a long report) and found a statistically significant effect (p < 0.01) from the reference family onto all metrics except reading ease. That is, how to create the ‘best report’ depends on the material against which it is judged. Since the previous section revealed the importance of the summarization algorithm, we also examined whether a best algorithm could systematically be found. That is, we assessed the stability in ranking of summarization algorithms across contexts defined by the type of reference material, ABM, and LLM. While ranking stability was high for METEOR (mean Kendall’s $\tau$ = 0.76) and reading ease (mean $\tau$ = 0.85), it was low for BLEU and ROUGE metrics (mean $\tau$ from 0.16 to 0.26), which means that further analyses are needed to identify a best summarization method.

We combined three complementary screening analyses to identify promising configurations (

Table 4. Retained vs. discarded factor levels after cross-run screening analyses. Decisions are based on aggregated ANOVA significance rates, factorial involvement shares, and cross-run level dominance comparisons.

Factor	Retained Level (s)	Discarded Level (s)	Rationale
Evidence	plot	Table, plot + table	Dominates lexical metrics across runs; most consistent winner
Example	True	False	Systematically improves BLEU and ROUGE metrics
Insights	False	True	Turning insights off improves most lexical metrics and readability
Role	False	True	Role instructions reduce performance in most comparisons
Summarizer	BERT, BART, T5	longformer_ext	BERT wins overlap metrics; T5 wins readability; BART wins R-L

):

• We aggregated ANOVA results across evaluation metrics and reference formats to determine how often each experimental factor shows a statistically significant effect. The goal is to establish whether a factor reliably alters evaluation outcomes. For example, the choice of summarization algorithm was statistically significant for all metrics and across all reference summaries. This indicates that changing the summarizer systematically changes report quality, making it a primary factor that would be varied during our next stage of optimization.
• We aggregated factorial contribution tables that quantify how much variance each factor explains when both main effects and pairwise interactions are considered. For instance, the role instruction does not always appear dominant in the ANOVA tables, yet its interaction with the summarization algorithm or with the LLM can account for a substantial portion of performance variability. This indicates that the effect of role is conditional: it shapes outcomes indirectly by modifying how other components operate.
• To translate statistical influence into actionable design choices, we conducted direct level-vs.-level comparisons (Figure 9). For each factor, we compared competing levels under otherwise identical experimental conditions (same ABM, LLM, evidence format, prompt configuration, and replication). We then counted how often each level produced the best evaluation score.

We define the ANOVA significance rate of a factor as the proportion of evaluated metric–reference combinations in which that factor is found statistically significant (at $\alpha =0.05$) across the experimental runs. For example, a value of $0.74$ means that the factor showed a statistically significant effect in 74% of the situations where we tested it. On this basis, the role prompt has moderate significance rates in the aggregated ANOVA analysis (mean ≈ 0.74), but the aggregated factorial analysis reveals that its contribution is largely interaction-driven rather than consistently dominant as a main effect. This conditional influence is reflected in the level-competition analysis, where disabling the role instruction wins all three configurations for all overlap-based metrics. Including a role thus does not reliably improve performance and may even introduce variability depending on the model or summarization approach. We thus removed the role for the optimization stage.

Having an insight has small-to-moderate statistical importance (mean ANOVA significance rate ≈ 0.57), and the aggregated factorial analysis indicates non-negligible interaction contributions. However, level-competition results reveal a clear directional pattern: not asking for insight wins all replications for ROUGE-1, ROUGE-2, ROUGE-L, and readability, while it only improves METEOR. Consequently, we also removed the insight component of the prompt for the optimization stage.

Unlike role and insights, the inclusion of an example shows a weaker statistical presence in the aggregated ANOVA tables (mean significance rate ≈ 0.24), indicating that it is not always detected as a dominant main effect. However, factorial contribution summaries confirm that examples still participate in performance variation through interactions. Most importantly, the level-competition analysis provides strong practical evidence: including examples wins all three replications on BLEU, METEOR, ROUGE-1, and ROUGE-2; it wins two out of three for ROUGE-L; and disabling examples only improves readability. Therefore, we kept the examples for the optimization stage.

Through direct level-vs.-level comparisons, we noted the clear practical dominance of plot-based inputs for lexical metrics: plots win all three replications for BLEU, ROUGE-1, ROUGE-2, and ROUGE-L, while table-based or combined formats rarely outperform them. Although combining plots and tables sometimes improves readability, plot-only inputs yield the most consistent gains on the reference-based evaluation metrics that guide optimization. For this reason, we retain plot-based evidence presentation for optimization.

4.3. Optimization Stage: Anthropic’s Claude Opus 4.6 and Google’s Gemini 3.1 Pro Preview

We evaluated the performance of two additional state-of-the-art LLMs (Anthropic’s Claude Opus 4.6 and Google’s Gemini 3.1 Pro Preview) using the experimental configuration retained after the screening stage (Table 4). To compare models in a practically meaningful way, we applied the same level-vs.-level dominance analysis used in the screening stage. For each metric and experimental context (reference format, application domain, summarization method, and repetition), we identified which LLM achieved the best score and computed the proportion of contexts in which each model ‘wins’. This approach allows us to assess robust performance dominance. Claude Opus 4.6 achieves the highest win rates across all metrics, while Gemini 3.1 Pro Preview shows comparatively lower dominance frequencies (Figure 10). Despite performances on general benchmarks, we note that performances from Gemini 3.1 Pro Preview are generally low in contrast to other LLMs for our application domain. Overall, these results suggest that the most robust LLMs for deployment would be either the proprietary LLM Claude Opus 4.6 (i.e., at a cost) or the open-weight Qwen 3.5, which can either be instantiated locally (on a professional workstation with sufficient memory) or through an API in a cost-efficient manner.

Figure 11 reports the distribution of scores achieved by each LLM across all experimental contexts. Consistent with Figure 10, Claude Opus 4.6 exhibits the strongest overall performance profile, combining high central tendencies with favorable upper bounds on lexical-overlap metrics. For example, Claude reaches maximum ROUGE-2 values close to 0.22 and ROUGE-1 values around 0.58 while maintaining comparatively high mean scores relative to other models. Qwen 3.5 shows a similar but slightly weaker pattern, achieving competitive maximum lexical-overlap scores and comparable variability ranges. In contrast, Gemini 3.1 Pro Preview displays systematically lower mean lexical-overlap performance and wider performance dispersion across contexts, indicating less consistent alignment with reference summaries. Differences are even more pronounced on readability, where abstractive generation capabilities lead to substantial performance gaps between models.

To complement the quantitative comparisons, we conducted a qualitative reading of reports drawn from a stratified sample covering all LLMs, all ABMs, both extractive and abstractive summarization approaches, and varied prompt configurations (

Table 5. Number of factual errors (#errors) and issues in a sample of summaries selected across combinations of LLMs, ABMs, extractive/abstractive summarizer, and other prompt parameters.

LLM	ABM	Prompt	Summarizer	#errors	Notes
Claude	Fauna	Plot, Examples	BERT	5	Lacks transitions between trend descriptions. Abrupt transitions from describing one graph to another, which confuses the reader regarding which metric is described. Overestimates some values or ticks. The report describes variable per variable rather than chronology. The report is overall very accurate but omits details.
	Grazing		BART	1	Confuses some values set to certain metrics. There is more consistency in the way metrics are introduced or the way transitions are made. Sentences make sense and the reader does not become confused regarding which metric is being described. It is better self-contained even though it misses some information. A bit too concise.
Gemini	Milk		T5	2	Very short report, missing a lot of information. Misses some numbers and lacks clarity on the exact metrics described. Poor transitions.
	Fauna		BERT	5	Many numbers missing. Many parameter values wrongly reported. Overestimates some values.
Qwen	Grazing	Plot + Table	BART	0	Very short and not informative enough. Omits a vast amount of information. Its formulations and trend descriptions do not fully support understanding.
	Milk	Table, Insights, role	LongFormer	1	Juxtaposition of sentences which makes it really hard to understand and contextualize which metric is being described. Poor coherence and narrative. The report is very short and not very informative.
Kimi	Fauna	Plot + Table, Insights, role	BERT	7	Some numbers missing. Hallucinated the number of simulations. Describes the trend of a metric without ever mentioning that metric, thus confusing the reader in thinking it still talks about the previous metric. Abrupt transitions.
	Grazing	Table, Examples, insights, role	T5	4	Imprecise on the specific metrics described. Does not present the parameters. Omits information. Describes the trend of a metric without ever mentioning that metric, thus confusing the reader by thinking it is still talks about the previous metric.

). A consistent pattern emerges when examining both narrative quality and factual reliability. Extractive summaries produced with BERT show the highest number of identified issues, ranging from five errors for Claude and Gemini on the fauna model to seven errors for Kimi under similar conditions. The errors include missing numerical values, abrupt transitions between variables, and ambiguity about which metric is being described. Since such problems occur across multiple LLMs and application domains, it suggests that they are primarily driven by the extractive summarization mechanism rather than by model-specific behavior. In contrast, abstractive summarizers such as BART and T5 generally exhibit fewer factual inconsistencies (e.g., one error for Claude with BART on the grazing model and two errors for Gemini with T5 on the milk model) and produce more coherent, self-contained narratives. While abstractive summaries can sometimes be overly concise or omit secondary details, they rarely display the fragmented sentence juxtaposition characteristic of extractive outputs. These qualitative findings are aligned with expectations: extractive approaches tend to preserve isolated textual fragments at the expense of narrative continuity and factual completeness, whereas abstractive approaches more consistently support intelligible executive-style communication of simulation outcomes.

As the most robust LLM, we focus the analysis on Claude Opus 4.6 across ABMs (Figure 12) while noting that our storage bucket provides the results for all LLMs. Results depend on the ABM, which is expected, as each model is built differently, involves a different vocabulary, and produces different dynamics. On lexical-overlap metrics, the milk consumption ABM systematically yields the strongest scores. For instance, with the BART summarizer, BLEU reaches approximately 0.058 for milk consumption compared with 0.033 for fauna and 0.022 for grazing (Figure 12a). We see similar effects for ROUGE metrics: ROUGE-2 peaks near 0.175 for milk consumption using BART compared with about 0.110 for fauna and 0.116 for grazing (Figure 12b), while ROUGE-1 reaches roughly 0.50 under the same configuration (Figure 12e). Reading ease behaves (Figure 12f) differently from lexical-overlap metrics, as we observe that the T5 summarizer consistently produces the highest scores across ABMs (about 39 to 44), whereas BERT produces substantially lower readability (from 9 to 18). The standard deviations across runs are modest relative to mean differences, indicating that these performance patterns are stable. As the results indicate that BERT’s extractive summaries tend to sacrifice fluency and lexical alignment without delivering consistent gains on overlap metrics, we recommend the two abstractive summarization algorithms BART and T5 for deployment.

5. Discussion

5.1. Key Contributions

This paper demonstrates that large language models can be systematically configured to translate well-documented agent-based models (ABMs) into executive-style reports that are quantitatively aligned with reference summaries and exhibit qualitative characteristics consistent with texts intended for non-technical audiences. As detailed in Section 5.3, we note that examining practical usability for such audiences is a study in itself, which would require ethical approval, enrollment of participants, and specific tools to evaluate how people react to text. A central contribution of this paper is the introduction of a staged Design of Experiments (DoE) pipeline for optimizing complex generative reporting systems. Rather than directly tuning high-capacity proprietary models, we first conduct a screening stage using lower-cost or open-weight LLMs to identify influential design factors, quantify interaction effects, and eliminate weak configurations. Aggregated ANOVA significance rates establish which components reliably influence evaluation outcomes, while factorial variance decomposition reveals that the dominant drivers of report quality are the summarization strategy and its interaction with the underlying LLM. Direct level-dominance comparisons then translate statistical influence into actionable configuration choices, enabling a substantial reduction of the experimental search space prior to deployment-stage evaluation.

Beyond methodological advances, the results provide practical insights into the construction of LLM-based reporting pipelines. We show that abstractive summarization consistently supports higher readability and more coherent narrative structure than extractive approaches, which tend to preserve fragmented textual elements that reduce the practical usability of generated reports. We also demonstrate that prompt refinements play a secondary but still actionable role once the core LLM–summarizer pairing is fixed with evidence representation and example-based guidance influencing lexical-alignment metrics more than stylistic prompt elements such as role instructions or insight requests. Overall, the proposed DoE framework illustrates how classical experimental-design principles can be adapted to systematically engineer reliable generative-AI workflows in resource-constrained settings.

5.2. Practical Implications

The optimization stage provides clear practical recommendations for deploying LLM-based reporting pipelines. Among the evaluated models, Claude Opus 4.6 demonstrates the most robust performance profile, combining the highest dominance rates across evaluation metrics with favorable score distributions and limited variability across experimental contexts. At the same time, Qwen 3.5 emerges as a competitive open-weight alternative, offering comparable lexical-overlap performance with lower operational costs and the possibility of local deployment. Prompt configuration also converges toward a stable design: presenting simulation evidence through plots and including illustrative examples consistently improves lexical-alignment metrics, whereas role instructions and additional insight-generation steps introduce variability without delivering systematic performance gains. These findings suggest that once an appropriate summarization strategy and LLM are selected, modern models require relatively limited prompt steering to produce high-quality executive reports.

Performance variations across ABMs further highlight the importance of domain characteristics in generative reporting. Lexical-overlap metrics are consistently higher for the milk-consumption model than for the fauna or grazing models, likely reflecting differences in vocabulary specificity, variable complexity, and the clarity of emergent simulation trends. These domain effects indicate that optimal reporting configurations cannot be fully decoupled from the structure and documentation quality of the underlying simulation. They also underscore the need to evaluate generated reports against multiple forms of reference material, as the perceived summary quality may depend on whether outputs are compared with expert-written summaries, long-form technical descriptions, or alternative machine-generated reports. Overall, these findings suggest that generative reporting pipelines should be configured with sensitivity to both model characteristics and application-domain features.

5.3. Limitations and Future Work

Despite the promising results, several limitations should be acknowledged. First, the proposed reporting pipeline assumes that agent-based models are accompanied by sufficiently detailed and structured documentation. Solidly anchoring generated summaries in factual content requires that the task remains primarily translational rather than speculative, as insufficient contextual grounding may otherwise lead to hallucinations or omissions. The simulation community has repeatedly emphasized the importance of thorough model documentation [36], and repositories such as CoMSES provide many well-documented examples. However, there is no guarantee that any ABM intended for stakeholder communication will meet this standard. In practice, weaker documentation may lead to identifiable failure modes such as missing descriptions of key variables, an incorrect attribution of causal mechanisms, reduced narrative coherence, or overly generic summaries that fail to capture model-specific dynamics. Conducting systematic stress tests across graded documentation regimes (e.g., rich, medium, sparse) would require constructing controlled degraded documentation variants for multiple models, which was beyond the scope of this paper but represents an important direction for future work.

Second, while the staged Design-of-Experiments approach substantially reduces computational cost and enables principled configuration selection, conclusions about optimal configurations remain contingent on the selected ABMs, evaluation metrics, and prompt design space explored. Performance differences observed across application domains suggest that generative reporting effectiveness depends partly on model structure, vocabulary specificity, and the clarity of emergent dynamics. Future research should therefore investigate how configuration choices generalize across broader classes of simulation models and whether the automatic characterization of model features (e.g., number of state variables, degree of stochasticity, interaction topology) can guide the selection of appropriate LLM–summarizer combinations. Empirical evidence indicates that related ABMs often share common “building blocks” [116]— for example, when simulating mobility or interaction processes; automatically detecting these structures could help reconstruct missing contextual explanations and improve report grounding.

Third, the qualitative evaluation relied on targeted sampling rather than an exhaustive inspection of all generated reports. Although recurring error patterns across models and domains increase confidence in the observed trends, larger-scale human evaluations would further strengthen conclusions regarding factual faithfulness and narrative usability. Prior work has highlighted the challenges of consistently assessing conceptual-model summaries and relating textual descriptions to simulation structure [117], suggesting that evaluation protocols tailored to simulation reporting remain an open research need.

Finally, an important direction for future work is the mitigation of documentation limitations through external knowledge integration. Retrieval-augmented generation (RAG) could incorporate information from research articles, repositories, or related simulation models to provide additional contextual grounding. Complementary approaches include clone-detection techniques that recognize similarities with well-documented models, allowing a partial reuse of validated documentation, although care must be taken to ensure that only applicable elements are transferred. As large language models continue to evolve and support longer context windows, further studies are needed to determine how additional evidence sources such as code comments or structured metadata can be incorporated without degrading narrative clarity or increasing sensitivity to prompt configuration.

6. Conclusions

This paper demonstrates that large language models can be systematically configured to translate simulation outcomes from well-documented agent-based models into concise, executive-style reports that remain aligned with reference summaries while exhibiting readability and narrative properties that may support comprehension by non-technical audiences. Across multiple ABMs and LLMs, our results show that report quality depends mostly on the choice of summarization strategy and underlying LLM with abstractive summarizers such as BART and T5 providing more coherent and practically useful narratives than extractive alternatives. This paper suggests that automated simulation-to-text can help bridge the gap between complex simulation outputs and decision support, while future research should examine how these configuration choices generalize to less-documented ABMs and broader classes of simulations.