Constrained LLM Reporting for Geospatial Climate Risk: A One-Shot In-Context Framework for Critical Infrastructure
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe topic is interesting. Below are some suggestions.
- The study relies heavily on automated NLP evaluation metrics, a human expert blind review or stakeholder utility evaluation is urgently needed to validate whether the AI-generated narratives truly offer clear readability and practical decision support.
- The experimental results show that the verifiable support rate is highly sensitive to the chosen judging LLM (marked swings under Llama 3.1 versus a flat trend under GPT-5.4 mini). The authors should discuss how this “judge bias” impacts evaluation credibility and how to standardize this metric.
- An excess of “neutral narratives”unsupported by raw data may inject unwarranted optimism or irrelevant implications, and their proportion should be formally constrained.
- In Section 2.2, the technical phrasing regarding the WGS84 sphere centroid and spherical excess formulas feels slightly disconnected.
- The authors should clarify the spatial-temporal resolution boundaries and uncertainty propagation when aligning the NASA-IBM climate foundation model with European high-resolution projection suites.
- The introduction and literature review sections lack a detailed contrast with existing framework benchmarks.
- The conclusion is currently weighted toward summarizing NLP technical metrics,it can add the description of practical engineering value.
- During the framework’s modification and fitting stages, certain parameters were adjusted to reach acceptable performance indices.Clarify the precise theoretical or operational support for these specific tuning choices.
Author Response
Dear Reviewer,
We sincerely thank you for this insightful and constructive critique. Your feedback has heavily guided the refinement of our evaluation methodology and framework boundaries. Below is our response following your layout questions:
Comment 1: The study relies heavily on automated NLP evaluation metrics; a human expert blind review or stakeholder utility evaluation is urgently needed to validate whether the AI-generated narratives truly offer clear readability and practical decision support.
Response 1: This is an important point we agree with, as human validation is crucial to complement automated metrics. Therefore, during the revision window, we designed and executed a blind expert evaluation panel with six researchers to rate a randomized, anonymized subset of our reports across four quality dimensions. The findings largely support our automated metrics: on the Hazard Report, the with-exemplar configuration scored significantly higher on factual accuracy (Mann–Whitney U, p = 0.041) and completeness (p = 0.039), with five of six raters agreeing on the completeness direction (sign test excluding ties, p = 0.031). For the Potential Risk Index (PRI) report, the effect was directionally positive but more modest across three of four dimensions. We have added these new metrics alongside a careful discussion of panel limitations (such as absolute inter-rater agreement bounds and sample constraints) to keep our claims appropriately scoped and realistic.
These changes can be found in Section 2.5 (new methodological subsubsection), Section 3.5 (new dedicated results section, including Table 7), and Section 4.3 (expanded limitations framing). Coordinated updates were also made to the Abstract and Section 4.1.
Comments 2: The experimental results show that the verifiable support rate is highly sensitive to the chosen judging LLM (marked swings under Llama 3.1 versus a flat trend under GPT-5.4 mini). The authors should discuss how this “judge bias” impacts evaluation credibility and how to standardize this metric.
Response 2: Thank you for highlighting this substantive methodological point. We agree that the divergence between our Llama 3.1 8B and GPT-5.4 mini judges provides direct empirical evidence of judge sensitivity in automated evaluation. In response, we have added a dedicated analysis in our discussion section exploring how this bias impacts evaluation credibility. Rather than treating it as a standard flaw, we utilize it to suggest a multi-layered standardization framework for the community which includes pairing automated judges with judge-independent token counters, evaluating across varied model tiers, and permanently pinning model versions for long-term consistency.
These changes can be found in a new paragraph at the end of Section 4.1 (Applied Contribution), with explicit cross-referencing added to Section 3.4 (Cross-Model Replication).
Comment 3: An excess of “neutral narratives” unsupported by raw data may inject unwarranted optimism or irrelevant implications, and their proportion should be formally constrained.
Response 3: We agree with the concern that an unconstrained proportion of neutral or unverifiable text can introduce narrative drift. In this revision, we clarify that the share of neutral text serves an important purpose (providing necessary stakeholder framing), but must be closely monitored. Our ablation study indicates that our full-prompt configuration is associated with shorter and more focused text outputs compared to the ablated configurations. Furthermore, we explicitly discuss a future engineering roadmap to convert this evaluation-time metric into a live, deployment-level quality check that flags any report where the unverifiable-claim share exceeds a calibrated threshold, allowing the operator to review or regenerate.
These changes can be found as new paragraphs in Section 2.5 (under metric definitions) and Section 4.2 (Practical Implications).
Comments 4: In Section 2.2, the technical phrasing regarding the WGS84 sphere centroid and spherical excess formulas feels slightly disconnected.
Response 4: We agree that the original text felt disconnected and lacked a clear real-world motivation for choosing spherical geometry. We have completely rewritten this paragraph to explicitly bridge our technical choices with our geographic constraints: since the platform must fluidly process sites spanning a wide latitude range across Europe (35°N to 70°N), standard flat projections introduce severe distortion at higher latitudes that breaks density metrics. Computing centroid coordinates and spatial areas directly on a WGS84 sphere entirely bypasses this issue.
These changes can be found in the revised site-characterization paragraph in Section 2.2.
Comment 5: The authors should clarify the spatial-temporal resolution boundaries and uncertainty propagation when aligning the NASA-IBM climate foundation model with European high-resolution projection suites.
Response 5: Thank you for raising this point, which helped us catch an important descriptive inaccuracy. In verifying our data backend with the API developer, we confirmed that our visualization engine directly queries native high-resolution EURO-CORDEX projections rather than data derived from the NASA-IBM Geospatial Foundation Model (which had been carried over from a legacy project blueprint). We have corrected this description across the relevant places in Section 2.
Additionally, we expanded Section 2.3 to map out the explicit spatial-temporal boundaries: our engine serves multi-model ensemble statistics over a 12.5 km resolution grid (EUR-11) across short-, medium-, and long-term 30-year horizons under both RCP4.5 and RCP8.5. Lastly, on uncertainty propagation, we added an open acknowledgement to our limitations section: while the API returns full distribution values (), our current platform only translates the ensemble mean into its discrete indexing logic, identifying end-to-end distribution propagation through the whole indexing chain as critical future work.
These changes can be found in the Section 2 introductory overview, the heavily expanded Section 2.3 technical breakdown, and a new paragraph in Section 4.3 (Limitations).
Comment 6: The introduction and literature review sections lack a detailed contrast with existing framework benchmarks.
Response 6: We agree that our original literature review relied on a purely qualitative contrast. To provide a sharper, more structured differentiation, we have introduced a new synthesis matrix to the Introduction (Table 1). This benchmark explicitly pits our Grounded Reporting Framework (GRF) against leading climate-adjacent architectures (ChatClimate, CHATREPORT, and ClimateBert) across four clear design dimensions, showing that our approach is specifically designed to transform complex, multi-layered tabular metrics from an analytical chain directly into non-technical narrative reports.
These changes can be found in the newly added Table 1 and its supporting introductory prose at the very end of the Introduction section.
Comment 7: The conclusion is currently weighted toward summarizing NLP technical metrics; it can add the description of practical engineering value.
Response 7: Thank you for this excellent recommendation. We fully agree that the original conclusion relied too heavily on text-processing numbers. We have added a dedicated paragraph focusing entirely on the real-world operational value this platform offers to engineers and asset managers. We emphasize three main practical benefits: delivering fully generated reports in under a minute per site, enabling interactive stakeholder communication rather than analyst-drafted narratives, establishing programmatic post-generation quality checks via numeric token counters, and ensuring cross-model portability so operators can maintain secure local deployments without proprietary API lock-in.
These changes can be found in the newly expanded final paragraphs of Section 5 (Conclusions).
Comment 8: During the framework’s modification and fitting stages, certain parameters were adjusted to reach acceptable performance indices. Clarify the precise theoretical or operational support for these specific tuning choices.
Response 8: We clarify that no parameter tuning was performed against any of the faithfulness metrics reported in this paper. We have added a clarifying sentence to Section 2.4 stating explicitly that the sampling temperatures and embedded exemplar pair are fixed properties of the deployed configuration, chosen on operational grounds prior to and independently of the faithfulness evaluation, and not tuned against any of the reported metrics. The PRI Assessment Report uses the lower temperature (T = 0.3) because its outputs benefit from run-to-run reproducibility, while the remaining reports tolerate stylistic variation (T = 0.7).
These changes can be found via a new clarifying sentence in Section 2.4.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe paper presents a LLM-based framework for generating narrative reports from quantitative climate-risk assessment results. The work is motivated by the fact that the outputs of climate-risk assessment tools may be technically complex for non-expert users, thus creating a communication gap between analyses and decision-making. The introduction provides sufficient context for the proposed work by presenting the application domain of a climate risk assessment platform based on open climate and infrastructure data, retrieved through online services, aiming at supporting decision-making process related to nature-based solutions. This application context could be further elaborated to better highlight the need for the proposed reporting framework in decision making processes.
The authors use their own platform to experimentally evaluate the automatic report generation using different LLMs.
The platform certainly provides a relevant application context, but the paper focuses on the technical design and evaluation of the LLM-based reporting modules. Therefore, although the introduction mentions the problem of supporting the integration of nature-based solutions into climate-risk decision making, the actual contribution is on the design and evaluation of LLM-based reporting tools. To be coherent and better aligned with the Special Issue topics, I would suggest the authors to revise the Introduction and further emphasize how such a framework could support NbS-related decision making processes in the paper.
Overall, the work is well presented and clearly structured. The techniques for use of LLMs are well known (strict instructions, one-shot in-context learning, and low-temperature decoding). However, the paper proposes an interesting and carefully designed evaluation methodology, combining complementary faithfulness metrics, ablation experiments, and cross-model validation. As the paper is not a scientific contribution to LLM research/experimentation (as also the authors state), to my opinion its exploratory evaluation and integration within a climate-risk assessment platform make it interesting to the readership of this journal.
The authors have clearly defined the scope and objectives of the experimental evaluation. However, the following aspects should also also be considered for generalizability: 1. The current evaluation is performed on relatively simple structured inputs (e.g., risk levels). The authors could discuss applicability to more complex structured outputs (e.g., simulation results or scenario-based analyses). 2. Since the target domain is critical infrastructure, data confidentiality could be an issue when using non-public infrastructure data.
Author Response
Dear Reviewer,
We are deeply grateful for your positive assessment of our work and your thoughtful understanding of our core narrative translation layer. Your insights regarding Generalizability and Data Confidentiality have helped us a lot. Please find the responses to your comments below:
Comment 1: This application context could be further elaborated to better highlight the need for the proposed reporting framework in decision-making processes... I would suggest the authors revise the Introduction and further emphasize how such a framework could support NbS-related decision-making processes in the paper.
Response 1: We fully agree that framing our reporting tool within real-world nature-based solution (NbS) deployment adds significant value to the paper. We have revised the Introduction to establish a smooth conceptual bridge: as conventional gray infrastructure cracks under climate extremes, engineers need to confidently communicate alternative NbS strategies. Our framework acts as this exact enabling tool, translating dense quantitative risk scores into clear, human-readable explanations that help non-technical stakeholders trust, approve, and fund nature-based adaptation investments.
These changes can be found in the rewritten second paragraph of the Introduction, with supporting operational expansions added to Section 4.2 (Practical Implications) and the final paragraph of Section 5 (Conclusions).
Comments 2: The current evaluation is performed on relatively simple structured inputs (e.g., risk levels). The authors could discuss applicability to more complex structured outputs (e.g., simulation results or scenario-based analyses).
Response 2: This is a highly perceptive comment regarding the generalizability of our framework. We agree that our current automated evaluation relies on moderately sized tabular index metrics. To explore complex data structures openly, we have added a new analysis to our limitations block. We discuss how high-dimensional datasets such as gridded raster simulation models, multi-scenario comparative timelines, or deep time-series projections would require specialized input serialization layers (such as chart-to-text translators or scenario-by-scenario sequential chunking blocks) before they can be effectively processed by our framework, marking this as an exciting path for future development.
These changes can be found in a new, dedicated generalizability paragraph inside Section 4.3 (Limitations).
Comment 3: Since the target domain is critical infrastructure, data confidentiality could be an issue when using non-public infrastructure data.
Response 3: Thank you for your comment. To address this directly, we have added a data security statement to the manuscript clarifying that operators handling sensitive infrastructure data can bypass the LLM-based narrative reporting layer entirely and use only the quantitative analytical outputs of the tool, with the alternative option to self-host the generation layer for narrative creation using open-weight models (such as our replicated Llama 3.1 setup). This completely guarantees that sensitive localized asset parameters remain fully inside the operator's private environment. It is important to note that in the decision support tool, generating reports is not mandatory.
These changes can be found in a new data confidentiality block inside Section 4.3 (Limitations).
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsMy Concerns have been resolved.
