Evaluating ChatGPT for Dam Behavior Prediction, Interpretation, and Safety Assessment: An Experimental Study

Abstract

Structural dam safety control is fundamentally based on visual inspections and on the analysis of monitoring data and their comparison with the expected behavior for a given load combination, which in turn is calculated with a predictive model. In recent decades, the development of machine learning algorithms has led engineers and researchers worldwide to explore their capabilities for predicting dam behavior and supporting safety assessment. Regardless of the model used, a prediction is generally obtained together with an expected model error, which is compared with the observed value in order to support decision-making. This process, however, requires the intervention of experts in dam engineering, and particularly in the prior behavior of the structure under analysis. In recent years, the emergence of generative artificial intelligence models has opened up considerable opportunities to improve processes in various fields, including civil engineering. The reported capabilities of these models make them well suited to some tasks in the field of dam safety, particularly for developing predictive models and interpreting their results. In this work, we tested ChatGPT to assess its potential in these tasks, using the data from the ICOLD 2022 Benchmark Workshop, in which 18 teams provided their best predictions of the behavior of an arch dam. ChatGPT was provided with the same problem description and monitoring data available to the participants and was asked, through brief and neutral prompts, to propose a modeling strategy, generate predictions for six response variables for short- and long-term periods, create the corresponding source code, and interpret the results from a dam safety perspective. The complete procedure was performed twice using the same information and prompts. The results show considerable potential for increasing productivity in building and applying machine learning models, with predictive accuracy comparable to that obtained by the Benchmark participants. ChatGPT achieved a lower mean absolute error than the participants’ average in 10 (first run) and 12 (second run) of the 12 variable–period combinations, and outperformed the best participant in one case. However, the interpretation of the results was considerably weaker, with important aspects being only partially addressed or overlooked. These findings highlight the potential of ChatGPT to assist in the development of data-based predictive models, but also reveal its limitations in terms of the domain-specific knowledge required to draw conclusions on dam safety.

1. Introduction

According to the latest data published by the International Commission on Large Dams (ICOLD) [1], there are 62,362 large dams worldwide. They play a key role in modern societies, as they are essential for water supply and irrigation, flood protection, and power generation.

Most of these dams were built in the mid-20th century, and a significant portion of them are now reaching the end of their expected service life. Regardless, their aging entails increasing maintenance requirements. A recent report highlights a rise in the number of dam-related accidents and directly links this trend to the age of the structures [2]. There is, therefore, strong interest in improving processes related to dam safety—both to reduce the occurrence of accidents and to optimize maintenance operations.

Structural health monitoring (SHM) constitutes an important component of these processes. In dams, SHM involves the systematic observation and analysis of structural and environmental variables to characterize the condition and performance of the structure, identify anomalous behavior, and provide information for safety assessment and maintenance decisions. Monitoring systems may include measurements of displacements, strains and stresses, seepage and uplift pressures, temperatures and joint movements, among other quantities [3,4,5]. Traditionally, these measurements have been complemented by periodic visual inspections, while advances in sensing, data acquisition and communication technologies have progressively enabled continuous and automated monitoring. More recently, techniques such as remote sensing, computer vision and UAV-based photogrammetry have further extended the spatial coverage of monitoring and inspection, particularly for areas that are difficult or unsafe to access [6].

At the core of dam safety monitoring activities lies a behavioral model. This model provides the expected response of the dam, which is then compared to the measurements obtained from monitoring instruments. A certain degree of discrepancy is tolerated, associated with the quality of the data, the accuracy of the model, or even the current understanding of the phenomena under study. If the observed value of a given variable deviates significantly from the prediction, safety control measures are implemented.

Significant advances have been made in the predictive capability and versatility of these models. This applies to finite element models (FEM), which now feature high computational capacity, complex constitutive laws, multi-physics considerations, and the ability to reproduce nonlinear behaviors, among other functionalities [7]. It also applies to models based on monitoring data, ranging from the traditional hydrostatic-season-time (HST) model [8] to more recent models based on machine learning (ML) algorithms [9] and fine-tuned deep learning (DL) techniques [10].

In recent years, large language models (LLMs), based on generative artificial intelligence (AI), have emerged. Their availability and broad capabilities have led to their early adoption across many sectors, including civil engineering. Their use and potential have been explored in water distribution networks [11,12], geotechnics [13,14,15,16,17,18,19,20] and construction [21,22,23,24,25,26]. However, their application in dam engineering remains mostly unexplored. Although recent literature reviews on the use of AI in this field [27,28,29,30] have primarily focused on machine learning applications for a wide range of engineering tasks, only a limited number of studies have investigated the use of LLM-based tools in dam structural health monitoring. These studies suggest the potential of LLMs for various tasks related to dam safety analysis, although they rely on highly sophisticated methodologies that are difficult to implement in standard workflows. In contrast, many professionals use more accessible, general-purpose commercial LLMs for a range of day-to-day tasks. In this paper, we evaluate the capabilities of one of these off-the-shelf LLMs in solving practical dam engineering tasks.

Specifically, we evaluated its coding skills to create ML-based predictive models and its reasoning capabilities to interpret the results and draw conclusions to support dam safety. We considered Theme A from the ICOLD 2022 Benchmark Workshop (BW) [31] as a case study, in which the behavior of an arch dam had to be predicted and interpreted. The study aimed to answer the following research questions:

• Can ChatGPT understand the problem statement, analyze the data provided and propose a sound methodology?
• Can ChatGPT generate code to create an ML model to produce accurate predictions of dam behavior?
• What is the effect of the inherent random component of ChatGPT when faced with the exact same prompts for this problem?
• Can ChatGPT correctly interpret the behavior of the dam and produce relevant engineering conclusions?

The remainder of the paper presents: a brief description of LLMs and their limitations for engineering tasks; a review of their applications in civil engineering; the methodology of our experiment; the results and discussion in terms of proposed methodology, ML model development, prediction accuracy and model interpretation; and, finally, a discussion of the possibilities, limitations, and future of this technology in the field.

2. Background

2.1. Large Language Models (LLMs)

In recent years, LLMs have transformed multiple aspects of daily life—from basic tasks, such as information retrieval, to the automation of highly specialized workflows. These models typically adopt a chat-based format in which users interact with the tool through text inputs, known as prompts, and the model generates responses in textual form. Most modern LLMs are based on the transformer architecture [32] and consist of very large neural networks, with millions or billions of parameters (weights). They generate words that are likely to be appropriate responses to the prompt, based on patterns learned from a massive and diverse text corpus during pretraining. The weights learned from text are further refined in two additional stages. The first is fine-tuning, in which examples of question-and-answer pairs are provided to illustrate the desired responses to specific tasks. The second is reinforcement learning from human feedback (RLHF), through which the model’s behavior is further aligned with users’ needs by evaluating alternative responses to given prompts and using this feedback to adjust the model.

The responses provided by these models have a certain degree of randomness, inherent to the model itself and controlled by the “temperature” parameter [33]. Increasing the temperature makes the model more likely to select a word with a high, but not necessarily the highest, probability. While this can be advantageous for some tasks, such as creative text generation, it can be a drawback in technical contexts that require consistency and reliability in responses.

The list of available models is long and constantly growing. A comprehensive list of all models and their release dates is beyond the scope of this work (and would quickly become outdated). Nonetheless, ChatGPT is arguably the most popular choice in many scientific areas. To estimate the volume of literature associated with each LLM, we searched Scopus on 17 July 2026 using the Keywords field. Each term was searched separately using the following exact query strings: KEY(“ChatGPT”), KEY(“Gemini”), KEY(“LLaMA”), KEY(“Perplexity”), KEY(“DeepSeek”), KEY(“Mistral”), KEY(“Claude”), and KEY(“Grok”). No filters were applied for publication date, document type, language, subject area, or any other criterion. The searches returned 20,198 documents for ChatGPT, 9439 for Gemini, 2301 for LLaMA, 1453 for Claude, 962 for DeepSeek, 543 for Perplexity, 455 for Mistral and 126 for Grok.

Conventional LLMs based on transformer architectures are highly capable, but also present several limitations:

• Prompt dependency: Responses depend on the text provided by the user. If queries are vague or imprecise, the answers may be incorrect or incomplete [34].
• Optimistic answers: The tone and level of detail of their responses are often generic and may not meet the user’s needs. For example, answers may sometimes be overly optimistic [35].
• Lack of domain-specific knowledge: The models are mostly based on general knowledge and therefore lack specialized information in certain fields [36]. This is particularly relevant for engineering-related tasks and, especially, for applications in dam safety, where the amount of available online information is limited.
• Lack of memory: Although LLMs rely on system prompts, the specific information provided during a chat session is only available within that session.
• Hallucinations: LLMs can provide incorrect or inaccurate answers without recognizing or acknowledging it [37]. Such responses are termed “hallucinations” and are highly damaging in technical settings [38].
• Randomness: As mentioned above, the temperature parameter controls the degree of randomness in LLM outputs, which is always present and can lead to inconsistent responses [39].
• Privacy of data and knowledge: Studies on confidentiality show that the nature of these models makes it extremely difficult to verify how they handle sensitive information and whether they may compromise privacy [40,41]. Infrastructure owners seek to prevent any dissemination or misuse of confidential information such as monitoring data or safety reports. Users may also be unwilling to share expert knowledge that could be inferred from their interactions with the models.

LLMs are constantly being updated and improved. Current models incorporate additional functionalities beyond text-based output. Some LLMs are being integrated into multimodal models, which can process or generate programming code [42] or analyze data files [43]. Examples include the Data Analysis tool from OpenAI [44] and the code interpreter from Mistral [45]. Another improvement over earlier models is their ability to complement their training knowledge with information retrieved from targeted internet searches related to the question posed, enabling the generation of more up-to-date answers. This overcomes an important limitation of previous models, whose responses were limited to the information available up to their training cut-off date. An additional feature, particularly interesting from an engineering perspective, is the possibility of calling external user-defined functions, generally through APIs [46].

Updates in commercial LLMs are often delivered without detailed information on the new features introduced. The resulting changes in their behavior for a given task can only be identified and assessed empirically. Chen et al. [47] reported significant variations in the behavior of GPT-3.5 and GPT-4 over relatively short periods, with improvements in some tasks and poorer performance in others. This has implications for reliability and reproducibility.

Several techniques have been developed to address the previously mentioned limitations. For this work, the most relevant are those aimed at providing domain-specific information related to dam engineering.

Domain-specific fine-tuning involves taking a pretrained model and adapting it to a specific domain through retraining with specialized information. The result is a stable model (with fixed parameters) that incorporates both the original training knowledge and the new, domain-specific information. In addition to requiring specialized knowledge of LLM training techniques, this method demands substantial computational resources and high-quality, well-indexed data [48].

Retrieval-Augmented Generation (RAG) also provides domain-specific information [49], but unlike fine-tuning, the data are not used to retrain the model. Instead, they are employed to build an enriched prompt that is sent to the base model. This eliminates the need for retraining but still requires encoding and structuring the external information [50].

Prompt engineering is the discipline aimed at designing optimized user inputs to obtain the desired response from the LLM. It is well known that, given an LLM with fixed parameters, the quality of the response depends heavily on the quality of the prompt [51]. This has led to the development of an entire field of research aimed at standardizing the way users interact with LLMs to obtain more accurate and useful outputs [52]. One of the most powerful strategies in prompt engineering is the use of patterns, as proposed by White et al. [51]. Although their original purpose was software development, some patterns can also be applied to technical tasks, including the Persona, Template, and Flip Interaction patterns. More advanced prompt engineering techniques include in-context learning (ICL), chain-of-thought (CoT) reasoning, task decomposition, ensembling, and self-criticism [52].

Although considerable effort has been devoted to prompt engineering, its relevance is decreasing as the latest models become more capable; they often include explanations, uncertainty ranges, or conditional answers by default. Such behavior previously had to be elicited with the Role or Self-ask patterns. For example, ChatGPT 5 can perform internet searches, which solves the problem of outdated training data. It also provides the sources or information on which its answers are based. Moreover, its Data Analyst mode allows the generation and execution of code to, for example, create a plot or even produce a slide presentation. Such functionalities transform plain LLMs into agents, defined as “GenAI systems that serve a user’s goals via actions that engage with systems outside the GenAI itself” [52].

2.2. Applications of LLMs in Civil Engineering

Many authors highlight the need to adapt general-purpose LLMs to domain-specific tasks, particularly in civil engineering. The customization of these models can be based on different strategies:

• Fine-tuning is the most resource-demanding option [53]. Domain-specific information must be selected, curated and formatted to be fed to a generic LLM, which is then retrained on the enriched dataset.
• RAG is a relatively lightweight approach that enhances LLMs by supplying task-specific documents or external knowledge at inference time, eliminating the need to retrain the model [54]. More recent advances, such as Bayesian-based RAG frameworks, extend this capability by explicitly modeling uncertainty. These methods assess whether the retrieved evidence is sufficient to support a reliable response, estimate the confidence associated with the generated answer, and determine when the system should defer or request additional information before providing a final output [55].
• AI agents can perform tasks beyond text generation, such as code generation, which makes them suitable for addressing some engineering problems [56].
• Some authors combine several agents, each designed to perform a specific task, to achieve an overall goal [57,58].

Although the adoption of these tools in professional practice is growing rapidly, the number of scientific publications remains limited and varies across fields. Geotechnics is arguably the field with the greatest activity, with applications to a variety of tasks, such as geotechnical design [15,18,33], analysis of landslide events [17], forecasting grouting construction parameters [14], or safety assessment during tunnel construction [59]. In other areas, these tools have been applied in transportation engineering [21,26,50,60,61], construction management [22,23,62,63], or analysis of engineering documents [64]. In the water sector, recent papers illustrate the potential of LLMs [11,12] for various specific tasks, including data analysis, predictive maintenance of water distribution networks, demand analysis or real-time operational control.

In dam safety, Liu et al. [65] proposed an AI agent incorporating Claude Sonnet 4.5 (Anthropic) and BGE-Large-ZH-v1.5 as its underlying LLMs for the structural health monitoring of concrete arch dams. Similarly, Pei et al. [66] employed DeepSeek-R1 to extract knowledge from text-based inspection reports of a concrete gravity dam for structural safety assessment and defect diagnosis. Furthermore, Li et al. [67] proposed a Time-LLM-based forecasting framework that integrates multisource dam monitoring data.

Related applications have also been reported in the broader fields of structural engineering and structural health monitoring, including human–AI teaming for structural analysis [68], knowledge-infused SHM frameworks based on prompt engineering [69] and LLM-based methods for the imputation of missing sensor data [70].

These advances run in parallel with improvements in the tools on which they are based, which means that newer versions may already succeed in some engineering problems in which previous versions failed. As a result, custom-developed solutions may lose part of their relevance, regardless of the complexity of the customization.

3. Materials and Methods

In this work, we considered specific tasks relevant to dam engineering that are suitable for generative AI. Among the available LLMs, we selected ChatGPT, as it is the most widely adopted model. Its performance was evaluated in terms of: (a) its ability to analyze the problem description and input data to draft an appropriate strategy; (b) the accuracy of the predictions; and (c) the interpretation of the results from a dam safety perspective. The following subsections describe the benchmark framework and the LLM-based analysis protocol. An overview of the experimental workflow is shown in Figure 1.

Figure 1. Experimental workflow of the analysis.

3.1. Benchmark Framework and Reference Case

To evaluate the ability of ChatGPT to predict dam behavior, Theme A from the ICOLD 2022 Benchmark Workshop (BW) was used as a reference [31]. In this event, participants were provided with monitoring data from an arch dam, along with related information and instructions to prepare the submissions. External (load) data for the period 1995–2017 were provided, whereas response data were only available until the end of 2012. In the first task of the theme, participants were asked to submit predictions for the 2013–2017 period. Accordingly, Theme A was organized into three cases, depending on the period of analysis:

• Calibration (Case A): 2000–2012.
• Short-term prediction (Case B): January 2013–June 2013.
• Long-term prediction (Case C): July 2013–December 2017.

The quantities under study were displacements measured by pendulums (CB2 and CB3), crackmeters (C4_C5), piezometric levels (PZCB2 and PZCB3), and seepage. Although FEM-based approaches were permitted, many participants used ML-based models. Figure 2 shows the location of the monitoring devices over a schematic plot of the dam.

Figure 2. Schematic representation of the dam cross section, showing the location of the monitoring devices. Elevations are shown in m.a.s.l.

Finally, the participants had to analyze the models and the predictions and provide insights into the dam response to external loads and its long-term behavior.

A total of 18 teams from 12 countries worldwide submitted their solutions to the problem. Only six of them provided predictions for all six variables, which illustrates the complexity of the proposed tasks. The full description of the problem and the synthesis of the contributions, including the approaches used and the behavior interpretation, can be found in [31].

This benchmark was selected because it provides a well-defined engineering problem based on real monitoring data, includes several response variables and prediction horizons, and offers independent reference results from multiple participants.

3.2. LLM-Based Analysis Protocol

In the present study, we evaluated the Medium reasoning mode of GPT-5.6 Sol. The purpose was not to determine the maximum performance of ChatGPT under the most computationally intensive configuration, but rather to evaluate a practical use of a general-purpose commercial LLM for an engineering task. The Medium reasoning setting was therefore selected as a compromise between reasoning capability and computational effort, consistent with our aim of assessing a general-purpose model rather than a specific, ad-hoc tool or interaction strategy. The model was provided with the same materials available to the human participants, including a PDF describing the benchmark problem and spreadsheets containing the corresponding datasets (Supplementary Material S1). The initial prompt was as follows: “I want to prepare a submission for Theme A. I will not use FEM. I want to send predictions for all variables. Read the description, check the data, and give me a plan.”

To assess the chatbot’s performance while minimizing the influence of user prompting skills, all prompts were intentionally kept brief and neutral (the complete interaction is provided in Supplementary Material S2). The objective was not to optimize the interaction with ChatGPT, but to evaluate its performance under minimal user guidance. Therefore, no advanced prompt-engineering strategies or domain-specific prompt templates were used. Following the initial response, the model was given the prompt: “Compute predictions for Case B and Case C and all output variables. Save them into an Excel file. Use the approach that you consider more appropriate.” This approach allowed the model to autonomously select the prediction algorithm, data preprocessing procedure, and training strategy.

Subsequently, the model was asked to describe its model development process. For further verification, it was instructed to provide the corresponding source code as a downloadable file. Finally, to evaluate the reasoning capabilities of the model and its ability to interpret the prediction results in accordance with the final task of the benchmark workshop, we provided the following prompt: “Go ahead with the interpretation.” As in the previous steps, this prompt was deliberately designed to be neutral in order to minimize user influence on the model’s responses.

The entire process was repeated using exactly the same methodology and prompts to explore the effect of the random component of the inference process on the reproducibility of the interaction. For clarity, the first and second runs are referred to as R1 and R2 throughout the remainder of the paper (the full interaction in R2 is included in Supplementary Material S3). The experiments were performed between 13 July and 17 July 2026.

3.3. External Assessment of Model Interpretation

To complement the qualitative analysis performed by the authors, the engineering interpretations generated in R1 and R2 were independently evaluated by external experts in dam monitoring and dam safety who were not involved in the study. A rubric was used to assess six aspects considered relevant from a practical perspective: interpretation of load-response relationships, consideration of temporal evolution, cross-variable consistency, physical plausibility, relevance for dam-safety assessment and recognition of uncertainty and limitations.

Each criterion was scored from 0 to 2, where 0 indicates an incorrect or absent interpretation, 1 a partially correct or incomplete interpretation and 2 a correct and sufficiently addressed interpretation. Therefore, the maximum subtotal was 12 points. In addition, a penalty ranging from 0 to −2 was considered for unsupported, misleading or potentially unsafe engineering claims.

The experts were provided with the Theme description, the complete interaction with the LLM for reference and the synthesis of the contributions to the Benchmark. They were instructed to assess only the engineering quality of the dam-behavior interpretation, regardless of writing style or length, and to evaluate R1 and R2 separately.

The complete rubric and the instructions provided to the evaluators are included in the Supplementary Material S4.

4. Results and Discussion

4.1. Problem Interpretation and Proposed Plan

ChatGPT-R1 responded with a detailed analysis of the information provided: the problem formulation was well understood, the data were properly analyzed, and a reasonable workflow was proposed, including data pre-processing, training and evaluating candidate models, selecting one or a combination of them, generating predictions and producing an interpretation.

The data analysis was particularly thorough and detailed. The inputs and outputs were correctly identified; the nature of each task (mandatory/optional) was recognized; the time periods for training and validation were checked; some exploratory analysis was performed, from which relevant features were identified (number of observations, typical reading frequency, range of variation in each variable, number of missing values and overall quality of the time series); the peculiarity of PZCB3, whose behavior changed after an intervention in 2008, was also noted.

The proposed algorithms seem reasonable and partially consistent with those suggested in the instructions, albeit with some added complexity. For instance, the conventional HST model is considered, but with regularization. A weighted combination of several models is also suggested.

Additional sophisticated steps are also considered, such as the inclusion of derived variables to account for the seasonality of the phenomenon and the inertia of the response to changes in loads (e.g., rolling averages of temperature or cumulative rainfall over several periods, among others).

The modeling approach is adapted to each variable, accounting for its nature, based on the correct identification of the conventional behavior: higher influence of water level and temperature for the displacement at the top, stronger influence of temperature with inertia for the displacement at the bottom, possible non-linear behavior due to joint opening, strong effect of water level on piezometers, possible influence of rainfall on seepage, which must be non-negative. All these considerations are relevant, reasonable, and reveal a good understanding of the problem and some field-specific knowledge.

The procedure proposed for model validation is particularly appropriate: it recommends temporal validation instead of the conventional cross-validation approach, to account for the time effect and avoid optimistic accuracy results. The proposed strategy is a moving-window validation, training over one year and validating on the following year, repeated along the entire training period. However, when the model was subsequently asked to generate the predictions, the implemented approach differed from the initial plan and was limited to conventional fixed training and test sets (see Section 4.2).

The approach used to compute the warning thresholds is also consistent with engineering practice: the distribution of the residuals is computed and quantiles are used to define the limits.

For model interpretation, conventional and useful approaches are proposed, such as partial dependence and accumulated local effects plots, permutation importance, and residual trend analysis. Only some of them are mentioned in the problem description, which demonstrates the reasoning capabilities of the chatbot in this regard.

As for R2, with exactly the same information and prompt, the answer was conceptually similar, although the details differed. For example, a combination of HST with an ML algorithm is suggested, but it is described as a hybrid model instead of an ensemble prediction. The suggested validation process is identical, with successive training for separate years and validation with the next year. However, the wording of the descriptions differs.

4.2. Modeling Approach

In response to the second prompt, which requested the predictions, the model generated code implementing a specific modeling approach. The answer was concise: a file containing the predictions was created, as requested, accompanied by a very brief description of the methodology, which was described in more detail in response to the next prompt. In addition to conventional modeling steps, some advanced decisions were made:

• For PZCB3, the training data were limited to the period after 2008, because the problem description reported an intervention in that year that affected the records.
• Some derived variables were generated, namely lagged and rolling variables of the environmental loads. Overall, 67 inputs were considered, including rolling means and lagged values for temperatures and water level, cumulative sums of rainfall, gradients of reservoir level and some combined features with no clear justification, such as the difference between the average temperatures over 7 and 60 days and combinations of water level with sine and cosine of the calendar day. This results in high collinearity among many predictors.
• Continuous elapsed time is included as an input to capture the potential effect of time on each response.
• Three types of models were generated and compared:

  ○

  Ridge-HST is a modification of the conventional HST, including regularization for robustness.

  ○

  HistGradientBoosting is an implementation of Boosted Regression Trees, optimized for fast training.

  ○

  ExtraTrees (Extremely Randomized Trees) is a variant of Random Forests, also optimized for speed.

As for the train/test split, and as stated in the plan, the model avoided a random split in order to replicate the actual application of the model, i.e., predicting future behavior. All three models were fitted to data before 31 December 2009, and their accuracy, in terms of mean absolute error (MAE), was computed for the period 2010–2012. Regarding PZCB3, for which only data after 2008 were considered for training, the validation period is limited to 2011–2012. The two best models in validation (lowest MAE) are then combined in a weighted ensemble model, in which each model’s prediction is multiplied by a coefficient inversely proportional to its validation error. Since HistGradientBoosting was always the least accurate, the final ensembles were formed by ExtraTrees and Ridge regression for all outputs. This strategy, reasonable and appropriate for the problem at hand, is different from the moving-window approach described in the previous prompt, as discussed in Section 4.1. This discrepancy is worth mentioning: we identified it by verifying the code used and by explicitly asking the LLM to explain the procedure. The code did not follow the initial plan proposed.

Once the final approach was defined, it was applied to the entire training period to fit a new model, which was then used to generate predictions for the requested time periods (Case B and Case C).

All these decisions can be considered appropriate and the final approach is relatively sophisticated but reasonable. The combination of two candidate models adds robustness to the result and helps control overfitting. However, the script only includes hyperparameter tuning for the alpha parameter of Ridge-HST. In contrast, the HistGradientBoosting and the ExtraTrees are built with fixed, hard-coded parameters. This is a limitation of the automatically generated workflow, since hyperparameter optimization could improve the predictive accuracy. No additional tuning was performed by the authors, since this would have modified the workflow generated by ChatGPT under the experimental conditions.

The choice of algorithms is also questionable. Although similar results can often be obtained with different algorithms, the choice of simplified versions of Boosted Regression Trees and Random Forests is not justified. These versions trade predictive power for reduced training time; however, the datasets used in this problem are small and computational cost is not a concern in this setting.

We asked the model to save the generated code as a file and verified that it was written in Python 3 and that the accompanying descriptions were consistent with the implemented code. Interestingly, the generated scripts were well structured and documented. Some small changes were required for the script to run correctly on a local computer, as the LLM generated file paths based on its own assumed working environment rather than the actual local environment. We ran the code and confirmed that the outputs were consistent with the prediction results previously reported by the model.

The availability of the code allows the response of the model to be audited. However, the complexity of the script makes verification far from straightforward. This is important because, while LLMs can help civil engineers with limited programming skills develop well-structured and complex software, programming expertise is still needed for verification.

As for R2, the approach is again similar, but with notable differences. Two algorithms were considered (instead of three): Ridge regression and ExtraTrees. As for the model selection strategy, instead of limiting the calibration period to dates before the end of 2010, an 80/20 split is used. Again, the implementation differs from the initial plan. The final model is selected from three options: each model separately and their simple average, rather than a weighted combination based on each model’s accuracy. The best option for each variable is refitted over the entire training period, as before. As a result, ExtraTrees is chosen for C4_C5, PZCB2 and Seepage, Ridge for CB3 and the average for CB2 and PZCB3.

The feature engineering process is identical, with the same input set. Again, no hyperparameter tuning is performed.

Overall, the comparison with R1 is qualitatively similar to that for the problem interpretation stage, i.e., the answers are conceptually similar, but include some differences. However, such small differences now have significant implications for the results, since they affect the algorithm chosen to generate the predictions for each output. In both cases, the implementation did not follow the proposed methodology, which illustrates the need for verification.

4.3. Prediction Accuracy

All target variables were predicted for both time periods, and the results were exported to an Excel file for download, as explicitly requested in the prompt. The primary goal of the Theme is prediction accuracy. Therefore, we compared the model outputs with the observations and with the predictions submitted by the participants. The results are summarized in Table 1, both for R1 and R2. It should be noted that the ground truth data for the test period (2013–2017) were never shared with ChatGPT to avoid information leakage.

Table 1. Prediction accuracy (mean absolute error, MAE) of the models created by ChatGPT and comparison with those provided by the participants in the BW.

Variable/Case 1 [Units]	ChatGPT	ChatGPT	Participants
	R1	R2	Average	Standard Deviation	Best
CB2/Case B [mm]	1.70	0.95	2.24	1.23	1.00
CB3/Case B [mm]	0.58	0.32	0.74	0.74	0.18
C4_C5/Case B [mm]	0.18	0.09	0.31	0.39	0.05
PZCB2/Case B [m]	0.83	0.69	1.20	1.45	0.29
PZCB3/Case B [m]	0.22	0.24	0.67	0.40	0.16
Seepage/Case B [L/min]	2.34	2.37	2.94	0.96	2.07
CB2/Case C [mm]	3.31	1.88	3.28	2.10	1.66
CB3/Case C [mm]	0.70	0.68	1.07	0.83	0.47
C4_C5/Case C [mm]	0.30	0.28	0.38	0.39	0.15
PZCB2/Case C [m]	0.96	1.20	1.75	1.57	0.42
PZCB3/Case C [m]	0.55	0.56	0.99	0.39	0.40
Seepage/Case C [L/min]	2.98	2.34	2.94	0.69	2.22

¹ Case B: January–June 2013; Case C: July/2013–December/2017. More details of the benchmark can be found in [31].

The results show that the predictions provided by ChatGPT-R1 are more accurate than those of the average participant for 10 out of the 12 cases considered. R2 performed even better, with a lower MAE than the participants’ average for all variables and prediction periods. The improvement over R1 is most notable for displacements and joint opening (CB2, CB3 and C4_C5) in Case B (short-term prediction), with a decrease in MAE of around 50%. Only for PZCB2 in Case C were the R2 predictions clearly less accurate than those of R1.

These results show that the predictive models generated by ChatGPT are comparable in accuracy to those presented by the participants in the Benchmark. Moreover, R2 outperformed all participants for CB2 in Case B. It should be noted that these models were created with minimal interaction with the chatbot, comparable to that of an inexperienced user who relies exclusively on the problem description and the data provided.

It is also worth noting that the entire process took only a few minutes, while the participants reported several days of work to produce their contributions. Although our experiment covered only part of the work (the writing of the paper is not included), the reduction in the time required to draft and execute a modeling plan is remarkable.

In both runs, the chatbot provided predictions for all outputs and cases, whereas most participants submitted predictions only for the mandatory outputs and a subset of the optional ones.

A more realistic interaction would involve providing ChatGPT with more specific instructions regarding feature engineering, the algorithms to be considered, and the training strategy. This approach would place greater emphasis on the user’s expertise and judgment. Although it would not necessarily ensure higher accuracy, it would provide greater control over the modeling approach. In this setting, the chatbot’s main added value would lie in the time saved on programming.

4.4. Model Interpretation

In response to the last prompt: “Go ahead with the interpretation”, ChatGPT-R1 provided an answer that was largely incorrect and of little practical value, with clear technical deficiencies. Although the proposed plan (response to the first prompt) included some of the suggestions of the problem description (partial dependence plots, permutation importance, decomposition into hydrostatic, thermal, rainfall and time-related effects and residual analysis to identify behavioral changes), the answer is based solely on the computation of the yearly average predictions for each variable, without relating them to the evolution of the loads. In addition, only the prediction period is analyzed, without any reference to the observed behavior over the training period, which is also relevant, since the observed data are available.

For instance, the minimum prediction for CB2 (displacement in the upstream direction) occurs in 2016, clearly due to the unusually low reservoir level recorded. This is easy to identify from the time series plot of the water level, but it is not mentioned in the interpretation provided by ChatGPT.

The only interesting contribution is the combined analysis of all six output variables, considering engineering knowledge and the relation between variables: the behavior is described as “coherent” in 2016, with displacement to the upstream direction, closure of the joint, low piezometric levels and reduced seepage. In this regard, there is also useful general guidance on how to interpret these results: “No single measurement should be used in isolation. A deviation in CB2 accompanied by normal CB3, joint, piezometric and seepage responses would warrant checking the pendulum or its local conditions. Conversely, simultaneous deviations in several related variables would be stronger evidence of an actual behavioral change”.

An engineer would typically apply specific methods to identify the effect of each load on each output, such as analyzing the coefficients of an HST model, or performing a sensitivity analysis and evaluating feature importance in an ML model. Examples of both approaches are provided in the problem description, available to ChatGPT. Quantifying the strength of the association of each load with the output is particularly relevant for understanding the behavior of the dam. In addition, one of the main results from a dam safety perspective is the time evolution of the dam response and its interpretation by the model. In this case, although time was included as an input, its effect was not analyzed.

The interpretation given in R2 is more appropriate. Permutation importance is used to compute the average contribution of the loads to each output, which is one of the methods suggested in the instructions. Since feature engineering had been applied and many inputs are highly correlated, these are grouped according to the corresponding load, e.g., water level, temperature, seasonal terms and rainfall. Interestingly, the permutation importance is computed by permuting all related variables at once, which is correct and not obvious. Another sophisticated choice is to average the importance over 15 permutation iterations, after setting negative values to zero.

The approach, though, includes some questionable decisions: the decrease in accuracy after permutation is compared with that of the reference model only during the validation period. This results in an unbiased estimate, but one based on a limited period, which has implications: for instance, the importance of the time effect is not mentioned, probably because it is negligible over such a short period. In addition, seasonality is considered as a grouped effect at the same level as the physical loads (temperature, rainfall and water level). This is incorrect from an engineering perspective. Indeed, the seasonal term in the conventional HST model was introduced as a proxy for the actual temperature, which was unavailable for many dams at the time it was first proposed. When daily temperature records are available, the calendar-day predictors should be omitted, since they mask the actual effect of temperature (with which they are strongly correlated) and, to a lesser extent, that of the reservoir level (with which they may also be substantially correlated).

In general, the results in terms of the strength of the influence of each group on every output are reasonable, well interpreted and consistent with dam engineering knowledge. Regarding the displacements, the thermal effect is stronger near the crest than at the bottom, which is reasonable from an engineering perspective. The crack opening is mostly driven by the changes in water level, which is consistent with the results of the participants in the BW. Seepage and PZCB2 are also mostly influenced by the hydrostatic load.

As for PZCB3, the feature importance results suggest a strong response to the temperature variation. The chatbot clarifies that “the surprisingly large temperature contribution should be treated carefully” and provides some potential explanations for this result. The numerical results are indeed surprising from an engineering perspective. Since interpretation is an open question, the answer, together with the explanation, can be considered appropriate.

Other disclaimers are included in the interpretation for the remaining variables, always insisting that the results represent the predictive information of each input to the model, not necessarily associated with a physical effect.

The remaining information is largely generic, consisting mainly of the average values and ranges of variation for all variables and time periods—namely, the training period, Case B, and Case C—accompanied by only brief and minimally informative interpretations.

The time evolution is mostly overlooked, which is a significant shortcoming. The importance of time as a feature is not reported, and no information is provided on its contribution. This is particularly relevant, since temporal changes, either sudden or gradual, are essential to identify damage or degradation of the structure.

Overall, the interpretation task showed the greatest differences between R1 and R2. Both focused excessively on the average predictions; R2 took feature importance into account, but the conclusions were generic and time effects were not considered; in neither case was partial dependence computed; the overall conclusions on the dam behavior were superficial.

4.5. External Evaluation

To complement the qualitative assessment described above, the interpretations generated in R1 and R2 were independently evaluated by two external experts using the rubric described in Section 3.3. The results are summarized in Table 2.

Table 2. Results of the external evaluation of the dam behavior interpretation.

Criterion	R1 Expert 1	R2 Expert 1	R1 Expert 2	R2 Expert 2
Load-response relationship	1	1	1	2
Temporal evolution	0	1	0	1
Cross-variable consistency	1	1	1	2
Physical plausibility	1	1	1	2
Dam safety relevance	0	1	0	1
Limitations and uncertainty	1	1	1	1
Subtotal	4	6	4	9
Penalty	−2	0	−2	0
Final Score	2	6	2	9
Overall assessment	Inadequate	Partially adequate	Inadequate	Generally adequate with expert supervision

For R1, both experts identified relevant limitations. The first evaluator assigned a subtotal of 4/12 and applied a −2 penalty for potentially unsafe or misleading omissions, resulting in a final score of 2/12. The second assigned 6/12 with no penalty. Despite this difference, both assessments highlighted similar weaknesses: limited physical interpretation of the load–response relationships, insufficient consideration of temporal evolution, and an incomplete connection between the results and dam safety assessment. Both experts also noted that relevant aspects of the 2016 response were not properly interpreted. This suggests that the difference in the overall scores reflects differences in how the rubric was applied, rather than disagreement about the weaknesses identified.

The assessment of R2 showed a larger discrepancy between evaluators. The first expert again assigned 4/12 with a −2 penalty, resulting in a final score of 2/12. The second assigned 9/12 with no penalty and classified the interpretation as “Generally adequate with expert supervision”. This second assessment recognized improvements in the interpretation of load–response relationships, cross-variable consistency and physical plausibility, but still identified weaknesses in the treatment of temporal evolution and in the interpretation of conditions outside the calibration range.

Overall, the independent assessment supports the conclusion that interpretation quality was lower, and less consistent, than prediction accuracy. It also suggests that R2 provided a more technically useful interpretation, although this improvement was recognized by only one of the two evaluators. These differences indicate that some aspects of engineering interpretation remain dependent on professional judgement, even when a predefined rubric is used. They may also partly reflect the evaluators’ different backgrounds: although both are experts in dam engineering, the first specializes in practical dam safety assessment, while the second works in academia. Nevertheless, both agreed that expert supervision is required before the generated interpretations are used in a dam safety context.

4.6. Practical Considerations

The experiment was conducted using publicly available data provided for the BW. However, in practical dam safety applications, data and knowledge privacy is a major concern for dam owners and consultants. Whenever feasible, raw data should be replaced with synthetic, anonymized, or aggregated alternatives. Sensitive personal information should be identified and removed from prompts. These procedural controls should be supported by technical safeguards, including data encryption [71]. Sensitive actions, such as transmitting files, modifying databases, or accessing operational systems, should also require deterministic policy checks or explicit human approval [72,73].

A safer alternative is to install models within the entity’s local systems, but this requires substantial investment in hardware, consulting, and ongoing maintenance. Moreover, although some authors report good performance for locally deployable models [74], their performance is generally lower than that of cloud-based commercial models.

The choice of LLM deployment should depend on the sensitivity of the information and the consequences of disclosure, with external services suitable for non-sensitive tasks and private, contractually protected, or local solutions preferred for confidential dam documentation and operational data.

Reproducibility is a significant concern when using these tools. We found notable differences between answers given to the same prompts and input files, which demonstrated the existence and relevance of run-to-run variability, even in a very small test with just two runs. While the outputs were conceptually similar, differences in the model training strategies and algorithms considered resulted in variations in prediction accuracy, although both approaches were competitive. Such variability arises from the inherent random component of the LLM inference process and could be reduced by using more specific prompts that define the model, the features to be considered, the training parameters, and the approach to model interpretation. However, reproducibility can also be affected by changes due to model updates over time, as studied by Chen et al. [47]; this aspect was not assessed in this study.

In practical applications, retrieval-augmented generation (RAG) could provide a useful intermediate approach between the direct use of a general-purpose LLM and the development of a fully domain-specific model. In such a workflow, the model could retrieve relevant information from a controlled knowledge base containing dam documentation, monitoring reports, instrumentation descriptions, historical assessments and engineering procedures. This would provide dam-specific context without requiring such information to be permanently incorporated into the model, while also allowing the sources supporting a response to be identified and verified.

Regardless of how they are used, in technical applications it is essential not only to supervise the responses provided by LLMs but also to verify them in detail. For behavior prediction, this verification can be carried out through historical data validation, sensitivity analysis, predictive validation and the assessment of time evolution of the residuals, among others [75]. LLMs should therefore be integrated into a human-in-the-loop workflow and considered as assistants rather than autonomous decision-making systems. They may support tasks such as data exploration, code generation, preliminary interpretation and report preparation, but the verification of the results, assessment of their physical plausibility and final dam-safety conclusions should remain under the responsibility of qualified professionals. In addition, any interaction with operational databases or monitoring systems should be subject to predefined access permissions and explicit human authorization. In this regard, LLMs are no different from any other engineering software application: even well-established, commercial software tools can generate incorrect results if not properly used and verified.

4.7. Limitations

The experiment comprised only two independent runs. This was sufficient to identify notable differences arising from the stochastic nature of the interaction, but not to characterize their frequency, magnitude or statistical distribution. A systematic analysis of run-to-run variability would require a larger number of iterations under the same model version and experimental conditions.

Only the Medium reasoning setting was considered, as a compromise between reasoning capability and computational effort. Higher reasoning effort may lead to different results.

A further limitation is related to the continuous evolution of commercial LLMs. The results reported correspond to the GPT-5.6 Sol version available when the experiments were conducted, in July 2026. Commercial models are constantly updated, access to previous versions is not always available and previous studies have shown that their behavior may change significantly over time [47]. Therefore, exact reproduction of this experiment cannot be guaranteed.

The study was limited to a single general-purpose commercial LLM. Therefore, the results obtained cannot be assumed to represent the behavior of other models, such as those provided by other commercial or open-source platforms. A systematic comparison among different LLMs would be necessary to distinguish model-specific behavior from more general characteristics of this technology.

The experimental analysis was based on a single case study, corresponding to the ICOLD 2022 Benchmark Workshop and to the monitored behavior of a concrete arch dam. Although this benchmark provides a well-defined problem and an independent reference for comparison, the results cannot be directly generalized to other dam types, monitoring systems, environmental conditions or datasets with different levels of noise and missing information.

The interaction strategy was deliberately based on brief and neutral prompts, with the objective of minimizing the influence of the user and allowing ChatGPT to make most modeling decisions autonomously. Therefore, the results should not be interpreted as the maximum performance attainable with the model. More detailed prompts, domain-specific instructions or advanced prompt-engineering strategies could lead to different results.

Finally, the ML workflows were evaluated as generated by ChatGPT, without subsequent manual optimization by the authors. Therefore, the reported prediction accuracy reflects the performance of the autonomously generated workflow rather than the maximum accuracy potentially achievable with the selected algorithms.

5. Conclusions

The proposed tasks relate to a specific, well-defined dam engineering problem described in a document provided to the model. Solving them therefore requires a combination of skills: understanding the problem description, analyzing the input data, training ML models (coding), and interpreting the ML models in light of dam engineering knowledge.

The model appears to have correctly understood what was required and proposed a correct and detailed plan. For model training, the answers combine some sophisticated steps related to ML (comparison of algorithms, combination of models, time-dependent training and calibration, feature engineering), detailed and correct Python code, but also methodological deficiencies, such as the lack of hyperparameter tuning or the treatment of seasonality as a driver alongside temperature.

In terms of prediction accuracy, the results are remarkably competitive: both runs outperformed the average participant (except in two cases for R1), and R2 was even more accurate than the best participant in one case. This is a noteworthy result, given that all modeling decisions were made by the chatbot, whereas the contributions to the BW came from teams of experts who spent a considerable amount of time on the task, some of whom did not even submit predictions for all variables.

In this regard, it can be concluded that the use of ChatGPT may facilitate access to relatively advanced technologies (such as ML models) for users who are not familiar with them. This may therefore represent a significant step forward in the adoption of these methods in fields such as civil engineering. Even with limited ML expertise and no coding skills, a motivated user with knowledge of dam engineering can interact with ChatGPT to test different algorithms among those proposed in the description, or ask the chatbot for ideas.

In terms of model interpretation and dam behavior analysis, the output from ChatGPT was incomplete, mostly generic and included errors. This is clearly a task in which domain-specific knowledge plays a greater role. The conclusions given by the chatbot in this regard are mostly based on the raw comparison of average predictions over different time periods. The chatbot overlooked aspects such as the examination of partial dependence plots or the analysis of the time evolution. The independent expert assessment broadly confirmed these limitations, particularly regarding temporal evolution and the connection of the results with dam-safety assessment. Although one evaluator identified a clear improvement in R2, the assessments also showed that the generated interpretations still require expert verification before being used in a dam-safety context. This highlights the limitations observed for ChatGPT in solving complex, domain-specific engineering tasks.