Investigation into the Hyperparameters of Error-Based Adaptive Sampling Approach for Surrogate Modeling

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors proposed an adaptive sampling methods to iteratively select design-of-experiment (DoE) points for surrogate modeling based on squared errors. While the idea of adaptive DoE is not new (see, for example, Bayesian optimization methods), the authors applied the approach to engineering applications (building energy consumption prediction) and showed value in this area. The visualizations are well-made and the results are well-explained. Both accuracy and efficiency aspects of surrogate modeling are discussed in the manuscript.

Writing suggestions for authors:

1. Minor English writing issues exist. For example, L11: "can be reduces to" -> "can be reduced to". 

2. No reference in the first three paragraphs when introducing machine learning, surrogate modeling, and energy consumption prediction.

3. Empty block in the flowchart in Figure 1.

4. In Figure 1., there is a dataset split step to partition the dataset into train and test splits. It is not mentioned in the algorithm starting L249. In the later figures, the splits are referred to as Val and Test instead.

Author Response

Reviewer 1

Comments 1:  Minor English writing issues exist. For example, L11: "can be reduces to" -> "can be reduced to".

Response 1: Thank you for pointing this out. Minor English corrections applied to the text.

 

Comments 2: No reference in the first three paragraphs when introducing machine learning, surrogate modeling, and energy consumption prediction.

Response 2: References added to the Introduction section, new text highlighted in green color.

 

Comments 3: Empty block in the flowchart in Figure 1.

Response 3: Thank you for pointing this out. Empty block is used for an increment of iterations. Figure 1 is edited.

 

Comments 4: In Figure 1., there is a dataset split step to partition the dataset into train and test splits. It is not mentioned in the algorithm starting L249. In the later figures, the splits are referred to as Val and Test instead.

Response 4: Thank you for pointing this out. Algorithm steps are corrected. In the later figures test data is used from train/test split partition, but it should be noted that validation of the trained model is performed on new generated data.

Reviewer 2 Report

Comments and Suggestions for Authors

Referee's Report

Title: Investigation into the hyperparameters of error-based adaptive sampling approach for surrogate modelling 

Authors: Leonid Legashev, Sergey Tolmachev, Irina Bolodurina, Alexander Shukhman and Lyubov Grishina 

MS. Ref. No. modelling-3351828

 

    Dear Authors

    This study is related to a surrogate model for predicting annual energy consumption using the open-source EnergyPlus software and various sampling techniques. The authors present a general algorithm for an error-based adaptive sampling technique to build this surrogate model and observed that the best results were shown by the composite Mixed Sampling method with data refining window size of 70% and LightGBM regression model; the best MSE metric value attained is 7.76.

    The manuscript is organized and presented very well.

    The usage of data science and coding the model using Python tools making the results more presentable and clearer.

    The work has the potential to help develop innovative transportation services in future cities, as well as improve urban mobility and energy efficiency.

    The manuscript has the potential to be published in Modelling and can I strongly recommend the manuscript for publication in the "Modelling" in its current form.

    sincerely yours,

    Reviewer

Author Response

Thank you for your comments. We are hope that our manuscript will contribute to the "Modelling" journal.

Reviewer 3 Report

Comments and Suggestions for Authors

Dear Authors/Editor,

Please find my comments in the attachment.

Regards 

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

In simpler terms, the authors explain a method to create a simplified model (surrogate model) to predict a complex system, like energy consumption in a building. They used a specific example from EnergyPlus to demonstrate this method. 

To build this simplified model, they experimented with different techniques, such as: 

• Sampling techniques: How to choose the data points to use. 

• Sampling methods: The specific strategies for selecting these data points. 

• Regression models: The mathematical models used to fit the data. 

• Data refining window size: How much data to consider at each step of the model-building process. 

Their goal was to create a model that was accurate enough, as measured by the Mean Squared Error (MSE), while using as few data points as possible. 

The paper is well written and clear. The introduction has several bibliographic references, well outlined. 

I would improve it by emphasizing at the beginning that also in many other application cases the choice of hyperparameters is important, just think of the following recently published paper that deals with UAVs and as a network structure the PINNs (Physical-informed Neural Networks), to make it clear that it is a general scale problem and where this problem is faced: 

Bianchi, D.; Epicoco, N.; Di Ferdinando, M.; Di Gennaro, S.; Pepe, P. Physics-Informed Neural Networks for Unmanned Aerial Vehicle System Estimation. Drones 2024, 8, 716. https://doi.org/10.3390/drones8120716. 

I would also specify whether the choice of hyperparameters was made extensively or randomly or with statistical techniques, typical methods used in their choice. 

We also consider the work that is currently being done to develop the web application to be very interesting, hoping to make public another work that can support the dissemination of the research and its use. 

Author Response

Reviewer 4

Comments 1:  I would improve it by emphasizing at the beginning that also in many other application cases the choice of hyperparameters is important, just think of the following recently published paper that deals with UAVs and as a network structure the PINNs (Physical-informed Neural Networks), to make it clear that it is a general scale problem and where this problem is faced.

Response 1: Thank you for pointing this out. Suggested publications added to Introduction section, new text highlighted in green color.

 

Comments 2:  I would also specify whether the choice of hyperparameters was made extensively or randomly or with statistical techniques, typical methods used in their choice.

Response 2: The choice of hyperparameters was made extensively, based on our experience in surrogate modelling of different software and is consistent with other studies.

 

Comments 3:  We also consider the work that is currently being done to develop the web application to be very interesting, hoping to make public another work that can support the dissemination of the research and its use.

Response 3: Thank you for your interest in our work of developing a web application for surrogate modeling. We hope to publish more results on this topic in the future.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

The revised version shows significant improvements; all my comments have been addressed. However, I have a minor comment: Could the authors please include the table presented in the revision letter in the main manuscript file?

Author Response

Reviewer 3

Comments 1:  The revised version shows significant improvements; all my comments have been addressed. However, I have a minor comment: Could the authors please include the table presented in the revision letter in the main manuscript file?

Response 1: Table with calculated ANOVA value added to the manuscript. New text highlighted in green color.