Model Validation of a Single Degree-of-Freedom Oscillator: A Case Study

Round 1

Reviewer 1 Report

General comments:

Paper does a good job linking dynamical systems terminology and Bayesian approaches.

The notion of a thoughtful, internal “discrepancy” model is a good one.  And this paper makes a good presentation of the idea.  

Clearly differentiates observation error and structural error.

Shows how inference can be carried out regarding model predictions using an incorrect (non oracle) model, and the consequences for this particular system.

Demonstrates the advantage of developing error models that are internal to the modeled system to account for the differences between model and reality.

Comments:

Nice use of a “reduced order” (non-oracle) model.  Is it worth connecting/mentioning UQ with the reduced order models literature?

Ref 6: Is Bayarri spelled correctly?

The approach of this paper also seems quite close to that of Wikle and colleagues (e.g. Wikle et al 1998, below).  Is it worth making this connection in the paper?  I’m not certain it is.  Just an optional suggestion.

Wikle, Christopher K., L. Mark Berliner, and Noel Cressie. "Hierarchical Bayesian space-time models." Environmental and ecological statistics 5, no. 2 (1998): 117-154.

In seeing a dynamical system, I was expecting the “validation” to look at predictions a few time-steps ahead.  But this validation really considers the data and process at all N time steps at once (do I have that right?), under an uncertain forcing (and model parameters).  I think that’s fine, but it may help readers to make a bigger point of this in describing the validation problem and simulation study.

Here’s another (seemingly related) reference along this line that might be of interest to readers.  Have a look and see if it makes sense to cite this one as well.

Morrison, Rebecca E., Todd A. Oliver, and Robert D. Moser. "Representing model inadequacy: A stochastic operator approach." SIAM/ASA Journal on Uncertainty Quantification 6, no. 2 (2018): 457-496.

Author Response

Thanks a lot for your very helpful comments

Author Response File: Author Response.pdf

Reviewer 2 Report

Please see the attached file.

Comments for author File: Comments.pdf

Author Response

Thanks a lot for your very helpful comments

Author Response File: Author Response.pdf

Reviewer 3 Report

The paper applies a well established bayesian calibration/updating approach to simulation study using a single degree-of-freedom oscillator problem. Quantifying the model misspecification uncertainty using the Bayesian calibration approach has been studied in many areas and a comprehensive framework was introduced in “Bayesian calibration of computer models” by Kennedy, M.C. and O'Hagan, A., to the best of my knowledge. So the proposed approach is not particularly interesting to me and I don’ agree with line 240 “This work introduced a novel approach…“. 

However, the case study presented in the paper is well written and clearly demonstrated the effectiveness of the proposed approach, I have the following suggestions:

1. Line 136 “The following vague prior distribution”. Have the authors conducted any sensitivity study on the priors? Would they lead to the same results? Uniform priors are not necessarily “vague” because they assign equal prior probability to even very extreme values . They are often used to reflect our lack of understanding on the probability distribution. So I think the paper may benefit from more physical justifications on the chosen priors. 

2. Line 139 . A Metropolis-Hastings embedded in a Gibbs 139 sampler is implemented in Matlab to obtain samples from the posterior distribution. Is there any reason to choose this algorithm from the family of MCMC algorithms? The paper could also benefit from more details about the implementation details of the algorithm, or which package/references the authors used for the computation.

3.  Line 145, the authors should discuss the traces and auto-correlations of the MCMC samples. It is necessary to demonstrate that the Markov chains are well mixed and converge to the stationary distributions, otherwise the obtained samples cannot be regarded as the posterior distributions of the parameters. 

4. Line 54 notation: add notations for all the special characters that occurred in the paper, for example, e, sigma, etc.

5. Line 260 , for computationally intensive systems, besides the increasing computation speed, there are surrogate model methods that people usually use to bypass the expensive computation.

Author Response

Thanks a lot for your very helpful comments

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Most of my comments have been successfully addressed. However, a couple of points were not revised:

1. Regarding the impact of the Newmark-scheme parameters (e.g. β, λ), this reviewer is interested in numerical results after a sensibility analysis. Please show how different values for β, λ affect the computational burden.

2. Figures should be formatted with a bigger font size (e.g. Fig. 1, 3, 4, 5)

Author Response

Thanks a lot for your careful review of our paper

Author Response File: Author Response.pdf

Reviewer 3 Report

Thank the authors for addressing my questions. However, I think not all my questions are properly answered. For example:

1. My original comments:

Line 136 “The following vague prior distribution”. Have the authors conducted any sensitivity study on the priors? Would they lead to the same results? Uniform priors are not necessarily “vague” because they assign equal prior probability to even very extreme values . They are often used to reflect our lack of understanding on the probability distribution. So I think the paper may benefit from more physical justifications on the chosen priors. 

Authors response:

We changed Line 382 to be: 

“Using this prior distribution and likelihood specification the posterior distribution is not analytically tractable hence sampling is employed. Note that a uniform prior distribution is employed for $\sigma^2$ to allow for high values (near 10) without the need to worry about extreme values as an Inverse-Gamma, etc. might provide. A Metropolis-Hastings embedded in a Gibbs sampler is implemented in MATLAB to obtain samples from the posterior distribution, with details given in Section \ref{sect-ss}” 

I don’t think my questions are answered. Sensitivity study on the priors? Would they lead to the same results? Are there any physical justification to the selected priors? Why do we want or don’t want to worry about extreme value? 

2. My original comments:

Line 139 . A Metropolis-Hastings embedded in a Gibbs sampler is implemented in Matlab to obtain samples from the posterior distribution. Is there any reason to choose this algorithm from the family of MCMC algorithms? The paper could also benefit from more details about the implementation details of the algorithm, or which package/references the authors used for the computation. 

Line 145, the authors should discuss the traces and autocorrelations of the MCMC samples. It is necessary to demonstrate that the Markov chains are well mixed and converge to the stationary distributions, otherwise the obtained samples cannot be regarded as the posterior distributions of the parameters. 

Authors response:

“

It seems as though these two comments address the same issue and we have addressed both by changing the following text in the manuscript. No packages were used. 

A Metropolis-Hastings embedded in a Gibbs sampler was implemented in MATLAB to obtain samples from the posterior predictive distribution. A total of 30,000 samples were taken from the posterior distribution with the first 20,000 samples discarded as burn-in samples and the remaining 10,000 samples thinned by 10 resulting in a set of 1,000 samples from which all inferences will be made. Sampler diagnostics such as traceplots as well as autocorrelation within chains were examined for convergence, mixing and to determine the thinning rate. 

Again, I think my original concerns are not addressed. 

My question: “Is there any reason to choose this algorithm from the family of MCMC algorithms?”I don’t see any answer for it.

My other question is about  traceplots and autocorrelation plots, and details of the algorithm. The authors merely mentioned “we checked traceplots and autocorrelation, and they look fine” - it does not convince me. In addition, I suggested more algorithm details to be added, but I don’t see any algorithm details, specifically, if Gibbs sampler is used, what’s the derived conditional probability used for sampling? and which algorithm is used? Please provide a reference.

Author Response

Thanks a lot for your careful review of our paper

Author Response File: Author Response.pdf