Applying Monte Carlo Dropout to Quantify the Uncertainty of Skip Connection-Based Convolutional Neural Networks Optimized by Big Data
Round 1
Reviewer 1 Report
This study aimed to apply and evaluate Monte Carlo (MC) dropout, a computationally efficient approach, to investigate the reliability of several skip connection-based Convolutional Neural Network (CNN) models while keeping their high accuracy.
I consider that it is relevant for research and technical community because it has originality and rationality. The manuscript is well written and organized, although I suggest describing the problem, i.e., the motivation that generated this study before line 49.
The paper can be followed without undue difficulty. It presents the work in a logical order, with appropriate section content and headings and is mostly readily readable. Moreover, the manuscript is consistent with the subjects of the “Electronics / Artificial Intelligence”.
After evaluating all manuscript, I have some suggestions and doubts concerning the work:
1 – To improve the visibility of this paper, I suggest changing the keywords, because they are the same of the title of the manuscript.
2 – Line 217: “The R2 value lies between −∞ and 1”. I understand that R2 value lies between −1 and +1.
3 – Line 219: “The models without dropout yield very good results for the training subset”. Please, what do the authors consider as “very good results”? Metrologically, it is meaningless.
4 – I suppose that there is a conceptual flaw related to R2 value and Mean Squared Error (MSE) of the regression. This must be clarified before I recommend the acceptance of this paper. It seems that the authors considered that the linear regressions were significant based only the R2, what not always is true. I suggest that the ANOVA parameters (F-tests) be used. Please, see Table 5 in Oliveira, E.C., 2011. Critical metrological evaluation of fuel analyses by measurement uncertainty. Metrology and Measurement Systems, Vol. XVIII, No. 2, pp. 235–248. http://dx.doi.org/10.2478/v10178-011-0006-4
5 – Table 1: Visually, the difference between CNNinitial and CNNdropout for R2 and MSE are marginal in several basis and cannot guarantee a critical analysis of the performance of the developed models.
6 – I suggest including a section, detailing how the measurement uncertainty was calculated.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Interesting application of MC dropout to a numerical procedure to solve an important and relevant engineering problem.
The main question addressed in this research work, as it is indicated by the title of the article, is to try to quantify the inherent uncertainty that we always have to face when modeling many complex processes related to engineering problems in oil extraction and storage. It is true that except for quite simple reservoir the only alternative is to apply some numerical approximate solutions and the authors have chosen to apply a relatively new numerical approach called mixed GMsFEM (Generalized Multiscale Finite Element Method), where we have to consider several multiscale basis functions to a single coarse grid of the reservoir volume. In this sense, the authors indicate sufficient current references on this technique.
They also give additional references about different efficient and computationally feasible techniques to deal with the important issue of the quantification of uncertainty, both in general deep learning problems and particular real-time applications as seismic, CO2 saturation, and others geological monitoring processes. Apart from the usual factors as variance, entropy and mutual information, one of the best approach could be the Monte Carlo dropout technique, which consists in trying to regularize a given neural network by randomly switching off neurons until an acceptable and stable output is obtained from the inputs.
It could therefore be said that the main technical components of the work in question were already fully developed earlier. So, the main contribution of the authors would be to have considered the use of a certain technique, which has finally been useful to solve the difficulty of managing the uncertainty that occurs in the modeling of these natural oil deposits, where inevitably there are countless data and physical characteristics that we can not know completely. In this aspect, we could say that they have found a way of addressing a specific gap in the field, adding this particular technique to the published material in the area.
Particularly, the specific improvements that the authors contribute with this work are the introduction of a specific and convenient procedure for taking into account the inherent uncertainty on the distribution of pressure in the development of real oil/gas engineering problems, when applying some deep learning tools in combination with the corresponding numerical techniques that modelize the physical processes.
In this aspect, the statistical control of the performance of the developed models seem to be correct, as given in Tables 1 and 2, together with the corresponding numerical solutions showed in the Figures 3 and 4.
Also the structure of the skip connection-based convolutional neural network model is well designed and the final conclusions are reasonable. Of course that this study could be improved and the authors also suggest some possibilities and the end of the conclusions section:
- the use of more dropout ratios to compare with the ratio considered could be the first and more obvious one,
- but the possible theoretical quantification of the aleatoric uncertainty for the particular models developed would be the more important and interesting, because it could serve as model or generalization for many other applications.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
2 – Line 221: The 221 R2 value lies between −∞ and 1. Please insert a reference to support this statement.
4 – I suppose that there is a conceptual flaw related to R2 value and Mean Squared Error (MSE) of the regression. This must be clarified before I recommend the acceptance of this paper. It seems that the authors considered that the linear regressions were significant based only the R2, what not always is true. I suggest that the ANOVA parameters (F-tests) be used. Please, see Table 5 in Oliveira, E.C., 2011. Critical metrological evaluation of fuel analyses by measurement uncertainty. Metrology and Measurement Systems, Vol. XVIII, No. 2, pp. 235–248. http://dx.doi.org/10.2478/v10178-011-0006-4
Authors’ Response: To our knowledge, ANOVA is typically used to test the significance of linear regression models. In this study, our case is mapping an input of 100*9 (permeability field) to an output (basis function) of 900*1 using a skip-connection Convolutional Neural Network (CNN). Since CNN models are non-linear in nature due to using activation functions, ANOVA may not be well-suited for our research. According to lines 250-260, standard deviation, which is the square root of the variance, is used to assess the reliability of the CNN models. In addition, as mentioned in lines 308-311, other indices, such as entropy and negative log likelihood were used, but they yielded meaningless results. It is not feasible to provide a similar table to Table 5 in Oliveira’s work. Our research is based on simulation in which the output is a 900*1 vector. However, there is apparently one output for each of the five case studies investigated in the Oliveira’s work. If the reviewer still insists on ANOVA, we kindly and respectfully request a more detailed explanation of how it could be applied to our study.
The ANOVA approach can be used in linear and non-linear regression. Thus, the authors must ensure that the pertinence of the degree regression (linear, second order, etc...), using two F-tests, was adequate: Mean of Squares of the Regression / Mean of Squares of the Residual and Mean of Squares of the Lack of Fit / Mean of Squares of the Pure Error.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
