Machine-Learning Approach to Determine Surface Quality on a Reactor Pressure Vessel (RPV) Steel

Round 1

Reviewer 1 Report

The practical significance of the work is beyond doubt. The aim of this work is to find a correlation between surface roughness and magnetic behavior. The authors propose to solve the problem set using machine learning methods: neural networks and classification trees. However, the rationale for using the chosen methods for solving the problem is not given.

Also, the choice of neural network architecture used in the work is not described. It is not clear how the selected number of neurons in the input layer is related to the dimension of the input data.

Author Response

Reviewer 1

The practical significance of the work is beyond doubt. The aim of this work is to find a correlation between surface roughness and magnetic behaviour. The authors propose to solve the problem set using machine learning methods: neural networks and classification trees.

Many thanks for these comments but also a very important addition is the use of different spacers to mitigate against surface roughness this is especially necessary when considering the field conditions. As this is not clear and based on your following suggestion, changes will be made to make this more clear.

1. However the rationale for using the chosen methods for solving the problem is not given.

This is fair point and thank you for suggesting additions/modifications to help the paper readability. The following paragraphs from lines 70 -89 have been added to address such issues.

For safety critical components, it is important to characterise the presence of any surface defects including cracks, manufacturing flaws, service-induced cracking or suspected degradation as these defects can initiate and grow during service and may cause catastrophic failure by fracture. Hence, most of the structural integrity assessment methodologies tend to be highly conservative. By accurately predicting the surface quality levels it is possible to obtain more reliable NDT measurements, since this creates an opportunity to feed back this information to the signal evaluation and to mitigate this side effect, as well. By manufacturing different surface roughness profiles on the samples it is possible to study the effects on the magnetic response.

Another relevant issue that the application of machine learning approach faces with NDT measurements is the lack of the sufficient scale of data that is obtained and can be used for teaching. This is why the next two subsections address methods of increasing the data set through machine learning methods to increase the data amounts in terms of data reinforcement. With the data space being highly non-linear in nature, ML methods are also used to provide outputs based on the obtained input NDT data. A comparison is made between these ML methods in what is termed as good artificial data reinforcement and what is not.

The work proposed in this paper will therefore tackle a number of different engineering problems, namely the encountered inspection field conditions of RPV steels, the lack of data experienced in most measurement campaigns and, the associated curse of dimensionality. For this scoping study, the verification was made from visual representations to show change/improvements as well as metrics to display accuracy and precision differences.  

2. Also, the choice of neural network architecture used in the work is not described. It is not clear how the selected number of neurons in the input layer is related to the dimension of the input data.

We thank you for this question and I have to say it’s a very important question and often overlooked in NN work. First of all the sizes made in the text are incorrect and based on other work. These have been updated and so many thanks for this as it was overlooked by both myself and the other authors. Ok so we have 6 inputs, 36 inputs to the first hidden layer and 72 inputs to the second hidden layer and 36 inputs to the third hidden layer. This is still larger than the maximum 6 input variables used but due to the nature of the data, there is information included which segregates the data in different sets using the associated film data and therefore, this is classed as more complex than from straight forward data from a single measurement system. To test for issues of over fitting, metrics are displayed that have been taken from the test data. It is not fully understood why the NN performs better using more neurons than what is present as the number of input variables. A trial and error approach was adopted for this work as we would like to show the performance of the technologies especially when confronted with the ‘curse of dimensionality.’ This work is very much a forerunner of ideas however it has pointed out a number of areas for more focused studies.   

The following has therefore been modified and added in red:

The Neural Networks used were four layer networks with two hidden layers. The input layer was 6 neurons, with the hidden layers increasing to 36, 72 and 36 neurons respectively. The output number was based on the number of output values/classes. These values were chosen to address all the different pattern variations between the presented input data. All of the neuron transfer functions for the first three layers used tan-sigmoid and for the final output layer, pure-linear. In terms of neural network parameters, maximum epochs were set to 20000, learning rate set to 0.1 * 10^-10, momentum set to 0.8 and finally in terms of learning rules, resilient back-propagation along with Kohonen weight learning function was used for training the network. The input layer has been chosen to be more than the amount of input variables due to the nature of the data where there are different measurements made from using different layer technologies and this needs to be considered when segregating the total pattern space. Also by using a trial and error approach, six neuron inputs did not provide useful results where thirty-six neuron inputs gave good account and trade-off against further increases. Using this trial and error approach to display performances, metrics were obtained from testing the network with unseen data. Such results are used to display data generalisation as opposed to data fitting.     

Also another error was noted where the columns of data should have included the layer technology parameters and this has been updated also.     

Reviewer 2 Report

This paper investigates the effect of different processing parameters on surface quality levels of nuclear reactor materials manufactured from the milling Process. However, the following questions need to be answered.
1. The abstract of the paper is ponderous. It is suggested to the author to simplify the description and focus on the main idea of the paper.
2. It is suggested to the authors to show the purpose of the experiment at the beginning for better reading experience.
3. According to the title of the paper, why it is important to distinguish the surface quality levels on Reactor Pressure Vessel Steel? The background of the paper should depict the necessity and the importance of the researches.
4. The accuracy of the predictions in Table 2 and Table3 is not 100%, which is an inevitable problem of machine learning. What are the consequences of failing to predict the surface quality levels of specific Reactor Pressure Vessel, and is it affordable?
5. There are some spelling errors in the paper. The quality of the figures and the formulas need to be improved.  The authors are suggested to make a thorough check.

Author Response

Reviewer 2

This paper investigates the effect of different processing parameters on surface quality levels of nuclear reactor materials manufactured from the milling process. However, the following questions need to be answered.

1. The abstract of the paper is ponderous. It is suggested to the author to simplify the description and focus on the main idea of the paper.

The abstract has been modified and improved to address the total paper connectivity please see the new abstract below:

Surface quality such as roughness, especially its uncertain character, affects most magnetic non-destructive testing methods and limits their performance in terms of achievable signal to noise ratio and reliability. This paper is primarily focused on an experimental study targeting nuclear reactor materials manufactured from the milling process with various machining parameters to produce varying surface quality conditions to mimic the varying material surface quality of in field conditions. From energising a local area electro-magnetically, a receiver coil is used to obtain the emitted Barkhausen noise from which the condition of the material surface can be inspected. Investigations were carried out with the support of machine learning algorithms such as Neural Networks (NN) and Classification and Regression Trees (CART) to identify the differences in surface quality. Another aspect often faced with small experimental  data measurements is the amount of data obtained. Other non-destructive methods such as Magnetic Adaptive Testing (MAT) was used to provide data imputation of missing data using other intelligent algorithms. For data reinforcement, data augmentation was used. With more data the problem of ‘the curse of data dimensionality’ Is addressed. It was demonstrated how both data imputation and augmentation can improve measurement data sets.

2. It is suggested to the authors to show the purpose of the experiment at the beginning for better reading experience.

I agree with your statements, there is a lot going on with this paper and the flow is easily lost. By adding the following between lines 86 to 92 it should give the reader a more clearer picture of what will follow:

Another issue that is faced with NDT measurements is amount of data that is obtained and this is why the next two subsections address methods of increasing the data set through machine learning methods to increase the data amounts in terms of data reinforcement. With the data space being highly non-linear in nature, ML methods are also used to provide outputs based on the obtained input NDT data. A comparison is made between these ML methods in what is termed as good artificial data reinforcement and what is not.  

3. According to the title of the paper, why is it important to distinguish the surface quality levels on reactor pressure vessel Steel? The background of the paper should depict the necessity and importance of the researches.

Thank you for the advice and so we have included the following paragraph at the end of the introduction between lines 72-78:

For safety critical components, it is important to characterise the presence of any surface defects including cracks, manufacturing flaws, service-induced cracking or suspected degradation as these defects can initiate and grow during service and may cause catastrophic failure by fracture. Hence, most of the structural integrity assessment methodologies tend to be highly conservative. By accurately predicting the surface quality levels it is possible to obtain more reliable NDT measurements that can avoid being overly conservative with fracture assessments and thus improve the possibility of life extension of existing power plants. By manufacturing different surface roughness profiles on the samples it is possible to see the effects magnetic response.  

4. The accuracy of the predictions in Table 2 and Table 3 is not 100%, which is an inevitable problem of machine learning. What are the consequences of failing to predict the surface quality levels of specific reactor pressure vessel, and is it affordable?

This is a fair question and more than likely one from a reader not from the nuclear sector therefore the following text has been addressed to tackle this (added at the end of Section 5 lines 70-77):

For safety critical components, it is important to characterise the presence of any surface defects including cracks, manufacturing flaws, service-induced cracking or suspected degradation as these defects can initiate and grow during service and may cause catastrophic failure by fracture. Hence, most of the structural integrity assessment methodologies tend to be highly conservative. By accurately predicting the surface quality levels, it is possible to obtain more reliable NDT measurements, since this creates an opportunity to feed back this information to the signal evaluation and to mitigate this side effect, as well. By manufacturing different surface roughness profiles on the samples it is possible to study the effects on the magnetic response where error sources are simulated bridging the gap to in-field conditions. 

5. There are some spelling errors in the paper. The quality of the figures and the formulas need to be improved. The authors are suggested to make a thorough check.

Many thanks for noting these issues; the authors have read though the paper, figures have been checked and any spellings noted and changed accordingly.

Reviewer 3 Report

This work tried to consider machine learning techniques to identify the material surface qualities (i.e., surface roughness; by means of regression) using the factors in the milling process with various machining parameters. The author addressed the data augmentation and imputation so as to enhance the performance of the prediction. 

Overall, I don't think the manuscript meets the merit of Applied Sciences. The result is neither sound nor interesting. The writing must be greatly improved. It is very difficult to read. 

Other comments:

• The reference should be numbers. For example, [1], [2], ...
• Remove the dot before the section name (Section 1.0 and 6.0).
• Page 2, Line 61: A citation here is missing. 
• Page 2, Sect. 1.1:  It is not clear about what the first paragraph wants to speak for. It should be improved. 
• Page 2, Line 74: ths datais => this data is
• Page 2, Line 80: isinvestigated => is investigated 
• Page 2, Line 83-84: "To date, there is .... curse of dimensionality'. I am afraid that, maybe, it cannot be solved using data augmentation. 
• Page 2, Line 88-100: Please improve this paragraph which is very difficult to understand. 
• Page 3, Line 101: ... is different to ... => ... is different from ...
• Page 5, Line 213: NDE needs to be defined. 
• Page 5, Line 225: Remove 'Although'
• Page 6: Please define and explain Ra, Rz and Rsm. Also, RMS needs to be defined at Line 258. 
• Page 6, Line 261: The position of the superscript of y_i^2 is too high.
• Page 6 (throughout the manuscript): Please italicize the math variables. 
• Page 9, Line 322-324: "... as the retrieved signal to noise ratio is at ... is ignored)." I don't understand this sentence and I think it must be improved. 
• Page 9, Line 335: "The aim of the work is ..." => It seems not a good idea to postpone the aim of this work here. 
• Page 8, Line 343-345: Please explain why Figure 8 gave best correlation of surface roughness. 
• Page 10, Line 350-: "The next Section applies ... " The writing of this whole paragraph must be improved. 
• Page 10, Line 362: Please remove the 'x' or ''. I suggest not to use  as the multiplication operator. 
• Page 10, Line 364: Please explain what FFT and ps_i are.
• Page 12, Line 374-387: Please improve the writing of this paragraph. 
• Page 13, Line 409: Please define DBTT, USE, ... 
• Page 15, Line 470: "the sum of target minus actual" => the sum of the difference between the predicted value and the real value.
• Page 15, Line 481: Please do not use '*' as the multiplication operator. 
• Page 16: Please explain Tx, TXLx and SSE (SSE is used here so it must be defined).

Author Response

Reviewer 3

This work tried to consider machine learning techniques to identify the material surface qualities (i.e., surface roughness; by means of regression) using the factors in the milling process with various machining parameters. The author addressed the data augmentation imputation so as to enhance the performance of the prediction.

Overall, I don't think the manuscript meets the merit of Applied Sciences. The result is neither sound nor interesting. The writing must be greatly improved. It is very difficult to read.

I am sorry you feel this work is ‘neither sound nor interesting’ however based on your follow on sentence I believe this is due to the fact the writing of the paper must be greatly improved as its very difficult to read. I believe this is due to the fact there is a lot going on in this paper and rather than a single focus there are several which relate to the associated problems when taking electro-magnetic measurements on RPV structures. Keeping that in mind and your comments  - I have made the following changes:

For safety critical components, it is important to characterise the presence of any surface defects including cracks, manufacturing flaws, service-induced cracking or suspected degradation as these defects can initiate and grow during service and may cause catastrophic failure by fracture. Hence, most of the structural integrity assessment methodologies tend to be highly conservative. By accurately predicting the surface quality levels it is possible to obtain more reliable NDT measurements, since this creates an opportunity to feed back this information to the signal evaluation and to mitigate this side effect, as well. By manufacturing different surface roughness profiles on the samples it is possible to study the effects on the magnetic response. 

Another relevant issue that the application of machine learning approach faces with NDT measurements is the lack of the sufficient scale of data that is obtained and can be used for teaching. This is why the next two subsections address methods of increasing the data set through machine learning methods to increase the data amounts in terms of data reinforcement. With the data space being highly non-linear in nature, ML methods are also used to provide outputs based on the obtained input NDT data. A comparison is made between these ML methods in what is termed as good artificial data reinforcement and what is not.

The work proposed in this paper will therefore tackle a number of different engineering problems, namely the encountered inspection field conditions of RPV steels, the lack of data experienced in most measurement campaigns and, the associated curse of dimensionality. For this scoping study, the verification was made from visual representations to show change/improvements as well as metrics to display accuracy and precision differences.  

In summary, these problems have not been tackled date and need to be tackled so we can achieve higher TRL levels and safer inspection systems for safety critical power systems. 

Other comments:

1. The reference should be numbers. For example, [1], [2],...

This has been done – many thanks.

2. Remove the dot before the section name [section 1.0 and 6.0]

Many thanks for noting this issue which has now been modified.

3. Page 2, line 61: A citation here is missing.

Citation has now been added via referencing system.

4. Page 2, section 1.1: it is not clear about what the first paragraph wants to speak for. It should be improved.

Thank you for this point and the following has been modified:

A Relevant facet of obtaining sensor information is the amount of data recorded. In most cases the data is not sufficient for statistical approaches like machine learning techniques to find general trend patterns as opposed to simple data fitting. Jakobsen et al. discussed that this must be done in an intelligent fashion otherwise unwanted data bias may be found (Jakobsen, J.C., Gluud, C., Wetterslev, 2017). One way to tackle the small data sets is from applying imputation to increase data (McNeish, 2017).

5. Page 2. Line 74: ths datais => the data is

Many thanks for pointing this out. This has now been changed.

6. Page 2, line 80: isinvestigated => is investigated

Many thanks for pointing this out. This has now been changed.

7. Page 2 quote, line 83 to 84 “to date, there is... Curse of dimensionality.” I'm afraid that, maybe, it cannot be solved using data augmentation.

I can see your point – but this is the very reason why it is being tackled in this work in that data augmentation as well as data imputation are methods to address the curse of dimensionality through increased data reinforcement where the displayed results show promise. This has been addressed several times within the text.   

8. Page 2, line 88 to 100: please improve this paragraph which is very difficult to understand.

This paragraph has also been addressed

9. Page 3, line 101: … is different to … => … is different from

This was found on line 98 and has now been modified – many thanks

10. Page 5, line 213: NDE needs to be defined.

This was found on line 200 and has now been expanded and not abbreviated as it only feature once – many thanks for pointing out this.

11. Page 5. Line 225: removed ‘Although’

This was found on line 13 and has now been removed. Many thanks.

12. Page 6: please define and explain Ra, Rz and Rsm. Also, RMS needs to be defined at line 258.

All of these terms have been expanded in Table 1 to ease understanding to the readership – many thanks.

13. Page 6, line 261: The position of the subscript of y_i^2 is too high.

Thankyou for pointing this out and has been modified.

14. Page 6 (throughout the manuscript): Please italicise the math variables.

This has also been checked and carried out.

15. Page 9, line 322 to 324: “… as the retrieved signal to noise ratio is at... Is ignored).” I don't understand this sentence and I think it must be improved.

This was found at line 320 and the following sentence has been modified “At the point of 220 µm this could be considered as the limit for BN elecro-magnetic process whereas the retrieved response signal to noise ratio is at this point tending towards the levels of background electro-magnetic noise (where any BN reading that is within 5 times of background noise is ignored).”

16. Page 9, line 335: “The aim of the work is...” => it seems not a good idea to postpone the aim of the work here.

This has been removed and positioned much earlier on between lines 312 and 315:

The aim of the work is to investigate whether a correlation can be found between surface roughness and magnetic behaviour, and further evaluate, the role of a spacer to reduce the effect of surface roughness by providing a uniform airgap. The evaluation was made through direct experimental results and machine learning paradigms.

17. Page 8, Line 343 to 345: Please explain why Figure 8 gave the best correlation of surface roughness.

Thank you for this comment – the following from line 360 to 361 was added:

where the measured signal response correlated with the measured surface roughness

18. Figure 10, line 350-: “the next six Section applies...” The writing of this whole paragraph must be improved.

Thank you for these comments – bullets points have been added and formatting to segregate one parameter introduction from another.

19. Page 10, Line 362: Please remove the ‘x’ or “.” I suggest not to use as the multiplication operator.

Thankyou for pointing out this, it has been removed.

20. Page 10, line through 64: please explain what FFT and ps_i are.

Again, thank you for pointing this out, the following sentence has been added, “where ps is the power spectrum and FFT is Fast Fourier Transform.”

21. Page 12, line 374 to 387: please improve the writing of this paragraph.

Thank you for pointing this out and the following paragraph has been modified:

Figure 4 was identified as the best BN response in terms of placed sensor direction to distinguish the different surface material conditions and for these reasons; the extra parameter study was carried out for this measurement (including bottom surface, transversal sensor orientation). Figure 4 is based around transversal measurements from the bottom side and hence why the Figures 6, 7 and 8 are extra parameters obtained for the transversal measurements and should give more distinguishing features all correlated to the RMS BN response of Figure 4. These results provide an information structure to discerning which signal responses relate to which material and associated surface quality. The information correlated is in terms of extra dimensions of data. This data is therefore considered useful for studying the effects of n-dimensionality on data and, from increasing that data with augmentation to overcome such effects. Section 5 will explore these effects in terms of the two chosen machine learning techniques of neural networks and CART. First, Section 4 looks at data augmentation in the form of imputation of missing data provided from similar electro-magnetic quantities such as those given by the MAT technique.

22. Page 13, line 409: please define DBTT, USE,...

The following has been expanded: “Ductile to Brittle Transition Temperature (DBTT), Upper Shelf Energy (USE)”

23. Page 15, line 470: “the sum of the target minus actual” => the sum of the difference between the predicted value and the real value.

This has been changed and I admit your account was a lot clearer than mine! Again, thank you for pointing this out.

24. Page 15, line 481: please do not use ‘*’ as a multiplication operator.

These have been removed from Tables 2, 3 and 4.

25. Page 16: please explain Tx, TXLX and SSE (SSE is used here so it must be defined).

These expansions have been added to Tables 2, 3 and 4 for more clarity to the reader. Many thanks for pointing these out as they certainly improve the readability.