Assessment of the Generalization Ability of the ASTM E900-15 Embrittlement Trend Curve by Means of Monte Carlo Cross-Validation

The standard ASTM E900-15 provides an analytical expression to determine the transition temperature shift exhibited by Charpy V-notch data at 41-J for irradiated pressure vessel materials as a function of the variables copper, nickel, phosphorus, manganese, irradiation temperature, neutron fluence, and product form. The 26 free parameters included in this embrittlement correlation were fitted through maximum likelihood estimation using the PLOTTER—BASELINE database, which contains 1878 observations from commercial power reactors. The complexity of this model, derived from its high number of free parameters, invites a consideration of the possible existence of overfitting. The undeniable goal of a good predictive model is to generalize well from the training data that was used to fit its free parameters to new data from the problem domain. Overfitting takes place when a model, due to its high complexity, is able to learn not only the signal but also the noise in the training data to the extent that it negatively impacts the performance of the model on new data. This paper proposes the resampling method of Monte Carlo cross-validation to estimate the putative overfitting level of the ASTM E900-15 predictive model. This methodology is general and can be employed with any predictive model. After 5000 iterations of Monte Carlo cross-validation, large training and test datasets (7,035,000 and 2,355,000 instances, respectively) were obtained and compared to measure the amount of overfitting. A slightly lower prediction capacity was observed in the test set, both in terms of R² (0.871 vs. 0.877 in the train set) and the RMSE (13.53 °C vs. 13.22 °C in the train set). Besides, strong statistically significant differences, which contrast with the subtle differences observed in R² and RMSE, were obtained both between the means and the variances of the training and test sets. This result, which may seem paradoxical, can be properly interpreted from a correct understanding of the meaning of the p-value in practical terms. In conclusion, the ASTM E900-15 embrittlement trend curve possess good generalization ability and experiences a limited amount of overfitting.