A Novel Approach for Classification and Forecasting of Time Series in Particle Accelerators

Round 1

Reviewer 1 Report

The authors aimed in this work to capture the precursor of beam interruptions of the High Intensity Proton Accelerators by classifying whether the accelerator is in stable or unstable operation mode. A novel approach, a Recurrence Plots based Convolutional Neural Network is introduced and adapted to the problem setting. 

I have some minor comments:

Please provide a justification for giving figure 1 in connection with the study.

"Figure 3 show the four types and their statistics in the considered dataset that contains 2027 interlocks in total." & "Figure 3. Distribution of the interlock events by type. 'Other Types' denotes all interlock types that
are not prevalent enough to have a separate category. Note that an interlock may be labeled as more than one type." - by my calculations less than 4% of the interlocks were labeled with more than one type. This is a very low rate of an event to be considered regular. Actually is below 5% so is in the risk of being in error. Because of this, please provide a much more detailed insight of the phenomena before regarding this situation as normal.

Even more, "Other" category I'm assuming that did not mix with the others, so as are now the categories does not fall into equiprobable type. You may consider to move the crosses into the "Other" category.

Please specify in relation with Figure 4 & "... by taking the last point that was recorded before each point in the grid ..." the points in question.

Please provide more details about bootstrapping (in relation with Table 3).

"Figure 6 explains the process of transforming an original signal to a recurrence plot, with fixed e = 2" - Figure 6 doesn't explain by itself. Please provide more.

Please explain also the asymmetry (e.g. ||x_3 - x_1|| <> ||x_1 - x_3||) in Fig.6.

Algorithm 1 - is a hybrid of detailed steps (for - end captions) and syntactical recipes (ex. "Repeat the subsampling among the currently selected set, to obtain the best performing set;" may be also specified with syntax 'Repeat - until' for instance). Please consider to be uniform in the depth of the description of the algorithm.

If Table 6 is suppose to contain occurrences of certain contingent events (e.g. True Positive, False Negative, etc.) then it fails miserably because the number of occurrences is by all means a nonnegative integer. 

Please use a symbolic math calculator for the eq.2 and please provide only the beginning and the end of the calculation.

p = 1/46 - in relation with eq.2 - what about its confidence interval? is small, is big? is containing 0?

"Currently, a GUI for real-time implementation" - please detail the acronym used.

Appendix Figure A1 - "Interlock distribution per day" - type type of the interlock is not depicted; it is more appropriate 'interlock occurrences'

Please provide the program associated with the algorithm in the appendix.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

“A Novel Approach for Classification and Forecasting of Time Series in Particle Accelerators”

Comments to the Author This is an interesting paper that presents  a time series classification approach applied to decrease beam time loss in the High Intensity Proton Accelerator complex by forecasting interlock events. The paper could be potentially publishable in "Information" subject to some revisions that are discussed in more detail below.

I have a couple of suggestions that may improve the paper.

Major concerns:

1. Preprocessing:

“The selection was done by a combination of random search and evaluation of past experimental results” Have the author considered other techniques? The random selection may be improved using the reduction of dimensionality of the channels based in techniques similar to factor analysis. Furthermore, the channel selection derived from Algorithm 1, may be a combination of channels that presents high correlation problems.

2. Results

The area under the ROC curve may not be enough to conclude the goodness of fit of the proposed technique. The point estimate of the area under the curve is 0.710.71 ± 0.01 with the RPCNN method compared to an Area under the ROC Curve (AUC) value of 0.65 ± 0.01. Still is not enough to consider the method efficiently. Confidence intervals must be provide to conclude if there are differences between both methods and to be able to affirm that the area under the ROC curve is significantly greater than the 0.5 minimum enforceable.

Table 8 may gain in interpretability if the associated confusion matrix is presented.

Please explain the reasoning underlying this assumption:  

“During the HIPA run from September to December 2019, there are 894 interlocks of the “Losses” type, ignoring the secondary ones. Taking the result of the best RPCNN model, 0.5 seconds would be saved for each interlock, which means we would deliver 7.45 minutes more beam altogether.” This would be the case in which the 100% of the interlocks of the “Losses” type could be detected but the proposed methodology doesn’t have this capacity.

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This manuscript proposes a method to predict the interlocks samples by first transforming the input data in to recurrence plots and then applying a convolutional neural network to implement a classification model.

I believe that it brings a good contribution to this area of research, but before publication there are some aspects that need to be addressed. For instance:

• At the end of section 2, it must be made clear that the interlocks data is being used as input for the RPCNN model and that this data is being transformed into recurrence plots.
• I would suggest to change the name of section 2.3 to “RPCNN model”, and provide a more detailed explanation of this model and how it fits within the overall objective of your work.
• Then describe the “preprocessing” as sub-section 2.3.1, combine the “theory” for recurrence plots sub-section with the “RP generation” sub-section as 2.3.2, and finally the “classification” sub-section as 2.3.3.
• Considering an imbalance ratio of almost 50 to 1, isn’t it problematic to use sampling with resampling?
• Considering the vast area of research behind the imbalance problem for classification tasks, have you tried other sampling methods? 
• Please argument your decision in selecting this sampling method and if the prediction results can be considered reliable.

Minor suggestions:

• Please number all equations.
• For a clearer picture of the results, please include the true negatives (TN) in Table 8.
• Please move the appendix section after references.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Manuscript seems ready for publication.

Reviewer 2 Report

The revision has improved the quality and clarity of the manuscript. Good work!

Reviewer 3 Report

The revision has improved the quality and clarity of the manuscript. Good work!