Power Prediction of Airborne Wind Energy Systems Using Multivariate Machine Learning

Round 1

Reviewer 1 Report

In this paper the authors employ machine learning algorithms for the prediction of the power generated by AWE systems. The paper is well-written and easy to follow. The main issues is every ML/DL model has several hyper-parameters. The overall performance of all ML/DL model heavily depends on this parameters. Could the author comments on the impact of these parameters such as the number of layers, number of neurons, learning rate, etc.There are minor issues which can be discussed in more details:

• Line 306: "linear activation function"...Could the authors please explain more about this phrase?
• in table 5, what is the unit of train_time
• what is the number of epoch in NN? all these parameters should be specified in a table.
• In figure 21, one of the inputs of NN is time. Could the author explain more on that?

Author Response

Response to review #1

*******************
Response: First, we would like to thank the reviewer you for their valuable comments and recommendations that drastically improved the quality of the manuscript.
*******************

In this paper the authors employ machine learning algorithms for the prediction of the power generated by AWE systems. The paper is well-written and easy to follow. The main issues is every ML/DL model has several hyper-parameters. The overall performance of all ML/DL model heavily depends on this parameters. Could the author comments on the impact of these parameters such as the number of layers, number of neurons, learning rate, etc. There are minor issues which can be discussed in more details:

*******************
Response: Thank you.
> We have indeed added a few remarks to explain the significance of some parameters and their impact on the computational cost of training an ML model. Please see Table 4 and the associated text.
*******************

- Line 306: "linear activation function"...Could the authors please explain more about this phrase?

*******************
Response: Thank you.
> We have further explained the need for activation functions in ML models.
*******************

- In table 5, what is the unit of train_time

*******************
Response: Thank you.
> Using standard time.process_time(), train_time reports the value (in fractional seconds) of the sum of the system and user CPU time of the current process. The main takeaway here is observing the "relative" relations rather than absolute run times.
*******************

- What is the number of epoch in NN? all these parameters should be specified in a table.

*******************
Response: Thank you.
> In the neural network terminology: one epoch = one forward pass and one backward pass of all the training examples. So it is considered as the iterations until convergence.
> We added many computational remarks to further explain some of the key NN model parameters, and refer to the included citations for the comprehensive definitions to maintain the focus of this manuscript.
*******************

- In figure 21, one of the inputs of NN is time. Could the author explain more on that?

*******************
Response: Thank you.
> We added time to perform the test as a feature, even though we know it will not affect the output. However, we found that from the correlation heat map that the time is not highly correlated with the output tension.
*******************

Reviewer 2 Report

The authors provide a lengthy, well written introduction of the field of Airborne Wind Energy (AWE) but there are some gaps in there references. When introducing pumping-cycle operation with figure-of-eight crosswind maneuvers, it would be appropriate to cite the work of Fagiano et al. (e.g. Fagiano, Lorenzo, et al. "Automatic crosswind flight of tethered wings for airborne wind energy: Modeling, control design, and experimental results." IEEE Transactions on Control Systems Technology 22.4 (2013): 1433-1447.). The authors claim that there is a lack of papers in the field of AWE that are based on experimental data or data-driven methods and do not reference the work on real-time model identification of Wood et al. (e.g. Wood, Tony A., Henrik Hesse, and Roy S. Smith. "Predictive control of autonomous kites in tow test experiments." IEEE control systems letters 1.1 (2017): 110-115.). Most crucially however the authors write that their work is the first attempt in the AWE community to employ machine learning methods and therefore apparently are not aware of the work of Diwale et al. (e.g. Diwale, Sanket Sanjay, Ioannis Lymperopoulos, and Colin N. Jones. "Optimization of an airborne wind energy system using constrained gaussian processes." 2014 IEEE Conference on Control Applications (CCA). IEEE, 2014.)

The main contribution of the paper is a presentation of a prototype AWE system, a detailed presentation of tow-test experiments, data analysis and the motivation of using machine learning techniques to model the produced tether force.

It would be good for the authors to clarify how the kite is controlled during the experiments. It is mentioned that the kite is launched manually and that it is controlled remotely during the rest of the tests. What does this remote controlling consist of. How are the figure-of-eight paths achieved? Is a pre-defined path followed? Are the paths the same for different wind speeds?

The experiment data is presented nicely. Some more interpretation of the data processing would be helpful however (the Pearson correlation results in Figure 18 and the Keras model summary in Table 4 for instance). What is the loss function used in Figure 22? What can be said about the voting regressor model that achieves the smallest mean squared error?

The authors assume statistical input independence. This sound like a strong assumption. It would be helpful if the authors could comment on this. Furthermore, it is not clear how the sensitivity analysis is incorporated if at all in the modelling of the tether force model.

A lot of data is provided and many different off-the-shelf methods are tested. The drawn conclusion that the neural network is accurate is not obvious from the presented results.

Overall, the experiment design is valuable and the idea of applying learning-based methods in order to model the tether force is interesting. Some refinement in how conclusions are made and a clarification of the novelty of the different aspects of the work would greatly improve the article.

Minor comments:

-Define the variables an acronyms appearing in the labels of the figure axis. Some of the axis are too small to read.

-Page 16, Line 271 'then then'

-Brackets on Page 19, Line 356 not closed. Potentially an overuse of parenthesis throughout the article

Author Response

Response to review #2

*******************
Response: First, we would like to thank the reviewer you for their valuable comments and recommendations that drastically improved the quality of the manuscript.
*******************

The authors provide a lengthy, well written introduction of the field of Airborne Wind Energy (AWE) but there are some gaps in there references. When introducing pumping-cycle operation with figure-of-eight crosswind maneuvers, it would be appropriate to cite the work of Fagiano et al. (e.g. Fagiano, Lorenzo, et al. "Automatic crosswind flight of tethered wings for airborne wind energy: Modeling, control design, and experimental results." IEEE Transactions on Control Systems Technology 22.4 (2013): 1433-1447.). The authors claim that there is a lack of papers in the field of AWE that are based on experimental data or data-driven methods and do not reference the work on real-time model identification of Wood et al. (e.g. Wood, Tony A., Henrik Hesse, and Roy S. Smith. "Predictive control of autonomous kites in tow test experiments." IEEE control systems letters 1.1 (2017): 110-115.). Most crucially however the authors write that their work is the first attempt in the AWE community to employ machine learning methods and therefore apparently are not aware of the work of Diwale et al. (e.g. Diwale, Sanket Sanjay, Ioannis Lymperopoulos, and Colin N. Jones. "Optimization of an airborne wind energy system using constrained Gaussian processes." 2014 IEEE Conference on Control Applications (CCA). IEEE, 2014.)

*******************
Response: Thank you.
> We cited the work of Fagiano et al. when we introduced the pumping cycle.
> We cited the work of Wood et al. as a data-driven control method.
> We recognize the significance of the work of Diwale et al. and have indeed cited it. However, our approach is heavily dependent on experimental data for training our ML model, rather than the simulated GP approach. Furthermore, our model incorporates multiple feature inputs compared to their simple model where the mass is neglected and the flexible tether is substituted by a rigid rod.
*******************

The main contribution of the paper is a presentation of a prototype AWE system, a detailed presentation of tow-test experiments, data analysis and the motivation of using machine learning techniques to model the produced tether force.

It would be good for the authors to clarify how the kite is controlled during the experiments. It is mentioned that the kite is launched manually and that it is controlled remotely during the rest of the tests. What does this remote controlling consist of. How are the figure-of-eight paths achieved? Is a pre-defined path followed? Are the paths the same for different wind speeds?

*******************
Response: Thank you.
> We clarified that the Remote Controller (RC) is handled by a trained human pilot who operated the kite during the experiments.
> The RC sends a signal to the KCU. This signal, in turn, is considered as a control action for the motor, to rotate clockwise or anti-clockwise.
> The figure-of-eight paths were achieved with the sense of the pilot (human-control), so it is not pre-defined, and for sure they are not exactly the same for different wind speeds.
*******************

The experiment data is presented nicely. Some more interpretation of the data processing would be helpful however (the Pearson correlation results in Figure 18 and the Keras model summary in Table 4 for instance). What is the loss function used in Figure 22? What can be said about the voting regressor model that achieves the smallest mean squared error?

*******************
Response: Thank you.
> We added further numerical clarifications to explain the Keras summary model in Table 4.
> The loss function used in Figure 22 is the Mean Squared Error (MSE) between the true function evaluation and the ML model approximation, as given by Equation (6). We clarified that in the figure caption.
> We added further illustrative remarks to the voting regressors, in the paragraph starting at Line: 357.
*******************

The authors assume statistical input independence. This sound like a strong assumption. It would be helpful if the authors could comment on this. Furthermore, it is not clear how the sensitivity analysis is incorporated if at all in the modelling of the tether force model.

*******************
Response: Thank you.
> We have indeed revisited the flow of this section. While we start our analysis with a univariate feature selection approach, we examine the correlation (dependence) relations between the top features correlated with the tether force in Table 3.
> We want to confirm our intuitive understanding of measurement ranking in impacting the predicted tether force. In any ML project, this is an important step to know the important features. In case if we have large number of features, we could train the model based on the high ranked features only.
*******************

A lot of data is provided and many different off-the-shelf methods are tested. The drawn conclusion that the neural network is accurate is not obvious from the presented results.

*******************
Response: Thank you.
> Based on the quality metrics, the NN is the second best algorithm after voting regressors. However, as stated in the paragraph started with Line: 357, the accuracy of voting regression depends on the availability of training data, how powerful each predictor in the group is, as well as their independence. furthermore, we didn't consider hyper-parameter tuning of the NN model (as stated in the paragraph started with Line: 328), which could result in further optimized NN performance.
*******************

Overall, the experiment design is valuable and the idea of applying learning-based methods in order to model the tether force is interesting. Some refinement in how conclusions are made and a clarification of the novelty of the different aspects of the work would greatly improve the article.

*******************
Response: Thank you.
> We added a few clarifications to the introduction and conclusions sections. We emphasize that the novelty of our contribution is centered around the use of experimental data, rather than simulation models, to train ML models in order to predict tether force based on many input data features. While several model-based and simulated approaches have attempted the same prediction, we offer a new perspective for incorporating experimental design with ML modeling, which would open doors for new emerging ML techniques in AWE applications.
*******************

Minor comments:

-Define the variables an acronyms appearing in the labels of the figure axis. Some of the axis are too small to read.

*******************
Response: Thank you.
>
*******************

-Page 16, Line 271 'then then'
*******************
Response: Thank you.
> We corrected this typo.
*******************

-Brackets on Page 19, Line 356 not closed. Potentially an overuse of parenthesis throughout the article
*******************
Response: Thank you.
> We corrected this typo.
*******************