Contracts for Difference: A Reinforcement Learning Approach

Round 1

Reviewer 1 Report

Zengeler and Handmann proposed a reinforcement learning approach with LSTM and feedforward architectures for the trading of CfD. The topic is interesting and the approach looks novel.

My comments on the paper are as the following:

1) The background of the research should be improved to give readers much information. In addition, what is the motivation if the paper?
2) Basically, the main work of this paper is the application of reinforcement learning to automatic trading of CfD, and hence the scientific contributions of the paper are not clear. Specifically, I suggest the authors highlight the contributions in theory and/or in applications.
3) A short paragraph on State of the Art (L47-L55) and only 11 references shows that the research field is not a hot topic.
4) In evaluation, the authors did not compare the presented approach with some state-of-the-art methods (for example, the methods in L47-L55), making the evaluation not convincing. In addition, the evaluation data should be described in detail so that the readers can reproduce the evaluation.
5) As for the proposed LSTM model, how did the authors use it as a reinforcement learning approach? If the authors can describe it using a formal algorithm, I think it can improve the paper.
6) I think that running the proposed approach in the real-world application for 10 trading days is not sufficient to demonstrate the effectiveness of the proposed approach.

Author Response

Dear reviewer,

thank you for your valuable feedback.
We have revised our manuscript according to your comments.

1) The background of the research should be improved to give readers much information. In addition, what is the motivation if the paper?

We have rewritten both the abstract and the introduction, highlighting that the motivation of this paper lies in academic analysis of high frequency trading methods, which we think receive too little attention.

2) Basically, the main work of this paper is the application of reinforcement learning to automatic trading of CfD, and hence the scientific contributions of the paper are not clear. Specifically, I suggest the authors highlight the contributions in theory and/or in applications.

In the abstract we specified that our contribution lies in both proving the effectiveness of reinforcement learning with LSTM in this domain and, with our real world example, showing that increasing model sizes may compensate for latency.

3) A short paragraph on State of the Art (L47-L55) and only 11 references shows that the research field is not a hot topic.

We have deepened our research on state-of-the-art and added further sources.

4) In evaluation, the authors did not compare the presented approach with some state-of-the-art methods (for example, the methods in L47-L55), making the evaluation not convincing. In addition, the evaluation data should be described in detail so that the readers can reproduce the evaluation.

Our simulated market environment does not allow testing models with other specifications. Therefore, we have added a paragraph in the future work section, in which we discuss the necessary steps for such a baseline comparison research.

5) As for the proposed LSTM model, how did the authors use it as a reinforcement learning approach? If the authors can describe it using a formal algorithm, I think it can improve the paper.

In the LSTM model description section, have added a reference to "Learning to navigate in complex environments", which basically describes the way we use LSTM models for reinforcement learning. We have also added explanatory sentences on how to integrate the architecture into the learning problem.

6) I think that running the proposed approach in the real-world application for 10 trading days is not sufficient to demonstrate the effectiveness of the proposed approach.

We were limited in our testing time as the demo accounts provided by XOpenHub have a limited validity. We added detail to the explanation of this insufficiency in our discussion section.

Yours truly,
Nico Zengeler

Reviewer 2 Report

The authors should provide a more detailed description of similar works.
The authors should describe with pseudocode the whole algorithm.

The authors should better describe the dataset used for the real world application.
A statistical test should be used for the comparison of the examined methods.
The authors should better explain why the proposed methodology seems to work well and present some information about the time efficiency of the presented method.

Author Response

Dear reviewer,

thank you for your valuable feedback.
We have revised our manuscript according to your comments.

1) The authors should provide a more detailed description of similar works.

We have deepened our research on state-of-the-art and added further sources.

2) The authors should describe with pseudocode the whole algorithm.

We have rewritten the pseudo code in the appendix, now explaining the whole algorithm in greater detail. We have also provided a link to out GitLab repository, thus allowing everybody to check the concrete algorithm and reproduce our experiments.

3) The authors should better describe the dataset used for the real world application.

In the evaluation section we have explained the dataset in greater detail.

4) A statistical test should be used for the comparison of the examined methods.

We have performed a grid test over all models for 1.000 trades each, as explained in the evaluation section, but only presented the best models. We could add all our results in the appendix but felt this would not benefit the comparison of our methods. For a statistical test, we did not find a formulation of a null hypothesis which leads to a higher clarity than the grid tests we have performed.

5) The authors should better explain why the proposed methodology seems to work well and present some information about the time efficiency of the presented method.

We have added a sentence about the training time efficiency in the evaluation section.

Yours truly,
Nico Zengeler

Round 2

Reviewer 1 Report

The authors answered most of my questions quite well and the paper has been improved. Although there still exist some limitations in the study, the authors have explained them carefully. I think that the current version can be accepted for publication in JRFM.

Reviewer 2 Report

The paper could be accepted in the current form