Benchmarking GHG Emissions Forecasting Models for Global Climate Policy

Round 1

Reviewer 1 Report

The paper suggests a univariate approach to produce accurate forecasts for the GHG emissions based on time series analysis by using data from 1970 to 2018 provided by the WDI database.

Obviously, the paper shows the big effort made by the authors in this work. However, the main issue in my mind is the suitability of this study with the journal subject and areas.

Also, I believe that minor points should be considered while submitting the revised paper:

• The paper is well written, and the “materials and methods” section is well exposed and appropriately detailed. But the introduction is somewhat repetitive and need to be more concise.
• Authors used many universes predictive models (econometric and machine learning) using R software packages. They affirm that the NNAR model (Neural Network Autoregressive model) outperform the other models. I just wonder what is the “nnetar” function used in the NNAR and how it used to automatically fit the multilayer feed forward neural network?
• Authors employed seven predictive models. Also, they work with relevant metrics for air pollution (GHG emissions instead of only CO2 emission like the case of other studies in literature). But they should also compare the results of their proposed approach to existing ones.
• What are the perspectives of this work?
• References format should be checked .

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Reviewer’s comments to the paper:

Forecasting GHG Emissions in top polluting countries through autoregressive neural network models

The paper is dealing with the use of nonlinear NNAR predictive models that are compared with five other types of forecasting models for the purpose of predictions of future trends of GHG emissions in 12 different countries. Here, 44 years long time horizon is used as a training dataset (from year 1970-2013), while 5 years long time horizon is used as a testing dataset.  The study sounds interesting and is likely important for the researchers in the field.

In general, the paper more or less satisfies rigor requirements that are demanded by the journal. The red clue suffers an occasional inconsistency throughout the paper; some figures and tables are of unsatisfactory quality and/or explanation power; the main contributions and findings might have been more clearly emphasized and more in-depth, analyzed, and justified. Moreover, in general, the paper is at several places, more or less sufficiently organized and written. To summarize, it is clearly seen that the authors have invested significant efforts in the research. On the other hand, the way of reporting and presentation of the study and its results in this paper has not been carried out in such an adequate way as the research itself. Thus, while the research efforts seem to be noticeable, the quality of the reporting presented in this paper is, unfortunately, not at the same level.

The reviewer has detected some issues that are recommended to be corrected prior to continuing with a further reviewing and editorial process. Some major comments are:

1. The introduction section should be rewritten. At its present form, it is too condensed, at some places confused, some text is redundant, etc. It should also be more structurally organized, perhaps with including of sub-sections.
2. I recommend that the literature review is removed from the introduction section to the separated individual section titled “Literature review”. The latter should also be structurally organized, perhaps with including of sub-sections. Here, please systematically summarize the state of the art in the field. Also, it must be more clearly emphasized what the main contribution of the paper is, i.e., what has been done new, and what are the main differences, if the novelties in this paper are compared with the latest state-of-the-art. So, to summarize, please, clearly point to the research gap, that has been targeted by this paper, and mention where, how, and to what extent similar studies have been conducted, with the precise border between this study, and the other similar studies reflecting the newest state of art.
3. The lack of mathematical rigornouness is detected in the paper. Please include the table of notations, symbols of all variables, and abbreviations. Here, all the variables must be precisely and clearly defined and explained. For example, we have 12 real time series data for 12 countries, thus, yi(t), i=1…12, t=1,…,49. On the other side, we have six models j={M1,M2,…,M6}. Moreover, we have 6 X 12 = 72 predictive variables representing the forecasted estimated time series based outputs of these models, i.e., yijm(t), i=1…12, t=1,…,49, j={M1,M2,…,M6}, m-model. We have 72 models’ errors variables eijm(t) = yi(t) - yijm(t), i=1…12, t=1,…,49, j={M1,M2,…,M6}. Please, include somehow such kind of mathematical rigornouness in all elements of the paper (figures, tables, text, etc.).
4. Please, add a new section titled “The conceptual framework”. In this section, please give some block diagram or flow chart of the consecutive steps that have been carried out in this research. It must be clearly seen what are the inputs (variables yi(t), i=1…12, t=1,…,49) to the model blocks (six blocks), for which the outputs are yijm(t), i=1…12, t=1,…,49, j={M1,M2,…,M6}, m-model. Furthermore, please schematically introduce the models’ errors variables eijm(t) = yi(t) - yijm(t), i=1…12, t=1,…,49, j={M1,M2,…,M6}. These error variables should be further directed to the blocks representing the Goodness-of-fit measures (GOF): GOF(eijm(t),s), s=1,…,k, where k is the number of these measures (e.g., RMSE, RRMSE, etc.).
5. While testing the predictive out-of-sample performance of the NNAR and other, comparative (competitive) benchmarking models, the various newest state of the art residual-based criteria should have been also investigated within the scope of previously mentioned GOF measures (besides RMSE, RRMSE), such as: Hyndman’s MASE, sMAPE, MBRAE, UMBRAE, MAAPE, GMRAE, etc. (please check the literature). While doing benchmarking, many authors also suggest the Diebold-Mariano test, White's Reality Check and Hansen's Superior Predictive Ability (SPA) Test.
6. Please, improve the explanatory power and aesthetical look of all figures and tables. All the figures (and tables) must be presented in a such way that the reader immediately understands the main points without even looking at the corresponding text.
7. Mathtype is recommended to be used for all symbols of variables in the text and equations as well. Now, in the current form, the aesthetical look of the latter is not appropriate.
8. The key findings of the research should be more in-depth discussed.
9. I am missing time-graphs representing the fit of the primary model’s output (NNAR) to the real data of time series of all 12 countries. In the paper, table 6 is included (point forecasts for different countries), it is true. However, the graphical representation showing the NNAR model’s fit to the real data would be perhaps much more ilustrative than only some numerical values (point forecasts) given in table 6. Namely, then it could be clearly seen how accurately the model’s dynamics follows the real data dynamics. More importantly, in this case, it could be also identified at which time segments the NNAR model (despite its nonlinear nature) fails to precisely follow the complex real data dynamics behavior (due to real data highly nonlinear characteristics).  

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Paper is good contribution, please correct the following issues:

Title is not good enough, this would work better: "Benchmarking GHG emissions forecasting models for global climate policy". Benchmarking methods is key in the title.

Please remove words like horizon and universe if not literal use.

Define RMSE in the abstract

Substitute "very high in absolute terms" for dieciles (i.e. d1-d10) as quartiles

It would be desirable to put the neat global increase or decrease as your probably can compute it easily

Define NDC and which is the target of Paris agreement as the abstract is very climate policy-oriented.

Avoid any commentary on Paris 21 that is not index definition, numbers or specified targets, PLEASE revise this in depth (i.e. US and China participation out).

Results in the intro sound like an assessment of Paris 21 targets, do they appear numeric somewhere? please put numbers if it is an assessment, otherwise leave it for conclusion/outlook. Revisit the paper and put assessment of Paris 21 in the abstract if the paper presents an assessment. I will suggest this paper to be transferred to Sustainability as it is policy-oriented.

Figures should be improved significantly if they reflect economic trends as presented in the introduction. Please put bars and marks on historical events and reflect proper sources or references (including Wikipedia articles).

Again, you mention Paris 21 target as 1.5º increase, is there any corresponding rank of GHG to this target? any literature on this? this is significant for the assessment. If not, all Paris 21 should be put in an extended conclusion. The reader expects and evaluation of Paris 21 in the intro.

The intro must be shorten... please focus on models and ML revision and what is taken from climate policy for the results/benchmarking/assessment.

All the text regarding economic interlinks with GHG or climate should be put in a specific section and not the intro as you use economic models. Please restructure the paper accordingly.

Improve figure 3, for instance you can put a rescaled version by the side (0-1 in the y-axis). Show also relative increase (%) from reference point chosen as 1970 in the time series.

"the Paris Agreement requires yearly cuts of 8%", please move this to the intro. Is it equal for all countries?

Section name change: Sample -> Database

Section 2 contains Discussion on policy based on the data, restructure the whole paper to put discussion in a separate section and shorten all sections.

Section 3 is not cross-validation, it is benchmarking, please change throughout.

Benchmarking is properly done, results presented on tables are appropriate.

Please include a Discussion section as mentioned with results, policy evaluation and Paris 21-related comments.

Conclusions should be divided on benchmarking and climate policy subsections.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

After extensive corrections, the paper is now much better and more transparent. All comments have been considered. Accordingly, I recommend the acceptance of the paper.

Reviewer 3 Report

The paper has been significantly improved after the revision as well as the figures. I consider that everything is fine for publication as it is quite interesting work. 

I can recommend it for MDPI Sustainability too.