Statically Analyzing the Energy Efficiency of Software Product Lines^†

Round 1

Reviewer 1 Report

This paper presents an approach to estimating the energy consumption in all the products in a software product line without measuring the energy consumption of each individual product, which is cost prohibitive due to combinatorial explosion of choices/feature combinations.  The proposed technique combine static analysis, worst-case prediction, and feature-oriented analysis (instead of product oriented analysis).

The paper is written well. The approach is sound, and the experiments are sufficient, and shows promising results.

However, this reviewer has one major comment. In the software product lines research, there are machine learning based approaches that support predicting the quality attributes of the all products or configurations, by only measuring one or few samples (i.e., a systematically selected subset of products). How do you compare your approach with such methods? What are the differences/similarities/strengths/weakness?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

I have reviewed the article "Statically analyzing the Energy Efficiency of SPLs", Manuscript ID: jlpea-1113213, that has been submitted for publication in the MDPI Journal of Low Power Electronics and Applications and I have identified a series of aspects that in my opinion must be addressed in order to bring a benefit to the manuscript.

In this paper, the authors approach issues regarding the energy efficiency of Software Product Lines (SPLs), an important software engineering discipline allowing the development of software that shares a common set of features.

I consider that the article will benefit if the authors make remarks within the manuscript regarding the following aspects:

Remark 1. The Manuscript ID: jlpea-1113213 is interesting, but it still remains an aspect that must be clarified by the authors, namely the fact that they must assume more clearly their own results.

Firstly, as the main result of the manuscript is related to software, the authors should provide precise details within the manuscript regarding how their approach relates to the most important software quality characteristics that software must satisfy to a certain degree in order to be considered a quality approach (according to the ISO/IEC 25010 Standard). The authors should also provide details within the manuscript regarding to what degree their approach considers the most important software quality characteristics when the software application is put into use (according to the ISO/IEC 25010 Standard).

Secondly, I consider that the authors must assume more clearly in the paper their original contribution by specifying this fact and by highlighting the fact that starting from a certain point there are presented the original and novel aspects of their paper. The authors must state more clearly their original methods, their original results and conclusions, the novelty of their study. In the current form of the paper, this aspect is unclear. Another weak point consists in the structure of the manuscript. If the authors make an effort to improve the structure of the manuscript and complement it with the recommendations from the specific comments, the authors will arrive at an article that can bring a valuable insight to the current state of knowledge.

Remark 2 - The title of the manuscript, "Statically analyzing the Energy Efficiency of SPLs". Acronyms and/or abbreviations must be avoided in the title, even if they are widely known by the experts in the field. Therefore, I consider that the authors should refer to "SPLs" as "Software Product Lines (SPLs)", therefore modifying the title accordingly. Regarding the other acronyms used in the manuscript, they should be explained the first time when they are introduced.

Remark 3 – Lines 1-16, the "Abstract" of the manuscript. In the "Abstract" of the paper, along with the elements that have already been presented, the authors should briefly state the novelty of their study.

Remark 4 – Lines 45-56, the "Introduction" section. In the "Introduction" section, the authors have presented the literature review of the cited papers in the following manner: "In practice, SPLs are widely used to create operational flight programs [5], software to control gasoline system engines [6], medical systems for image-supported diagnosis [7], or even television sets’ operating systems [8], among several other examples. Although several techniques have been proposed to measure, reason about and improve the energy efficiency within software and language engineering [9–13], only recently there has been an effort towards addressing this issue in SPLs [14,15]. In a nutshell, a SPL development approach was presented in [15], mainly focused on building a DSPL that can adapt itself with energy-saving capacities, through establishing a relation between variability features and their energy impact and continuous monitoring the context on which the SPL is used. Regarding energy analysis of all products within a SPL, the first emerging work [14] attempted to explore static analysis to reason about such products, in order to obtain for each one accurate energy consumption estimations." I consider that it is not appropriate for the manuscript to cover 11 (like the authors have did) or even more scientific works in a few lines just for the sake of obtaining an appropriate size of the References section. I consider that the literature review should be improved by performing a careful analysis of the cited works. The authors must highlight exactly, for each of the involved referenced papers the main contribution that the authors of the referenced papers have brought to the current state of knowledge, the method used by the authors of the referenced papers, a brief presentation of the main obtained results and some limitations of the referenced article. This is the only way to contextualize the current state of the art in which the authors of the manuscript position their paper, identify and address aspects that have not been tackled/solved yet by the existing studies.

Remark 5 - the "Introduction" section. After having performed a critical survey of what has been done up to this point in the scientific literature, the authors must identify and state more clearly in the paper a gap in the current state of knowledge that needs to be filled, a gap that is being addressed by their manuscript. This gap must also be used afterwards by the authors, when they discuss the obtained results as well, when the authors should justify why their approach fills the identified gap in rapport with previous studies from the scientific literature. It will benefit the paper if in the final part of the "Introduction" section, the authors present the main contributions of their paper, eventually synthetized within a bulleted list.

Remark 6 - the "Materials and Methods" section. First of all, according to the "Instructions for Authors" from the Journal of Low Power Electronics and Applications website (https://www.mdpi.com/journal/jlpea/instructions), each manuscript must contain a "Materials and Methods" section ("JLPEA now accepts free format submission: We do not have strict formatting requirements, but all manuscripts must contain the required sections: Author Information, Abstract, Keywords, Introduction, Materials & Methods, Results, Conclusions, Figures and Tables with Captions, Funding Information, Author Contributions, Conflict of Interest and other Ethics Statements."). Therefore, I consider that the authors must devise a proper "Materials and Methods" section. In order to bring a benefit to the manuscript, the authors should mention early in the "Materials and Methods" section, preferably in the first sentence, the choices they have made in their study. The authors should state what has justified using the given method, what is special, unexpected, or different in their approach. If the authors make use of a standard or usual procedure, this aspect should also be mentioned upfront, from the very beginning. I consider that the manuscript under review will benefit if the authors make all of these aspects as clear as possible to the readers starting from the first sentence of the paragraph in order to give them a clear idea of how their approach was devised and obtained.

Remark 7 - the "Materials and Methods" section. I consider that in addition to the actual explanations, in order to help the readers better understand the methodology of the conducted research, the authors should devise a flowchart within the "Materials and Methods" section, a flowchart that depicts the steps that the authors have processed in developing their research and most important of all, the final target. This flowchart will facilitate the understanding of the proposed approach and at the same time will make the article more interesting to the readers if used as a graphical abstract.

Remark 8 - the equations within the manuscript should be explained, demonstrated or cited, as there are some equations that have not been introduced in the literature for the first time by the authors and that are not cited.

Remark 9 - the "Results" section. This mandatory section is missing, being partially replaced by other ones. However, a proper "Results" section should be presented in the paper. In this section the authors should provide a concise and precise description of their obtained results, their interpretation as well as the conclusions that can be drawn.

Remark 10 - the "Results" section. In order to validate the usefulness of their research, in the extremely important "Results" section, the authors should make a comparison between their study from the manuscript and other ones that have been developed and used in the literature for the same or similar purposes. There are a lot of valuable studies in the scientific literature related to the subject of the manuscript to which the authors can compare to and this comparison will highlight even more the insights that their paper has brought in contrast to the existing studies. This comparison is mandatory in order to highlight the clear contribution to the current state of knowledge that the authors have brought.

Remark 11 – the "Results" section. The authors should present the findings and their main implications in the "Results" section, also highlighting current limitations of their study, and briefly mention some precise directions that they intend to follow in their future research work.

Remark 12 – the "Results" section. Can the authors mention to which extent the methodology used within the developed research can be easily applied to other situations when the SPLs are different? In this way, the authors could highlight more the generalization capability of their approach in order to be able to justify a wider contribution that has been brought to the current state of art.

Remark 13 - the "Results" section. The paper will benefit if the authors make a step further, beyond their analysis and provide an insight at the end of the "Results" section regarding what they consider to be, based on the obtained results, the most important, appropriate and concrete actions that the decisional factors and all the involved parties should take in order to benefit from the results of the research conducted within the manuscript.

Remark 14 – Lines 990-1031, the section "8. Related Work". This section is positioned by the authors as the penultimate section of the manuscript, before the "Conclusions" section. It will benefit the manuscript if the authors reposition this section after the "Introduction" one, and afterwards, in the "Results" section, or in a separate "Discussion" one, the authors compare their obtained approach and results to the ones from the cited papers (as I have previously mentioned).

Remark 15 - the "Conclusions" section. It will benefit the manuscript if the authors devise a proper "Conclusions" section in which they state the most important outcome of their work. The authors should avoid simply summarizing the aspects that they have already stated in the body of the manuscript. Instead, they should interpret their findings at a higher level of abstraction than in the "Results" section. The authors should highlight whether, or to what extent they have managed to address the target proposed within the "Introduction" section. The authors should avoid restating everything they did once again, but instead they should emphasize what their findings actually mean to the readers, therefore making the "Conclusions" section interesting and memorable to them.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf