Computer-Aided Formalization of Internal Consistent Product Family Models
Round 1
Reviewer 1 Report
The paper addresses the problem of the detection and identification of inconsistencies when feature specifications are present in feature modeling. In order to solve this problem, the authors propose a method using template-based feature modeling and rule-based consistency check approaches. A tool to automate this process is shown.
I would encourage the authors to take into account the following comments to improve the article in a more rigorous and complete way.
General comments
The paper is well structured placing the audience in the context of the problem to research, however the title is too general for the specific content that is addressed in this paper. It can create too much expectation to interested readers, therefore it would be required an adequate title that could reflect a more real content of this article.
Also, I consider that it is important to highlight that this approach not only detects inconsistencies (Usually, “detect” is understood only as if there are or are not inconsistencies in a feature model but without indicating where), but the proposal also identifies them, because it is able to determine where the inconsistencies are present. Moreover, it seems that the tool also offers recommendations to solve inconsistencies. For this reason, it is necessary one more detailed technical explanation about how all these capabilities is achieved. It could add more value to the article.
Finally, I would recommend showing a practical and clear motivation for the interested parties in order to adopt this proposal and the assessment of the advantages and disadvantages that it would bring them. It must include a more rigorous and significant experimentation with feature models of a complexity closer real cases and not only toy cases.
Specific comments
Comments related to the feature specifications scope of the article are as follows:
The feature specifications could use variables of different types (Boolean, integer, float, double, string,…) and different kinds of expressions or constraints among these variables (i.e. reified constraints). In order to concrete the feature specifications scope of this article it is necessary to clarify the readers the variables types and expressions that are permitted in these specifications, because the lack of it could generate false expectations respect to the possibilities of the approach and the tool.
Comments related to the algorithms to automate the inconsistencies detection approach
The approach seems to be too dependent on the tool that will later be used for reasoning. Since the feature specifications could have a lot of variability and there are a lot of solvers (CSP, SAT, Binary Decision Diagrams, Stochastic), the algorithms to automate the search of possible inconsistencies must be adaptative or interactive to Software Product Line (SPL) researchers and practitioners.
Comments related to the Performance of the tool
With the explosive growth of features, inconsistent feature specifications are increasingly common. Therefore, it is necessary to consider the computational efficiency to search inconsistencies. The problem is NP-hard so it is necessary develop an adequate algorithm to solve this problem or to determine the threats of validity of the tool. More evaluation results of the tool could show that the approach is effective and accurate for the feature models scalable up to thousands of features and thus, is an important contribution to Software Product Line Engineering (SPLE).
Comments related to the lack of a significant and rigorous experimentation:
The experimentation must determine if all the inconsistencies in the experimental feature models (completeness) are found and also if all the found inconsistencies are valid (soundness), but it is not presented. Moreover, I consider that the experimentation about the evolutionary process is not relevant according to the goals of the article. For these reasons, I consider that a more significant and rigorous experimentation is necessary.
Comments related to the lack of a threat of validity and future works
The assessment of threats to validity is critical to ensure the quality of a research. However, the authors have presented some informal descriptions of threats of validity in the section conclusions, but according to Wohlin et al., four aspects of validity could be taken into account: Construct validity, Internal Validity, External validity and Conclusion validity. ( C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell and A. Wesslén, Experimentation in software engineering. Springer, 2012.)
Also, I consider that it is very important to know the future works of this research, as they are missing in this article.
Writing errors
In the text, it appears: “the product line project should be adapted to different types of reservoirs and should Finally”??? Thus, the sentence must be completed
In the text, it appears different acronyms without previously indicate their meanings (i.e. ABS, PFA, SOFL and RMS). This could makes the paper less readable and more difficult to understand.
Finally, in the text it appears “provide ablation methods”. The word ablation is typically understood as "surgical removal", which has nothing to do with the context of this article. My recommendation is to change this word.
Author Response
Dear Reviewer:
We are very grateful to Reviewer for reviewing the paper so carefully. We have carefully considered the suggestion of Reviewer and make some changes. Please see the attachment.
Kind regards,
Hongbo Liu
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The manuscript proposes a framework to model software in a product family and specify major features within the software. Based on the specification, three consistency checks, including variable consistency, functional scenario consistency, and functional path consistency, can be performed. A case study and supporting tool are also introduced to evaluate the proposed method.
Overall, the method is well described in the manuscript. However, it is suggested that the motivation, challenges and the benefits can be introduced in more detail. Some major suggestions are listed in the following for the authors to improve the manuscript.
1. It is suggested that the authors can elaborate on the reason for the integration between feature model and feature specifications. In other words, what are the challenges faced by software engineers when they develop a software system in a product family without the proposed method?
2. The benefit brought by the proposed method can be elaborated. In other words, "when to use the proposed method", "how to use the proposed method", and "the outcome and advantage after using the proposed method" can be introduced for readers. Thus, readers can get more insights into the contribution and how to use the proposed method in the software development life cycle.
3. In the manuscript, SOFL is used to describe the specific formal specification. The reason for choosing SOFL can be introduced.
4. In the case study, it is suggested that three consistency checks can be presented. In addition, when features are evolved, is it possible the detect the fault caused by new features (e.g., new features violate the rule in the previous evolution)?
Minor suggestions and comments are listed in the following.
1. It is suggested that the authors can proofread the manuscript again. There exists some typos in the manuscript.
2. Please check the reference again. Reference [37], [38], and [39] are missing.
3. Please give the full name when the abbreviation is used in the first time.
4. Please check captions of the tables. For instance, captions of Table 3 and Table 4 are the same ("Variable Detection"). Please refine captions for better readability.
5. Please check line 206 and the following sentences. Is it a "phone" product line?
6. Why RSA is not shown in the left sub-graph of Figure 4?
7. Additional examples are given in Table 5 and Table 6. However, readers cannot get more insights into the benefit brought by the proposed method. How many inconsistencies are found?
Author Response
Dear Reviewer:
We are very grateful to Reviewer for reviewing the paper so carefully. We have carefully considered the suggestion of Reviewer and make some changes. Please see the attachment.
Kind regards,
Hongbo Liu
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Thank you to the authors for taking into account some of my previous comments to improve the article in a more rigorous and complete way.
For the new version my comments are the following:
I think that to change the title into “Computer-aided Formalization of Internally Consistent Product Family Models” is a more adequate title.
Regarding to my previous comment about to show “a practical and clear motivation for the interested parties in order to adopt this proposal and the assessment of the advantages and disadvantages that it would bring them“ I consider that the modifications made by the authors do not really reflect what I proposed in my comment.
Also, in my previous comment about “It must include a more rigorous and significant experimentation with feature models of a complexity closer real cases and not only toy cases”, the new version has not considered it despite its importance to check the computational validity of the proposal.
Finally, regarding my previous comments about the feature specifications scope of the article, my new comments are as follows:
In the new version of the article, the following sentence is claimed: “Functional scenario detection rules currently support variable types of numeric classes” . In this context, I understand that the word numeric is synonymous with the word number and it includes decimal numbers.
In order to clarify the feature specifications scope about the possible expressions permitted in the proposal and the possibility to determine their inconsistencies, some doubts may arise to the reader:
· Could the expression “Var1>= 1/3” and the expression “Var1>= 0.33” be considered equivalent expressions?? Is the consistency principle of the formal description satisfied or not in this case according to what is indicated on line 383?? ( about “If a variable v belonging to the set of variables V is described as different in the formal definitions of process A and process B (A≠B), the consistency principle of the formal description is violated,…”)
· In the product line system ATM, the feature specification for “Withdraw” could be: the input is amount, precondition is amount > 0 and postcondition is balance = balance_ pre - amount , but in the high-level feature “Account” its formal description is balance >= 0. But if the balance_pre is less than the amount, then the previous formal description is inconsistent. Therefore the question is: could the tool detect this inconsistency for this part of the feature model???
· In the product line system ATM, the feature specification for “Withdraw” could be revised with the following new feature specification: the input is amount, precondition is amount > 0 and balance_pre >= amount and postcondition is balance = balance_ pre – amount. In this case the specification confirms that the user’s account holds a sufficient balance for each individual withdraw. In this case, the user could then submit several withdraws concurrently and potentially exceed the balance of their account. Given the significant economic effects it can have on product line system ATM, could also these inconsistencies be detected by the tool???
Writing errors
On line 74 of the new content, it appears “This is precise because the analysis of product family feature models is not precise enough,” How is “precise” and “not precise” at the same time???
On line 462 the expression “the arthmetic expressions” is wrong.
On line 459 it appears the word “datas”, but data is the plural of datum and that is how it is written in specialized scientific papers, it is also treated as a plural in English.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 2 Report
Major issues are carefully addressed in this revision. The effort of authors is appreciated. Because several modifications are made in the manuscript, it is suggested that the authors can proofread the manuscript again before publication.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
