Mind Your Outcomes: The ΔQSD Paradigm for Quality-Centric Systems Development and Its Application to a Blockchain Case Study^†

Round 1

Reviewer 1 Report

The paper presents a system development approach for a distributed and real-time software system. It adopts the concept of quality attenuation, a widely accepted concept in network performance measurement. For better and timely system design decision, authors propose an approach to analyze system performance in terms of timeliness and resource usage via decomposing the system using outcome diagrams. As a running example, the Cardano blockchain development case is used.

 

However, the evaluation results on the proposed method are not presented. The running example in Section 4 helps to understand the proposed method as a tutorial, but the effectiveness/usefulness and efficiency/easy-of-use of the proposed method are not addressed.

• Present a concrete case of design decision on Cardano with the analysis results
• Present a feedback and perceived satisfaction evaluation from Cardano design engineers who use the method.

Some other minor updates are as below:

• Introduction Section needs to be more concise.
• Related work Section needs to include recent studies on block propagation analysis in blockchain. And I think that the contents in Section 3.5.1 to 3.5.3 need to be placed in Related work Section.
• Figures in page 17 and 19 are missing image caption.

Author Response

The paper presents a system development approach for a distributed and real-time software system. It adopts the concept of quality attenuation, a widely accepted concept in network performance measurement. For better and timely system design decision, authors propose an approach to analyze system performance in terms of timeliness and resource usage via decomposing the system using outcome diagrams. As a running example, the Cardano blockchain development case is used.

However, the evaluation results on the proposed method are not presented. The running example in Section 4 helps to understand the proposed method as a tutorial, but the effectiveness/usefulness and efficiency/easy-of-use of the proposed method are not addressed.

Present a concrete case of design decision on Cardano with the analysis results

The summary at the end of section 4 has been amended to make it clear that specific design decisions were made on this basis.

Present a feedback and perceived satisfaction evaluation from Cardano design engineers who use the method.

The Cardano engineers who used the method are authors of the paper and have found it very useful. 

Some other minor updates are as below:

Introduction Section needs to be more concise.

The Introduction section has been edited and broken into subsections to improve clarity

Related work Section needs to include recent studies on block propagation analysis in blockchain. 

A new section has been added referencing recent studies.

And I think that the contents in Section 3.5.1 to 3.5.3 need to be placed in Related work Section.

The authors prefer to leave section 3.5 where it is  because that discussion does not compare "the paper" with the rest of the world, it, instead, compares outcome diagrams exclusively against some of their competitors. §6 is, however, the paper's related work and, thus, an inappropriate place for this discussion.

Figures in page 17 and 19 are missing image caption.

Captions have been added.

Reviewer 2 Report

Motivation of this study should be included and well supported by references in this regard.

Limitations of the study should be included.

Threats to validity should be provided.

All symbolic notations and formulas should be carefully cross checked. 

Practical implications of study to researchers and professionals should be included in simplified manner in last of Discussion section. 

 

Author Response

Comments and Suggestions for Authors

Motivation of this study should be included and well supported by references in this regard.

A new section on motivation has been separated from the main introduction.

Limitations of the study should be included.

A new section on ‘limitations’ has been added to section 7.

Threats to validity should be provided.

This is included in the 'limitations'

All symbolic notations and formulas should be carefully cross checked. 

They have been checked, and the symbol for convolution changed to be more standard

Practical implications of study to researchers and professionals should be included in simplified manner in last of Discussion section. 

A new section on practical implications has been added at the end.

Round 2

Reviewer 1 Report

Evaluation on effectiveness and/or efficiency of the proposed method is essential to the scientific research article.

This paper does not provide evaluation and discussion part. 

I think that section 4 shows how this method is developed and used rather than how the novel paradigm is effective and efficient in making design decisions.

 

 

Author Response

We have added an additional analysis to the end of section 4 comparing the example with an alternative approach using simulation, and added a remark to the section on 'future work' that more comprehensive comparisons should be undertaken once tool support is available (clearly it is unreasonable to attempt a meaningful comparison between a new method that still requires manual manipulation with established approaches that are well supported by tools).

Round 3

Reviewer 1 Report

Please refer to the TAM(Technology Acceptance Model) to evaluate the method by quantifying user's feedback in terms of  perceived usefulness and perceived ease-of-use for new technology.

Author Response

We thank the reviewer for introducing us to the TAM, as this will provide useful referents to guide the development of tool/educational support for the ∆QSD paradigm. However, the authors strongly believe that the TAM is inappropriate to the current state of development of ∆QSD. TAM involves assessment of perceived usefulness and perceived ease of use by potential adopters of an IT system. As ‘usefulness’ and ‘ease of use’ are precisely the things that the paper is attempting to outline in relation to ∆QSD, there is as yet no possibility of assessing the perception of these things until at least the paper has been published and there is some reaction to it. Currently, ∆QSD has only been used by its inventors; so there is no community of other users to whom it has been introduced whose perceptions could be evaluated. The paper has been modified to clarify this point.