Review Reports

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors of this manuscript outline the motivation and overall design of VNA Tools, which is a software already in productive use for quite some time.

In general the paper is well written, the references are reasonable and the explanations are clear. However, the actual novelty of the paper - discussing traceability chains - is quite brief. Instead, the authors repeat at length what they've published before. More precisely, a more detailed explanation of how LinProp objects are combined when originating from different sources; e.g., based on the traceability chain models cited by the authors; together with an actual example would have been more helpful than an explanation of S-parameter measurements and basic measurement uncertainty models. 

Therefore, I'd like the authors to extend the section 6 in terms of more detailed explanations and a suitable example. Given their expertise and long track record in this field, I don't expect this to be a major revision for the authors.

Minor comments

• The abstract does not state what the novelty or core aspect of the paper is. What are the findings and conclusions? Right now, the abstracts appears more like a teaser for the software than the paper.
• Page 3, the authors write "traceability is propagated". This formulation appears quite unusual. Uncertainties are propagated as basis for traceability.
• Page 3, bottom: The authors outline the multivariate character of the measurements and the considerations of correlations between real and imaginary part. However, I assume that there are also correlations between values at different frequencies. 
• Page 5, in the description for Figure 2, the authors state that blue boxes in the figure represent operations. Using the same blue box for quantities and operations makes interpreting the flowchart confusing.
• Page 7, the authors state that uncertainty calculations and updating the LinProp objects happens automatically due the object-oriented programming. Instead, I assume that the objects' methods are written such that they implement the uncertainty propagation and updating of objects' attributes. "Happens automatically" sounds as if some magic happens in the background. 
• Page 7: The purpose of the virtualization capability remains unclear.

Author Response

Please see attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The paper describes what the software VNA Tools is, and how it supports the digital traceability chain. It is very well written and clearly structured. I have some minor remarks.

I think the largest part of the paper covers the ideas of making traceability and uncertainty calculations along the full traceability chain possible. The abstract is rather focussing on section 2: VNA measurements. It would be good to rewrite the abstract such that sections 3 to 6 are addressed more explicitly. I also suppose that the newness of the proposed work is rather in that aspect.

It would be good if the paper addresses the following questions, which might be relatively simple to answer.

1. How are type-A uncertainty contributions stored? As mean and standard deviation or are all underlying measurement results stored as well?

2. Can the software calculate expanded uncertainties and, if yes, how is this done?

3. Can the software handle degrees-of-freedom and compute associated k-factors?

In addition, I think if a simplified example would be added to the paper, this would make all ideas clearer for readers not very familiar with the subject.

Furthermore, I have the following remarks:

l. 69: 'inherently inaccurate due to unavoidable systematic errors.' -> I suggest to add: ', if these systematic errors are not corrected for.'

l. 129: 'here' -> 'Here'

l. 153: 'Any measurement model...' This sentence seems incorrect to me. One may write down integral equations, differential equations, optimization equations, etc. Maybe just replace 'Any measurement model' by 'Many measurement models'. Furthermore, being able to write down the measurement model does not necessarily mean that you know how to solve it (e.g. if it is posed as an optimization goal, y = arg min (... complex expression ...) ).

l. 163: 'steps<' -> 'steps'

l. 185: 'Because dependencies are tracked explicitly,' -> 'Because dependencies are tracked explicitly, and all input quantities at the highest level are assumed to be independent,' Otherwise you would need the covariance matrix from you highest/ lowest level input quantities. How do you know if all your input quantities are independent? If your measurement uses two temperature sensors, both calibrated at METAS, then they are probably not independent.

l. 196: 'externally' -> remove or 'on the PC' if that is what you mean. It sounds to me that you write 'external to the PC'.

l. 268: 'This mechanism ensures that the full uncertainty budget' -> I think a correct book keeping of all GUIDs is needed to assure that all correlations are really properly addressed. Is there a way of ensuring GUIDs are unique within METAS? If in two measurements the tabulated thermal expansion factor or electrical resistance of copper with associated uncertainty is used, will both measurements use the same GUID for this input, even if done with different set-ups and operators? If not, this might be discussed as future work in the discussion session. And in particular if VNA tools would be used by different NMIs. Each NMI would need unique GUIDs, but also common ones for the same tabulated values, as noted above. Do you have any ideas on this? At least it is no so easy as just using VNA tools. Some sentences should be devoted to this. 

l. 281: 'underestimation and overestimation' A numerical example would be very good as to show what the numerical impact in practice is of all the work that went into the software. If it takes too much space and effort to fully describe an example, then at least a sentence and/or table with what differences have been encountered in the calculated uncertainty in practice would be very interesting.

Section 6: Adding a figure that may look quite similar to figure 2 would make the ideas clearer. One can make visually clear how by making use of the same GUIDs, correlations between different measurements are properly taken into account.

l. 291: 'tens or even hundreds of megabytes.' This surprises me. Can you comment on this? How many and which numbers need to be stored?

l. 315: Delete the empty line.

l. 392: hyperlink is not rendering in blue font

 

 

Author Response

please see attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Respected Authors,

After careful reading of your paper, I have the following recommendations:

R1: The paper is completely theoretical, without an experiment, simulation, or hardware realization. Additionally, a significant portion of the theory you have presented is already well-known and has been previously presented in the literature. That is ok for the introductory part of the paper; however, it is not enough to be the complete paper.

R2: I kindly advise you to include an experimental setup description and the numerical results obtained as an example of the described procedures and software.

R3: The paper needs to have an original scientific contribution in relation to the existing literature.

R4: I strongly suggest that you prepare a separate section of the paper devoted to the conclusions based on the results obtained by conducting experiments (calibration procedures with uncertainty calculations). Or, you can combine the Discussion section with the Conclusion section. 

R5: I kindly suggest that you condense the theoretical part of the paper and include the experimental setup description and numerical results, along with corresponding discussions. Also, some plans for future work can be included within the concluding section.

Kindest regards.

 

Author Response

please see attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

Respected Authors,

Thank you for addressing my remarks. 

Kindest regards.