A Truly Robust Signal Temporal Logic: Monitoring Safety Properties of Interacting Cyber-Physical Systems under Uncertain Observation

Round 1

Reviewer 1 Report

While the paper is easy to read (albeit some English sentences are a bit wordy), I also see some problems:

-  Technically, the superiority of this approach (in terms of conclusive-ness) crucially depends on what we know about sensor offset and random noise. For example, in the paper "0.5 thereof can be attributed to an unknown constant sensor offset, leaving another 0.5 to random measurement noise". How do we know these numbers? If these numbers are guessed,
is it still valid to say the approach is sound?

- This journal is "Algorithms", but the paper has no particular algorithmic content, just an approach based on
a fairly well-known and straightforward encoding into SMT formulas.

- The approach is not justified by enough practical examples.

- Do you want to do SMT-solving in real-time (as the system runs / evolves)?  Or this is about an offline-monitoring approach? Would the encoding still manageable by solvers if duration is much larger (say 200)?

I regret to say that the article must be significantly improved to include one (maybe two) proper implementation and
experiments section(s) to be accepted.

DETAILED COMMENTS
=================

- l. 4: specification -> specifications

- l. 11--13: This statement should be more clear. "evaluated pointwise" -> pointwise"ly"?
What do you mean by "genuinely" more informative? etc.

- l. 25: I suggest adding a reference about applying safety monitoring in DevOps.

- l. 87: The definition of G_{\leq t} \phi only holds in discrete-time semantics - probably mention this.

- l. 97: Please elaborate on continuous interpolation, and why / how it is "at the same price".

- l. 122--124: The statement is absolutely correct, but it will be better if you can add a reference.

- l. 128--129: missing parenthesis?

- l. 135: missing subscript {S} in (1)

Some definitions are also missing, such as the one for "time series".

- l. 216--218: This is a rather bold claim that I will rephrase.

- l. 244: I think this interval (or set) is incorrect.

- l. 304: even most "concise"? you mean "precise"?

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Summary

The authors present a monitoring procedure for (bounded) Signal Temporal Logic (STL), where the signals are a allowed to be imperfect, i.e., the signals do not necessarily correspond to the ground truth of the environment. To that end, the authors divide the error into a static error component that is the same for all readings of a signal, and a error that may change at every sample of the signal.

The authors consider two cases: in the model-free case, each sensor returns a (possibly erroneous) signal at each time point. In the linear-model case, they allow for missing sensor readings, but introduce a lineardynamical model that relates these readings. The model in the latter case may also contain uncertainties.

The monitoring problem in this setting is then to decide, given a series of sensor readings, whether all possible ground truths consistent with these series satisfy a given STL formula F, its negation ~F, or whether this cannot be concluded (if in some cases F is satisfied, and in some cases ~F is satisfied).

To solve this problem, the authors translate the relations between the reading series and ground truth (according to the error margins), and the STL formula into SMT-LA (linear arithmetic constraints combined with Boolean operators). . This allows them to check for both satisfiability of the translations of F and ~F.

General Comments

The approach of the authors is, while not conceptually new, a good and interesting application of SMT solving, similar to bounded model checking. The problem tackled by the authors is a clear extension of previous approaches, and is certainly relevant.

The paper is very well written, and the language is concise. The structure is clear and the sections flow well from one to the other.

My only criticism is the lack of a larger experimental evaluation, but since the paper is clearly of a more theoretical nature, this is acceptable to me (the authors at least mention that their example is easily solved on consumer-grade hardware).

Specific Comments

p1. the author affiliations of the third and fourth author should be referred to with (superscript) 3 and 4 (currently 2)
p4. Def.3 you define "time series" by the equation
\tau(\sigma)(t) +o+e = m(t)
but in your translation use it consistently the other way around:
\tau(\sigma)(t) = m(t) + o + e
Of course this is still correct (since o and e are defined to be within symmetric intervals), but please consider swapping the definition as well. This makes it less confusing to the reader.
p8. l.290 The dynamics of x should not refer to v but to y, I suppose?
x^\prime = \frac{x}{\sqrt(2)} - \frac{y}{\sqrt{s}}
p9. l390, item 6. ellipses between c_2 y_i + ... + c_n z_i are missing
p10. l.366 [...] satisfaction of _y_ >= 0.2
p.13 l. 499 same comment as above

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Summary: 

This paper tackles the problem of STL monitoring with imprecise signals. The problem has been studied in [Visconti et al.], and this work expands the analysis, by decomposing the error into a fixed offset introduced by sensors and other noise. Compared to [Visconti et al.], this work further exploits the constraints between observations of system states at different points, and proposes an SMT-based methodology for giving a more precise verdict on the satisfaction of STL properties. Specifically, the authors present the approaches of handling such uncertain observations in the absence and presence of system dynamics respectively. The examples show that the proposed approach can give more precise verdicts of monitoring. 

Overall, this work makes a sound contribution, and states clearly the motivation and the advances of the proposed framework. Also, the presentation is clear and polished. Therefore, no major concern is with the contribution of the work.

One point is on the practicality of this work. How likely is it that the model of errors in this work is practical? Especially, is there any application where the error exactly consists of sensor offset and noise, or the way that the sensored information shifts is the same as that is presented in the paper? It would be nice to see such discussion.

Another suggestion is that it would be nice to see the experimental comparison between this work and [Visconti et al.] in terms of efficiency, since that is an important issue in online monitoring. Currently only the performance of this work is shown, but we cannot see a comparison which indicates the cost of this work. 

Some minor issues:

• Can you confirm the correctness of the G operator in Line 87?
• Shouldn’t the specification in Line 300 be G y>0.2 ? The example in Fig. 1 gives a specification to avoid the red area.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

The numbers listed to the affiliations must be revised to be consistent with the ones given in the author's list.

Sampling error is an important and challenging issue. Some problems relating the measurement error were left out and should be addressed in connection with the study: systematic vs. random error, absolute vs. relative error, and error distribution (for instance one statistic measuring the relative magnitude of the error is Pearson Chi-Square see 10.3390/info2030528; normality assumption in the distribution of the error also plays an important role especially when relations are derived in the presence of the errors - 10.1007/s13201-019-0912-1).

Some more specific references should be given for  iSAT syntax.

Looking at algorithm given (l.330-) the errors considered are absolute and this state of fact should be mentioned.

The end of the page 10 is about computing instead of being about complexity. 

However, better of (additionally) giving some more explanations about how the problem (l.330-377) is solved.

Algorithms (journal) and algorithms (concept) are scarcely represented in your bibliography. Bibliography is short.

Conclusion is too long and contains too many assert(s).

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have satisfactorily amended the article according to the comments on the earlier draft.

Reviewer 4 Report

In my opinion the authors did not addressed all my comments. However, I retained the remark that can be considered a matter of taste, so I retain myself from further comments.