# Property Checking with Interpretable Error Characterization for Recurrent Neural Networks

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Probably Approximately Correct Learning

**Lemma**

**1.**

**Proof.**

**Proposition**

**1.**

**Proof.**

## 3. Black-Box Property Checking

#### 3.1. Post-Learning Verification

**Proposition**

**2.**

- 1.
- if $H\cap \overline{P}=\varnothing $ then ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in C\cap \overline{P}$, and
- 2.
- if $H\cap \overline{P}\ne \varnothing $ then ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in \overline{C}\cap H\cap \overline{P}$,

**Proof.**

- 1.
- If $H\cap \overline{P}=\varnothing $ then $\overline{P}=\overline{H}\cap \overline{P}$. Thus, $C\cap \overline{P}=C\cap \overline{H}\cap \overline{P}$ and from Proposition 1(3) it follows that ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in C\cap \overline{H}\cap \overline{P}$, with confidence at least $1-\delta $.
- 2.
- If $H\cap \overline{P}\ne \varnothing $, from Proposition 1(4) we have that ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in \overline{C}\cap H\cap \overline{P}$, with confidence at least $1-\delta $. ☐

#### 3.2. On-the-Fly Property Checking through Learning

**Proposition**

**3.**

- 1.
- if $H=\varnothing $ then ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in \mathsf{\Psi}\left(C\right)$, and
- 2.
- if $H\ne \varnothing $ then ${\mathbb{P}}_{x\sim \mathcal{D}}\left(\right)open="["\; close="]">x\in H\backslash \mathsf{\Psi}\left(C\right)$,

**Proof.**

## 4. On-the-Fly Property-Checking for RNN

#### 4.1. Bounded-${L}^{*}$: An Algorithm for Learning DFA from RNN

**Property**

**1**

Algorithm 1: Bounded-${L}^{*}$ |

**Corollary**

**1.**

**Proof.**

#### 4.2. Analysis of the Approximation Error of Bounded-${L}^{*}$

**Theorem**

**1.**

**Proof.**

#### 4.3. Characterization of the Error Incurred by the RNN

**Proposition**

**4.**

**Proof.**

**Corollary**

**2.**

**Proof.**

**Theorem**

**2.**

- 1.
- A is an $(\tilde{\u03f5}\left(k\right),\delta )$-approximation of $\mathsf{\Psi}\left(C\right)$.
- 2.
- If $A\ne \varnothing $ or $k>0$, then $\mathsf{\Psi}\left(C\right)\ne \varnothing $.

**Proof.**

- 1.
- Straightforward from Theorem 1.
- 2.
- By Corollary 2, it follows that $A\ne \varnothing $ implies $\mathsf{\Psi}\left(C\right)\ne \varnothing $. Let $A=\varnothing $ and $k>0$. By the fact that $k>0$, we have that $A\oplus \mathsf{\Psi}\left(C\right)\ne \varnothing $. Since $A=\varnothing $, it results that $\varnothing \oplus \mathsf{\Psi}\left(C\right)=\mathsf{\Psi}\left(C\right)$. Hence, $\mathsf{\Psi}\left(C\right)\ne \varnothing $. ☐

## 5. Case Studies

#### 5.1. Context-Free Language Modeling

- 1.
- The set of sequences recognized by the RNN C is included in the Dyck 1 grammar above. That is, ${\mathsf{\Psi}}_{1}\left(C\right)=C\cap \overline{S}$. Recall that $\overline{S}$ is not computed, since only membership queries are posed.
- 2.
- The set of sequences recognized by the RNN C are included in the regular property $P={\left(c\right)}^{*}$. In this case, ${\mathsf{\Psi}}_{2}\left(C\right)=C\cap \overline{P}$.
- 3.
- The set of sequences recognized by the RNN C are included in the context-free language $Q={{(}^{m})}^{n}$ with $m<n$. Here, ${\mathsf{\Psi}}_{3}\left(C\right)=C\cap \overline{Q}$. Again, $\overline{Q}$ is not computed.

#### 5.2. Checking Equivalence between RNNs

#### 5.3. An RNN Model of a Cruise Control Software

#### 5.4. An RNN Model of an E-Commerce Web Site

#### 5.5. An RNN for Classifying Hadoop File System Logs

#### 5.6. An RNN for Recognizing TATA-Boxes in DNA Promoter Sequences

## 6. Related Work

## 7. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Miller, T. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell.
**2019**, 267, 1–38. [Google Scholar] [CrossRef] - Biran, O.; Cotton, C.V. Explanation and Justification in Machine Learning: A Survey. In Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence (XAI), Melbourne, Australia 19–25 August 2017. [Google Scholar]
- Ahmad, M.; Teredesai, A.; Eckert, C. Interpretable Machine Learning in Healthcare. In Proceedings of the 2018 IEEE International Conference on Healthcare Informatics (ICHI); IEEE Computer Society: Los Alamitos, CA, USA, 2018; p. 447. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] - Ribeiro, M.T.; Singh, S.; Guestrin, C. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the SIGKDD Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 1135–1144. [Google Scholar]
- Scheiner, N.; Appenrodt, N.; Dickmann, J.; Sick, B. Radar-based Road User Classification and Novelty Detection with Recurrent Neural Network Ensembles. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; pp. 722–729. [Google Scholar]
- Kocić, J.; Jovičić, N.; Drndarević, V. An End-to-End Deep Neural Network for Autonomous Driving Designed for Embedded Automotive Platforms. Sensors
**2019**, 19, 2064. [Google Scholar] [CrossRef] [PubMed][Green Version] - Kim, J.; Kim, J.; Thu, H.L.T.; Kim, H. Long short term memory recurrent neural network classifier for intrusion detection. In Proceedings of the 2016 International Conference on Platform Technology and Service (PlatCon), Jeju, Korea, 15–17 February 2016; pp. 1–5. [Google Scholar]
- Yin, C.; Zhu, Y.; Fei, J.; He, X. A deep learning approach for intrusion detection using recurrent neural networks. IEEE Access
**2017**, 5, 21954–21961. [Google Scholar] [CrossRef] - Pascanu, R.; Stokes, J.W.; Sanossian, H.; Marinescu, M.; Thomas, A. Malware classification with recurrent networks. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia, 19–24 April 2015; pp. 1916–1920. [Google Scholar]
- Rhode, M.; Burnap, P.; Jones, K. Early Stage Malware Prediction Using Recurrent Neural Networks. Comput. Secur.
**2017**, 77, 578–594. [Google Scholar] [CrossRef] - Vinayakumar, R.; Alazab, M.; Soman, K.; Poornachandran, P.; Venkatraman, S. Robust Intelligent Malware Detection Using Deep Learning. IEEE Access
**2019**, 7, 46717–46738. [Google Scholar] [CrossRef] - Singh, D.; Merdivan, E.; Psychoula, I.; Kropf, J.; Hanke, S.; Geist, M.; Holzinger, A. Human activity recognition using recurrent neural networks. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction; Springer: Berlin/Heidelberg, Germany, 2017; pp. 267–274. [Google Scholar]
- Choi, E.; Bahadori, M.T.; Schuetz, A.; Stewart, W.F.; Sun, J. Doctor AI: Predicting Clinical Events via Recurrent Neural Networks. In Proceedings of the 1st Machine Learning for Healthcare Conference; Doshi-Velez, F., Fackler, J., Kale, D., Wallace, B., Wiens, J., Eds.; PMLR: Boston, MA, USA, 2016; Volume 56, pp. 301–318. [Google Scholar]
- Pham, T.; Tran, T.; Phung, D.; Venkatesh, S. Predicting healthcare trajectories from medical records: A deep learning approach. J. Biomed. Inform.
**2017**, 69, 218–229. [Google Scholar] [CrossRef] - Oubounyt, M.; Louadi, Z.; Tayara, H.; Chong, K.T. DeePromoter: Robust promoter predictor using deep learning. Front. Genet.
**2019**, 10, 286. [Google Scholar] [CrossRef] [PubMed][Green Version] - Clarke, E.M.; Grumberg, O.; Peled, D.A. Model Checking; MIT Press: Cambridge, MA, USA, 1999. [Google Scholar]
- Weiss, G.; Goldberg, Y.; Yahav, E. Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples. In Proceedings of the International Conference on Machine Learning ICML, PMLR, Stockholm, Sweden, 10–15 July 2018; Volume 80. [Google Scholar]
- Wang, Q.; Zhang, K.; Ororbia, A.G., II; Xing, X.; Liu, X.; Giles, C.L. An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks. Neural Comput.
**2018**, 30, 2568–2591. [Google Scholar] [CrossRef] - Wang, Q.; Zhang, K.; Ororbia, A.G., II; Xing, X.; Liu, X.; Giles, C.L. A Comparison of Rule Extraction for Different Recurrent Neural Network Models and Grammatical Complexity. arXiv
**2018**, arXiv:1801.05420. [Google Scholar] - Merrill, W. Sequential neural networks as automata. arXiv
**2019**, arXiv:1906.01615. [Google Scholar] - Mayr, F.; Yovine, S. Regular Inference on Artificial Neural Networks. In Machine Learning and Knowledge Extraction; Springer: Cham, Switzerland, 2018; pp. 350–369. [Google Scholar]
- Valiant, L.G. A Theory of the Learnable. Commun. ACM
**1984**, 27, 1134–1142. [Google Scholar] [CrossRef][Green Version] - Odena, A.; Olsson, C.; Andersen, D.; Goodfellow, I.J. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In Proceedings of the International Conference on Machine Learning ICML, PMLR, Long Beach, CA, USA, 9–15 June 2019; Volume 97, pp. 4901–4911. [Google Scholar]
- Mayr, F.; Visca, R.; Yovine, S. On-the-fly Black-Box Probably Approximately Correct Checking of Recurrent Neural Networks. In Proceedings of the 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference on Machine Learning and Knowledge Extraction (CD-MAKE 2020), Dublin, Ireland, 25–28 August 2020; Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E.R., Eds.; Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, Germany, 2020; Volume 12279, pp. 343–363. [Google Scholar] [CrossRef]
- Peled, D.; Vardi, M.Y.; Yannakakis, M. Black box checking. J. Autom. Lang. Comb.
**2002**, 7, 225–246. [Google Scholar] - Angluin, D. Computational Learning Theory: Survey and Selected Bibliography. In Proceedings of the Twenty-Fourth Annual ACM Symposium on Theory of Computing, Victoria, BC, Canada, 4–6 May 1992; pp. 351–369. [Google Scholar]
- Ben-David, S.; Shalev-Shwartz, S. Understanding Machine Learning: From Theory to Algorithms; Cambridge University Press: Cambridge, UK, 2014. [Google Scholar]
- Angluin, D. Learning Regular Sets from Queries and Counterexamples. Inf. Comput.
**1987**, 75, 87–106. [Google Scholar] [CrossRef][Green Version] - Siegelmann, H.T.; Sontag, E.D. On the Computational Power of Neural Nets. In Proceedings of the twenty-fourth annual ACM symposium on Theory of Computing, Victoria, BC, Canada, 4–6 May 1992; pp. 440–449. [Google Scholar]
- Suzgun, M.; Belinkov, Y.; Shieber, S.M. On Evaluating the Generalization of LSTM Models in Formal Languages. arXiv
**2018**, arXiv:1811.01001. [Google Scholar] - Heinz, J.; de la Higuera, C.; van Zaanen, M. Formal and Empirical Grammatical Inference. In Proceedings of the ACL Annual Meeting, ACL, Minneapolis, MN, USA, 10–13 November 2011; pp. 2:1–2:83. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] - Hao, Y.; Merrill, W.; Angluin, D.; Frank, R.; Amsel, N.; Benz, A.; Mendelsohn, S. Context-free transductions with neural stacks. arXiv
**2018**, arXiv:1809.02836. [Google Scholar] - Hopcroft, J.E.; Motwani, R.; Ullman, J.D. Introduction to automata theory, languages, and computation. ACM Sigact News
**2001**, 32, 60–65. [Google Scholar] [CrossRef] - Tomita, M. Dynamic Construction of Finite Automata from examples using Hill-climbing. In Proceedings of the Fourth Annual Conference of the Cognitive Science Society, Vancouver, BC, Canada, 26–29 July 2006; pp. 105–108. [Google Scholar]
- Meinke, K.; Sindhu, M.A. LBTest: A Learning-Based Testing Tool for Reactive Systems. In Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, Luxembourg, 18–22 March 2013; pp. 447–454. [Google Scholar]
- Merten, M. Active Automata Learning for Real Life Applications. Ph.D. Thesis, Technischen Universität Dortmund, Dortmund, Germany, 2013. [Google Scholar]
- Du, M.; Li, F.; Zheng, G.; Srikumar, V. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, 30 October–3 November 2017; pp. 1285–1298. [Google Scholar]
- Bengio, Y.; Ducharme, R.; Vincent, P.; Janvin, C. A Neural Probabilistic Language Model. J. Mach. Learn. Res.
**2003**, 3, 1137–1155. [Google Scholar] - Craven, M.W. Extracting Comprehensible Models from Trained Neural Networks. Ph.D. Thesis, The University of Wisconsin, Madison, WI, USA, 1996. AAI9700774. [Google Scholar]
- Cobleigh, J.M.; Giannakopoulou, D.; Păsăreanu, C.S. Learning assumptions for compositional verification. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems; Springer: Berlin/Heidelberg, Germany, 2003; pp. 331–346. [Google Scholar]
- Alur, R.; Madhusudan, P.; Nam, W. Symbolic compositional verification by learning assumptions. In Proceedings of the International Conference on Computer Aided Verification; Springer: Berlin/Heidelberg, Germany, 2005; pp. 548–562. [Google Scholar]
- Feng, L.; Han, T.; Kwiatkowska, M.; Parker, D. Learning-based compositional verification for synchronous probabilistic systems. In Proceedings of the International Symposium on Automated Technology for Verification and Analysis; Springer: Berlin/Heidelberg, Germany, 2011; pp. 511–521. [Google Scholar]
- Vardhan, A.; Sen, K.; Viswanathan, M.; Agha, G. Actively learning to verify safety for FIFO automata. In Foundations of Software Technology and Theoretical Computer Science; Springer: Berlin/Heidelberg, Germany, 2004; pp. 494–505. [Google Scholar]
- Habermehl, P.; Vojnar, T. Regular Model Checking Using Inference of Regular Languages. ENTCS
**2004**, 138, 21–36. [Google Scholar] [CrossRef] - Trakhtenbrot, B.A.; Barzdin, I.M. Finite Automata: Behavior and Synthesis; North-Holland: Amsterdam, The Netherlands, 1973; 321p. [Google Scholar]
- Meinke, K. Learning-based testing: Recent progress and future prospects. In Machine Learning for Dynamic Software Analysis: Potentials and Limits; Springer: Berlin/Heidelberg, Germany, 2018; pp. 53–73. [Google Scholar]
- Meijer, J.; van de Pol, J. Sound black-box checking in the LearnLib. Innov. Syst. Softw. Eng.
**2019**, 15, 267–287. [Google Scholar] [CrossRef][Green Version] - Pulina, L.; Tacchella, A. Challenging SMT solvers to verify neural networks. AI Commun.
**2012**, 25, 117–135. [Google Scholar] [CrossRef] - Katz, G.; Barrett, C.W.; Dill, D.L.; Julian, K.; Kochenderfer, M.J. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In Proceedings of the International Conference on Computer Aided Verification; Springer: Berlin/Heidelberg, Germany, 2017; Volume 10426, pp. 97–117. [Google Scholar]
- Huang, X.; Kwiatkowska, M.; Wang, S.; Wu, M. Safety Verification of Deep Neural Networks. In Proceedings of the International Conference on Computer Aided Verification; Springer: Berlin/Heidelberg, Germany, 2017; Volume 10426, pp. 3–29. [Google Scholar]
- Ehlers, R. Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks. In Proceedings of the International Symposium on Automated Technology for Verification and Analysis; Springer: Cham, Switzerland, 2017; Volume 10482, pp. 269–286. [Google Scholar]
- Gopinath, D.; Katz, G.; Pasareanu, C.S.; Barrett, C.W. DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks. In Proceedings of the 16th International Symposium on Automated Technology for Verification and Analysis (ATVA 2018); Los Angeles, CA, USA, 7–10 October 2018, Lahiri, S.K., Wang, C., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2018; Volume 11138, pp. 3–19. [Google Scholar] [CrossRef]
- Wicker, M.; Huang, X.; Kwiatkowska, M. Feature-guided black-box safety testing of deep neural networks. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems; Springer: Berlin/Heidelberg, Germany, 2018; pp. 408–426. [Google Scholar]
- Wang, Q.; Zhang, K.; Liu, X.; Giles, C.L. Verification of Recurrent Neural Networks Through Rule Extraction. In Proceedings of the AAAI Spring Symposium on Verification of Neural Networks (VNN19), Stanford, CA, USA, 25–27 March 2019. [Google Scholar]
- Kevorchian, A. Verification of Recurrent Neural Networks. Master’s Thesis, Imperial College London, London, UK, 2018. [Google Scholar]
- Akintunde, M.E.; Kevorchian, A.; Lomuscio, A.; Pirovano, E. Verification of RNN-Based Neural Agent-Environment Systems. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, New York, NY, USA, 7–12 February 2019; pp. 6006–6013. [Google Scholar]
- Legay, A.; Lukina, A.; Traonouez, L.M.; Yang, J.; Smolka, S.A.; Grosu, R. Statistical Model Checking. In Computing and Software Science: State of the Art and Perspectives; Springer: Cham, Switzerland, 2019; pp. 478–504. [Google Scholar]
- Agha, G.; Palmskog, K. A Survey of Statistical Model Checking. ACM Trans. Model. Comput. Simul.
**2018**, 28, 1–39. [Google Scholar] [CrossRef]

**Figure 10.**Sketch of the architecture of the language model of Hadoop Distributed File System (HDFS) logs.

**Figure 11.**Hadoop file system logs: Automaton for ${\mathsf{\Psi}}_{1}\left(C\right)$ obtained with $\u03f5=0.01$ and $\delta =0.01$.

**Table 1.**Dyck 1: Probably approximately correct (PAC) deterministic finite automata (DFA) extraction from recurrent neural networks (RNN).

Parameters | Running Time (in s) | Mean Sample Size | |||
---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | min | max | mean | |

0.005 | 0.005 | 1.984 | 7.205 | 3.072 | 1899 |

0.0005 | 0.005 | 3.713 | 10.445 | 5.997 | 20,093 |

0.00005 | 0.005 | 7.982 | 30.470 | 9.997 | 203,007 |

0.00005 | 0.0005 | 8.128 | 36.621 | 9.919 | 249,059 |

0.00005 | 0.00005 | 9.625 | 41.884 | 12.185 | 295,111 |

$\mathsf{\Psi}$ | Parameters | Running Time (in s) | First Positive $\mathbf{MQ}$ | Mean Sample Size | |||
---|---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | min | max | mean | |||

${\mathsf{\Psi}}_{1}$ | 0.005 | 0.005 | 0.004 | 0.012 | 0.006 | - | 1476 |

0.0005 | 0.005 | 0.051 | 0.125 | 0.067 | - | 14,756 | |

0.00005 | 0.005 | 0.682 | 0.833 | 0.747 | - | 147,556 | |

0.00005 | 0.0005 | 1.164 | 1.595 | 1.340 | - | 193,607 | |

0.00005 | 0.00005 | 1.272 | 1.809 | 1.386 | - | 239,659 | |

${\mathsf{\Psi}}_{2}$ | 0.005 | 0.005 | 0.031 | 34.525 | 5.762 | 0.099 | 1948 |

0.0005 | 0.005 | 0.397 | 37.846 | 10.245 | 0.084 | 20,370 | |

0.00005 | 0.005 | 4.713 | 30.714 | 6.547 | 0.825 | 206,473 | |

${\mathsf{\Psi}}_{3}$ | 0.005 | 0.005 | 0.025 | 0.966 | 0.302 | 0.006 | 1899 |

0.0005 | 0.005 | 0.267 | 1.985 | 0.787 | 0.070 | 20,093 | |

0.00005 | 0.005 | 4.376 | 6.479 | 4.775 | 0.764 | 203,007 |

Parameters | Running Time (in s) | Mean Sample Size | Mean $\tilde{\mathit{\u03f5}}$ | |||
---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | ||

0.005 | 0.005 | 2.753 | 149.214 | 19.958 | 1795 | 0.00559 |

0.0005 | 0.005 | 23.343 | 300.000 | 105.367 | 18,222 | 0.04432 |

0.00005 | 0.005 | 42.518 | 139.763 | 77.652 | 186,372 | 0.16248 |

Parameters | Running Time (in s) | First Positive $\mathbf{MQ}$ | Mean Sample Size | Mean $\tilde{\mathit{\u03f5}}$ | |||
---|---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | |||

0.005 | 0.005 | 0.004 | 122.388 | 24.483 | 90.285 | 1504 | 0.00618 |

0.0005 | 0.005 | 55.084 | 300.000 | 215.508 | 42.462 | 16,604 | 0.00895 |

0.00005 | 0.005 | 0.695 | 324.144 | 158.195 | 4.545 | 166,040 | 0.00005 |

Network | Dataset Size | Batch Size | Sequence Length | Learning Rate |
---|---|---|---|---|

${N}_{1}$ | 5K | 30 | 15 | 0.01 |

${N}_{2}$ | 1M | 100 | 10 | 0.001 |

Parameters | Running Times (in s) | First Positive $\mathbf{MQ}$ | Mean Sample Size | |||
---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | ||

0.01 | 0.01 | 0.003 | 0.006 | 0.004 | - | 669 |

0.001 | 0.01 | 0.061 | 0.096 | 0.075 | - | 6685 |

0.0001 | 0.01 | 0.341 | 0.626 | 0.497 | - | 66,847 |

Parameters | Running Times (in s) | Mean Sample Size | Mean $\tilde{\mathit{\u03f5}}$ | |||
---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | ||

0.01 | 0.01 | 11.633 | 200.000 | 67.662 | 808 | 0.07329 |

0.001 | 0.01 | 52.362 | 200.000 | 135.446 | 8071 | 0.22684 |

0.0001 | 0.01 | - | - | - | - | - |

Parameters | Running Times (in s) | Mean Sample Size | |||
---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | |

0.01 | 0.01 | 16.863 | 62.125 | 36.071 | 863 |

0.001 | 0.01 | 6.764 | 9.307 | 7.864 | 8487 |

0.0001 | 0.01 | 18.586 | 41.137 | 30.556 | 83,482 |

$\mathsf{\Psi}$ | Parameters | Running Times (in s) | First Positive $\mathbf{MQ}$ | Mean Sample Size | |||
---|---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | |||

${\mathsf{\Psi}}_{1}$ | 0.01 | 0.01 | 87.196 | 312.080 | 174.612 | 3.878 | 891 |

0.001 | 0.01 | 0.774 | 203.103 | 102.742 | 0.744 | 9181 | |

0.0001 | 0.01 | 105.705 | 273.278 | 190.948 | 2.627 | 94,573 | |

${\mathsf{\Psi}}_{2}$ | 0.01 | 0.01 | 0.002 | 487.709 | 148.027 | 80.738 | 752 |

0.001 | 0.01 | 62.457 | 600.000 | 428.400 | 36.606 | 8765 | |

0.0001 | 0.01 | 71.542 | 451.934 | 250.195 | 41.798 | 87,641 |

Prop | Parameters | Running Times (in s) | First Positive $\mathbf{MQ}$ | Mean Sample Size | |||
---|---|---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | |||

${\mathsf{\Psi}}_{1}$ | 0.01 | 0.01 | 209.409 | 1,121.360 | 555.454 | 5.623 | 932 |

0.001 | 0.001 | 221.397 | 812.764 | 455.660 | 1.321 | 12,037 | |

${\mathsf{\Psi}}_{2}$ | 0.01 | 0.01 | 35.131 | 39.762 | 37.226 | - | 600 |

0.001 | 0.001 | 252.202 | 257.312 | 254.479 | - | 8295 |

Parameters | Running Times (in s) | Mean Sample Size | |||
---|---|---|---|---|---|

$\mathit{\u03f5}$ | $\mathit{\delta}$ | Min | Max | Mean | |

0.01 | 0.01 | 5.098 | 5.259 | 5.168 | 600 |

0.001 | 0.001 | 65.366 | 66.479 | 65.812 | 8295 |

0.0001 | 0.0001 | 865.014 | 870.663 | 867.830 | 105,967 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Mayr, F.; Yovine, S.; Visca, R.
Property Checking with Interpretable Error Characterization for Recurrent Neural Networks. *Mach. Learn. Knowl. Extr.* **2021**, *3*, 205-227.
https://doi.org/10.3390/make3010010

**AMA Style**

Mayr F, Yovine S, Visca R.
Property Checking with Interpretable Error Characterization for Recurrent Neural Networks. *Machine Learning and Knowledge Extraction*. 2021; 3(1):205-227.
https://doi.org/10.3390/make3010010

**Chicago/Turabian Style**

Mayr, Franz, Sergio Yovine, and Ramiro Visca.
2021. "Property Checking with Interpretable Error Characterization for Recurrent Neural Networks" *Machine Learning and Knowledge Extraction* 3, no. 1: 205-227.
https://doi.org/10.3390/make3010010