# Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks

^{1}

^{2}

^{3}

^{4}

^{5}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Dual Attention Mechanism Structure and Learning Procedures

#### Choice of the Time Window for Network Training

#### 2.2. Deep Neural Network Parameter Analysis

#### 2.3. Response of the Prediction to Noise Addition

#### 2.4. Clustering and Principal Component Analysis

## 3. Results

#### 3.1. Gene Regulatory Network Dynamic

- ${J}_{ij}$ is the interaction matrix among the genes i and j, which is 1 if gene j expresses a protein, at concentration ${x}_{j}$, that is a TF for gene i, −1 if it is a repressor and 0 if the two genes do not interact.
- ${k}_{ij}$ and ${\tilde{k}}_{ij}$ are the expression and repression rates for the gene i from the TF (repressor) j.
- ${h}_{ij}$ is the cooperativity index of the $ij$ interaction.
- ${K}_{ij}$ is the dissociation constant of the reaction $ij$.
- ${\tau}_{ij}$ is the delay between the expression of the protein ${x}_{j}$ and the time when it is ready to activate (repress) the gene i. It allows one to account for the translation of mRNAs into proteins and it has been previously included in Gillespie simulations [50,51,52]. However, in the present study, we set it to zero for the sake of simplicity.

#### 3.2. Recurrent Neural Network for Stochastic Time Traces Generated by Different Gene Regulatory Network Architectures

#### 3.3. Studying the Input Attention Mechanism and the Response of the Prediction to Noise

#### 3.4. Network Properties of the Input Attention Distinguishing Gene Regulatory Network Architectures

#### 3.5. Differential Response to Noise in the Prediction of Time Series Gene Expression Data

#### 3.6. Comparison between Clustering of Gene Regulatory Networks

## 4. Discussion

## Supplementary Materials

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A

- Define the network, the matrix of interaction, the nature of the interactions, the rates of the reactions, the number of species (nodes), the final time ${T}_{max}$ of the simulation and the propensity functions ${a}_{j}$;
- Select the time of the first reaction from the exponential probability distribution of the times, $P\left(\tau \right)={e}^{-{a}_{tot}\tau}$;
- Select the reaction from the relative propensity, $P\left(j\right)=\frac{{a}_{j}}{{a}_{tot}}$;
- Upgrade the species according to the rules of the reaction j;
- Compute the new propensities ${a}_{j}$ and ${a}_{tot}={\sum}_{j}{a}_{j}$ from the updated set of the network;
- Update the time $t+=\tau $;
- Repeat from step 1 until $t<{T}_{max}$.

## Appendix B

## References

- Vidal, M.; Cusick, M.E.; Barabási, A.L. Interactome Networks and Human Disease. Cell
**2011**, 144, 986–998. [Google Scholar] [CrossRef] [PubMed][Green Version] - Barabási, A.L.; Gulbahce, N.; Loscalzo, J. Network medicine: A network-based approach to human disease. Nat. Rev. Genet.
**2011**, 12, 56–68. [Google Scholar] [CrossRef][Green Version] - Dimitrakopoulou, K.; Dimitrakopoulos, G.N.; Wilk, E.; Tsimpouris, C.; Sgarbas, K.N.; Schughart, K.; Bezerianos, A. Influenza A Immunomics and Public Health Omics: The Dynamic Pathway Interplay in Host Response to H1N1 Infection. OMICS
**2014**, 18, 167–183. [Google Scholar] [CrossRef] [PubMed] - Monti, M.; Guiducci, G.; Paone, A.; Rinaldo, S.; Giardina, G.; Liberati, F.R.; Cutruzzolá, F.; Tartaglia, G.G. Modelling of SHMT1 riboregulation predicts dynamic changes of serine and glycine levels across cellular compartments. Comput. Struct. Biotechnol. J.
**2021**, 19, 3034–3041. [Google Scholar] [CrossRef] [PubMed] - Milo, R.; Shen-Orr, S.; Itzkovitz, S.; Kashtan, N.; Chklovskii, D.; Alon, U. Network motifs: Simple building blocks of complex networks. Science
**2002**, 298, 824–827. [Google Scholar] [CrossRef][Green Version] - Mangan, S.; Alon, U. Structure and function of the feed-forward loop network motif. Proc. Natl. Acad. Sci. USA
**2003**, 100, 11980–11985. [Google Scholar] [CrossRef][Green Version] - Shen-Orr, S.S.; Milo, R.; Mangan, S.; Alon, U. Network motifs in the transcriptional regulation network of Escherichia coli. Nat. Genet.
**2002**, 31, 64–68. [Google Scholar] [CrossRef] - Karlebach, G.; Shamir, R. Modelling and analysis of gene regulatory networks. Nat. Rev. Mol. Cell Biol.
**2008**, 9, 770–780. [Google Scholar] [CrossRef] - Tkačik, G.; Bialek, W. Information processing in living systems. Annu. Rev. Condens. Matter Phys.
**2016**, 7, 89–117. [Google Scholar] [CrossRef][Green Version] - Tkačik, G.; Callan, C.G.; Bialek, W. Information flow and optimization in transcriptional regulation. Proc. Natl. Acad. Sci. USA
**2008**, 105, 12265–12270. [Google Scholar] [CrossRef][Green Version] - Fiorentino, J.; De Martino, A. Independent channels for miRNA biosynthesis ensure efficient static and dynamic control in the regulation of the early stages of myogenesis. J. Theor. Biol.
**2017**, 430, 53–63. [Google Scholar] [CrossRef][Green Version] - Delgado, F.M.; Gómez-Vela, F. Computational methods for gene regulatory networks reconstruction and analysis: A review. Artif. Intell. Med.
**2019**, 95, 133–145. [Google Scholar] [CrossRef] [PubMed] - Fiers, M.W.; Minnoye, L.; Aibar, S.; Bravo González-Blas, C.; Kalender Atak, Z.; Aerts, S. Mapping gene regulatory networks from single-cell omics data. Briefings Funct. Genom.
**2018**, 17, 246–254. [Google Scholar] [CrossRef] [PubMed] - Huynh-Thu, V.A.; Sanguinetti, G. Gene regulatory network inference: An introductory survey. In Gene Regulatory Networks; Springer: Cham, Switzerland, 2019; pp. 1–23. [Google Scholar]
- Pratapa, A.; Jalihal, A.P.; Law, J.N.; Bharadwaj, A.; Murali, T. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nat. Methods
**2020**, 17, 147–154. [Google Scholar] [CrossRef] - Alipanahi, B.; Delong, A.; Weirauch, M.T.; Frey, B.J. Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nat. Biotechnol.
**2015**, 33, 831–838. [Google Scholar] [CrossRef] [PubMed] - Zrimec, J.; Börlin, C.S.; Buric, F.; Muhammad, A.S.; Chen, R.; Siewers, V.; Verendel, V.; Nielsen, J.; Töpel, M.; Zelezniak, A. Deep learning suggests that gene expression is encoded in all parts of a co-evolving interacting gene regulatory structure. Nat. Commun.
**2020**, 11, 6141. [Google Scholar] [CrossRef] - Yang, Y.; Fang, Q.; Shen, H.B. Predicting gene regulatory interactions based on spatial gene expression data and deep learning. Plos Comput. Biol.
**2019**, 15, e1007324. [Google Scholar] [CrossRef] - Shu, H.; Zhou, J.; Lian, Q.; Li, H.; Zhao, D.; Zeng, J.; Ma, J. Modeling gene regulatory networks using neural network architectures. Nat. Comput. Sci.
**2021**, 1, 491–501. [Google Scholar] [CrossRef] - Chen, J.; Cheong, C.; Lan, L.; Zhou, X.M.; Liu, J.; Lu, A.; Cheung, W.K.; Zhang, L. DeepDRIM: A deep neural network to reconstruct cell-type-specific gene regulatory network using single-cell RNA-seq data. Briefings Bioinform.
**2021**, 22, bbab325. [Google Scholar] [CrossRef] - Wang, J.; Wang, J.; Fang, W.; Niu, H. Financial time series prediction using elman recurrent random neural networks. Comput. Intell. Neurosci.
**2016**, 2016, 4742515. [Google Scholar] [CrossRef][Green Version] - Ouma, Y.O.; Cheruyot, R.; Wachera, A.N. Rainfall and runoff time-series trend analysis using LSTM recurrent neural network and wavelet neural network with satellite-based meteorological data: Case study of Nzoia hydrologic basin. Complex Intell. Syst.
**2021**, 1–24. [Google Scholar] [CrossRef] - Zhang, Z.; Dong, Y. Temperature forecasting via convolutional recurrent neural networks based on time-series data. Complexity
**2020**, 2020, 3536572. [Google Scholar] [CrossRef] - Raeesi, M.; Mesgari, M.; Mahmoudi, P. Traffic time series forecasting by feedforward neural network: A case study based on traffic data of Monroe. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
**2014**, 40, 219. [Google Scholar] [CrossRef][Green Version] - Panella, M. Advances in biological time series prediction by neural networks. Biomed. Signal Process. Control
**2011**, 6, 112–120. [Google Scholar] [CrossRef] - Minas, G.; Rand, D.A. Long-time analytic approximation of large stochastic oscillators: Simulation, analysis and inference. PLoS Comput. Biol.
**2017**, 13, e1005676. [Google Scholar] [CrossRef][Green Version] - Boutaba, R.; Salahuddin, M.A.; Limam, N.; Ayoubi, S.; Shahriar, N.; Estrada-Solano, F.; Caicedo, O.M. A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities. J. Internet Serv. Appl.
**2018**, 9, 16. [Google Scholar] [CrossRef][Green Version] - Längkvist, M.; Karlsson, L.; Loutfi, A. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognit. Lett.
**2014**, 42, 11–24. [Google Scholar] [CrossRef][Green Version] - Muzio, G.; O’Bray, L.; Borgwardt, K. Biological network analysis with deep learning. Briefings Bioinform.
**2021**, 22, 1515–1530. [Google Scholar] [CrossRef] - Che, Z.; Purushotham, S.; Cho, K.; Sontag, D.; Liu, Y. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Sci. Rep.
**2018**, 8, 6085. [Google Scholar] [CrossRef][Green Version] - Vijayan, V.; Zuzow, R.; O’Shea, E.K. Oscillations in supercoiling drive circadian gene expression in cyanobacteria. Proc. Natl. Acad. Sci. USA
**2009**, 106, 22564–22568. [Google Scholar] [CrossRef][Green Version] - Markson, J.S.; Piechura, J.R.; Puszynska, A.M.; O’Shea, E.K. Circadian control of global gene expression by the cyanobacterial master regulator RpaA. Cell
**2013**, 155, 1396–1408. [Google Scholar] [CrossRef] [PubMed][Green Version] - Eser, P.; Demel, C.; Maier, K.C.; Schwalb, B.; Pirkl, N.; Martin, D.E.; Cramer, P.; Tresch, A. Periodic mRNA synthesis and degradation co-operate during cell cycle gene expression. Mol. Syst. Biol.
**2014**, 10, 717. [Google Scholar] [CrossRef] - Zeng, Y.; Liu, C.; Gong, Y.; Bai, Z.; Hou, S.; He, J.; Bian, Z.; Li, Z.; Ni, Y.; Yan, J.; et al. Single-cell RNA sequencing resolves spatiotemporal development of pre-thymic lymphoid progenitors and thymus organogenesis in human embryos. Immunity
**2019**, 51, 930–948. [Google Scholar] [CrossRef] [PubMed] - Treutlein, B.; Lee, Q.Y.; Camp, J.G.; Mall, M.; Koh, W.; Shariati, S.A.M.; Sim, S.; Neff, N.F.; Skotheim, J.M.; Wernig, M.; et al. Dissecting direct reprogramming from fibroblast to neuron using single-cell RNA-seq. Nature
**2016**, 534, 391–395. [Google Scholar] [CrossRef][Green Version] - Haghverdi, L.; Büttner, M.; Wolf, F.A.; Buettner, F.; Theis, F.J. Diffusion pseudotime robustly reconstructs lineage branching. Nat. Methods
**2016**, 13, 845–848. [Google Scholar] [CrossRef] [PubMed][Green Version] - Street, K.; Risso, D.; Fletcher, R.B.; Das, D.; Ngai, J.; Yosef, N.; Purdom, E.; Dudoit, S. Slingshot: Cell lineage and pseudotime inference for single-cell transcriptomics. BMC Genom.
**2018**, 19, 477. [Google Scholar] [CrossRef] [PubMed][Green Version] - Liu, Z.; Lou, H.; Xie, K.; Wang, H.; Chen, N.; Aparicio, O.M.; Zhang, M.Q.; Jiang, R.; Chen, T. Reconstructing cell cycle pseudo time-series via single-cell transcriptome data. Nat. Commun.
**2017**, 8, 22. [Google Scholar] [CrossRef] - Chen, W.; Guillaume-Gentil, O.; Dainese, R.; Rainer, P.Y.; Zachara, M.; Gabelein, C.G.; Vorholt, J.A.; Deplancke, B. Genome-wide molecular recording using Live-seq. bioRxiv
**2021**. [Google Scholar] [CrossRef] - Wang, Z.; Gerstein, M.; Snyder, M. RNA-Seq: A revolutionary tool for transcriptomics. Nat. Rev. Genet.
**2009**, 10, 57–63. [Google Scholar] [CrossRef] - Lindsay, G.W.; Miller, K.D. How biological attention mechanisms improve task performance in a large-scale visual system model. eLife
**2018**, 7, e38105. [Google Scholar] [CrossRef] - Folli, V.; Gosti, G.; Leonetti, M.; Ruocco, G. Effect of dilution in asymmetric recurrent neural networks. Neural Netw.
**2018**, 104, 50–59. [Google Scholar] [CrossRef] [PubMed] - Leonetti, M.; Folli, V.; Milanetti, E.; Ruocco, G.; Gosti, G. Network dilution and asymmetry in an efficient brain. Philos. Mag.
**2020**, 100, 2544–2555. [Google Scholar] [CrossRef][Green Version] - Gosti, G.; Folli, V.; Leonetti, M.; Ruocco, G. Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks. Entropy
**2019**, 21, 726. [Google Scholar] [CrossRef] [PubMed][Green Version] - Qin, Y.; Song, D.; Chen, H.; Cheng, W.; Jiang, G.; Cottrell, G. A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction. arXiv
**2017**, arXiv:1704.02971. [Google Scholar] - Tao, Y.; Ma, L.; Zhang, W.; Liu, J.; Liu, W.; Du, Q. Hierarchical attention-based recurrent highway networks for time series prediction. arXiv
**2018**, arXiv:1806.00685. [Google Scholar] - Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv
**2017**, arXiv:1412.6980v9. [Google Scholar] - Smith, M.R. Information theoretic generalized Robinson–Foulds metrics for comparing phylogenetic trees. Bioinformatics
**2020**, 36, 5007–5013. [Google Scholar] [CrossRef] [PubMed] - Meilă, M. Comparing clusterings—An information based distance. J. Multivar. Anal.
**2007**, 98, 873–895. [Google Scholar] [CrossRef][Green Version] - Zwicker, D.; Lubensky, D.K.; ten Wolde, P.R. Robust circadian clocks from coupled protein-modification and transcription–translation cycles. Proc. Natl. Acad. Sci. USA
**2010**, 107, 22540–22545. [Google Scholar] [CrossRef][Green Version] - Bratsun, D.; Volfson, D.; Tsimring, L.S.; Hasty, J. Delay-induced stochastic oscillations in gene regulation. Proc. Natl. Acad. Sci. USA
**2005**, 102, 14593–14598. [Google Scholar] [CrossRef][Green Version] - Cao, Y.; Gillespie, D.T.; Petzold, L.R. The slow-scale stochastic simulation algorithm. J. Chem. Phys.
**2005**, 122, 014116. [Google Scholar] [CrossRef][Green Version] - Goldbeter, A.; Dupont, G.; Berridge, M.J. Minimal model for signal-induced Ca
^{2+}oscillations and for their frequency encoding through protein phosphorylation. Proc. Natl. Acad. Sci. USA**1990**, 87, 1461–1465. [Google Scholar] [CrossRef][Green Version] - Tostevin, F.; ten Wolde, P.R. Mutual Information between Input and Output Trajectories of Biochemical Networks. Phys. Rev. Lett.
**2009**, 102, 218101. [Google Scholar] [CrossRef] [PubMed][Green Version] - Monti, M.; Lubensky, D.K.; ten Wolde, P.R. Robustness of Clocks to Input Noise. Phys. Rev. Lett.
**2018**, 121, 078101. [Google Scholar] [CrossRef][Green Version] - Monti, M.; Lubensky, D.K.; ten Wolde, P.R. Optimal entrainment of circadian clocks in the presence of noise. Phys. Rev. E
**2018**, 97, 032405. [Google Scholar] [CrossRef] [PubMed][Green Version] - Barić, D.; Fumić, P.; Horvatić, D.; Lipic, T. Benchmarking Attention-Based Interpretability of Deep Learning in Multivariate Time Series Predictions. Entropy
**2021**, 23, 143. [Google Scholar] [CrossRef] [PubMed] - de Jong, T.V.; Moshkin, Y.M.; Guryev, V. Gene expression variability: The other dimension in transcriptome analysis. Physiol. Genom.
**2019**, 51, 145–158. [Google Scholar] [CrossRef] [PubMed][Green Version] - Miotto, M.; Marinari, E.; De Martino, A. Competing endogenous RNA crosstalk at system level. PLoS Comput. Biol.
**2019**, 15, e1007474. [Google Scholar] [CrossRef] - Crisanti, A.; De Martino, A.; Fiorentino, J. Statistics of optimal information flow in ensembles of regulatory motifs. Phys. Rev. E
**2018**, 97, 022407. [Google Scholar] [CrossRef][Green Version] - Peruzzi, G.; Miotto, M.; Maggio, R.; Ruocco, G.; Gosti, G. Asymmetric binomial statistics explains organelle partitioning variance in cancer cell proliferation. Commun. Phys.
**2021**, 4, 188. [Google Scholar] [CrossRef] - Brennecke, P.; Anders, S.; Kim, J.K.; Kołodziejczyk, A.A.; Zhang, X.; Proserpio, V.; Baying, B.; Benes, V.; Teichmann, S.A.; Marioni, J.C.; et al. Accounting for technical noise in single-cell RNA-seq experiments. Nat. Methods
**2013**, 10, 1093–1095. [Google Scholar] [CrossRef] [PubMed] - Du, P.; Stolovitzky, G.; Horvatovich, P.; Bischoff, R.; Lim, J.; Suits, F. A noise model for mass spectrometry based proteomics. Bioinformatics
**2008**, 24, 1070–1077. [Google Scholar] [CrossRef] [PubMed][Green Version] - Selewa, A.; Dohn, R.; Eckart, H.; Lozano, S.; Xie, B.; Gauchat, E.; Elorbany, R.; Rhodes, K.; Burnett, J.; Gilad, Y.; et al. Systematic Comparison of High-throughput Single-Cell and Single-Nucleus Transcriptomes during Cardiomyocyte Differentiation. Sci. Rep.
**2020**, 10, 1535. [Google Scholar] [CrossRef] [PubMed][Green Version] - Marbach, D.; Costello, J.C.; Küffner, R.; Vega, N.M.; Prill, R.J.; Camacho, D.M.; Allison, K.R.; Kellis, M.; Collins, J.J.; Stolovitzky, G. Wisdom of crowds for robust gene network inference. Nat. Methods
**2012**, 9, 796–804. [Google Scholar] [CrossRef] [PubMed][Green Version] - Dominguez, D.; Tsai, Y.H.; Gomez, N.; Jha, D.K.; Davis, I.; Wang, Z. A high-resolution transcriptome map of cell cycle reveals novel connections between periodic genes and cancer. Cell Res.
**2016**, 26, 946–962. [Google Scholar] [CrossRef][Green Version] - Hannam, R.; Annibale, A.; Kühn, R. Cell reprogramming modelled as transitions in a hierarchy of cell cycles. J. Phys. Math. Theor.
**2017**, 50, 425601. [Google Scholar] [CrossRef][Green Version] - Szedlak, A.; Sims, S.; Smith, N.; Paternostro, G.; Piermarocchi, C. Cell cycle time series gene expression data encoded as cyclic attractors in Hopfield systems. PLoS Comput. Biol.
**2017**, 13, e1005849. [Google Scholar] [CrossRef][Green Version] - Barbuti, R.; Gori, R.; Milazzo, P.; Nasti, L. A survey of gene regulatory networks modelling methods: From differential equations, to Boolean and qualitative bioinspired models. J. Membr. Comput.
**2020**, 2, 207–226. [Google Scholar] [CrossRef] - Tkačik, G.; Walczak, A.M.; Bialek, W. Optimizing information flow in small genetic networks. III. A self-interacting gene. Phys. Rev. E
**2012**, 85, 041903. [Google Scholar] [CrossRef][Green Version]

**Figure 1.**(

**A**): Scheme of the dual attention recurrent neural network (DA-RNN) used in this study. From left to right: The network was trained on gene expression time traces of length T. The first attention layer (input attention) encoded which input time traces were more important for target prediction. The second attention layer (temporal attention) encoded the most important time points of the input time traces for the prediction. The LSTM chain on the right was the decoder that provided the final output. (

**B**): Architecture of the parallel DA-RNN. Each box represents a DA-RNN devoted to the prediction of gene i; after the prediction of the next time step, the data were collected to fuel the inputs for the parallel nets and to predict the next time point. (

**C**): Distribution of the autocorrelation time for the genes of a fully connected and an oscillating network (see Section 3.2 for details on different GRN architectures). (

**D**): Training time for all the genes as a function of the time window T used for training, for the same two networks. (

**E**): RMSE as a function of the time window T used for training, for the same two networks. (

**F**): Kullback–Leibler divergence (circles) and Jensen–Shannon distance (triangles) between the probability distribution of the real data and that of the predicted ones, plotted as a function of the training window, for the same two networks. The values shown for each value of the time window are averages over all the genes of the GRN.

**Figure 2.**Gene regulatory network dynamic with $N=3$ genes. The dynamic was generated using a random interaction matrix ${J}_{ij}$. The top panel shows the continuous time trace of proteins concentration for a random GRN with ${n}_{r}={n}_{a}=0.2$. The bottom panel shows the oscillatory dynamic of 3 genes relative to the core clock of an oscillatory network. In both panels, dots show the prediction of the neural network implemented using the first $T=10$ time points and then letting the system evolve on its own. The stochastic data were generated using the Gillespie algorithm.

**Figure 3.**Time dynamics and full probability function (integrated over all the time of the simulation) of having a protein at a certain concentration. We show selected example genes from an oscillatory GRN (panels

**B**,

**D**) and a GRN controlled by a master regulator gene (panels

**A**,

**C**), both with ${n}_{r}={n}_{a}=0.5$. The plots show the performance of the prediction comparing the time traces between the stochastic simulation and the DNN propagation. Moreover, the resulting probability distribution of the given protein concentration for all times is reported. We highlight that the protein concentration dynamic was reconstructed with high accuracy and even the noise amplitude and the mean were quite well reported by the neural net. Moreover, for the genes belonging to the oscillating GRN (panels

**B**,

**D**), it was possible to reconstruct the amplitude and the mean of the oscillations. We stress that the dynamic shown in panel

**B**could not be immediately addressed to a gene driven by an oscillatory clock, but the bi-modal distribution shows that this was the case. Interestingly, this reveals that, given a model trained on real data, we can look at its internal parameters that allow us to determine if the genes in the GRN belong or not to a certain network topology such as an oscillatory-driven network, such as, for instance, the cell cycle or circadian clock.

**Figure 4.**Comparison of the matrices resulting from the input attention and response-to-noise analyses. Scatter plots showing the relationship between the matrix elements of those obtained computing the network properties of the input attention (clustering coefficient, hub score and betweenness) and the matrix of normalized mean MSE obtained from the analysis of the response of the prediction to noise. Note that the data are represented in log–log scale using the transformation $sign\left(x\right)ln(1+|x\left|\right)$, since they included both positive and negative values. The Pearson correlation coefficient $\rho $ between each pair of properties, computed on the transformed data, is reported on top of each panel.

**Figure 5.**Clustering based on the network properties of the input attention matrices. We show the dendrograms obtained from a hierarchical clustering (

**left**), a PCA plot showing the 12 gene regulatory networks used (

**centre**) and the obtained partition in groups (

**right**) for the clustering coefficient (

**A**), betweenness (

**B**) and hub score (

**C**).

**Figure 6.**Impact of noise addition on time series gene expression prediction for different GRN architectures. (

**A**): Matrices of the mean square error (MSE) on the prediction by the DA-RNN, for an oscillating GRN with 10 repressors and activators per gene on average (

**left**) and a GRN controlled by a master regulator (

**right**). Rows (genes) are ranked according to the mean of the MSE (last column). (

**B**): Dendrogram obtained from the hierarchical clustering of the matrix summarizing the response of the prediction to noise for each gene regulatory network. (

**C**,

**D**): PCA showing the 12 GRNs used and their partition in clusters.

**Figure 7.**Comparison between the clusterings. (

**A**): Comparison between the dendrograms of the different hierarchical clustering shown in Figure 5 and Figure 6 using the information-based generalized Robinson–Foulds distance (or tree distance). (

**B**): Comparison between the partitions obtained using the variation of information.

**Table 1.**Prediction accuracy of the DA-RNN for different gene regulatory network architectures. The network types are those described in Section 3.2, where ${n}_{r}$ and ${n}_{a}$ indicate the mean number of repressors and activators per gene, respectively. We report the mean and standard deviation of the root-mean-square error (RMSE) computed over all the genes in each network.

Network | Mean (RMSE) | Std (RMSE) |
---|---|---|

FullyConnected | 0.12 | 0.04 |

FullyRepressed | 0.67 | 0.08 |

MasterRegulator | 0.83 | 0.04 |

SparseConnection | 0.14 | 0.05 |

mediumConnection | 0.35 | 0.23 |

Oscillating_nr1_na1 | 0.22 | 0.11 |

Oscillating_nr5_na5 | 0.23 | 0.13 |

Oscillating_nr10_na10 | 0.30 | 0.17 |

Oscillating_nr15_na15 | 0.12 | 0.09 |

ExternalSignal_nr1_na1 | 0.29 | 0.20 |

ExternalSignal_nr5_na5 | 0.39 | 0.28 |

ExternalSignal_nr10_na10 | 0.09 | 0.06 |

**Table 2.**Pearson correlation coefficient $\rho $ between input attention and true interaction matrix.

Network | $\mathit{\rho}$ |
---|---|

FullyConnected | −0.018 |

FullyRepressed | 0.0 |

MasterRegulator | −0.080 |

SparseConnection | −0.10 |

mediumConnection | −0.0087 |

Oscillating_nr1_na1 | −0.0070 |

Oscillating_nr5_na5 | 0.017 |

Oscillating_nr10_na10 | 0.061 |

Oscillating_nr15_na15 | 0.12 |

ExternalSignal_nr1_na1 | −0.0020 |

ExternalSignal_nr5_na5 | 0.075 |

ExternalSignal_nr10_na10 | 0.021 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Monti, M.; Fiorentino, J.; Milanetti, E.; Gosti, G.; Tartaglia, G.G. Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks. *Entropy* **2022**, *24*, 141.
https://doi.org/10.3390/e24020141

**AMA Style**

Monti M, Fiorentino J, Milanetti E, Gosti G, Tartaglia GG. Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks. *Entropy*. 2022; 24(2):141.
https://doi.org/10.3390/e24020141

**Chicago/Turabian Style**

Monti, Michele, Jonathan Fiorentino, Edoardo Milanetti, Giorgio Gosti, and Gian Gaetano Tartaglia. 2022. "Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks" *Entropy* 24, no. 2: 141.
https://doi.org/10.3390/e24020141