Prediction and Comparative Analysis of Software Reliability Model Based on NHPP and Deep Learning

Abstract

Over time, software has become increasingly important in various fields. If the current software is more dependent than in the past and is broken owing to large and small issues, such as coding and system errors, it is expected to cause significant damage to the entire industry. To address this problem, the field of software reliability is crucial. In the past, efforts in software reliability were made to develop models by assuming a nonhomogeneous Poisson-process model (NHPP); however, as models became more complex, there were many special cases in which models fit well. Hence, this study proposes a software reliability model using deep learning that relies on data rather than mathematical and statistical assumptions. A software reliability model based on recurrent neural networks (RNN), long short-term memory (LSTM), and gated recurrent units (GRU), which are the most basic deep and recurrent neural networks, was constructed. The dataset was divided into two, Datasets 1 and 2, which both used 80% and 90% of the entire data, respectively. Using 11 criteria, the estimated and learned results based on these datasets proved that the software reliability model using deep learning has excellent capabilities. The software reliability model using GRU showed the most satisfactory results.

1. Introduction

Over time, software has become increasingly important in various fields. In the past, software helped with simple tasks or small parts in each industry; however, it has now been developed into a form that can perform basic roles in all-inclusive roles. If the current software is more dependent than in the past and is broken owing to large and small issues, such as certain coding and system errors, it is expected to cause significant damage to the entire industry. To address this problem, studies on software reliability, which measure how effectively the software works, are constantly being conducted. Software reliability tests the ability of software to not fail for a specified period and indicates how long the software can be used without failure.

Studies on software reliability have been conducted based on the software reliability model, assuming the Markov model and the non-homogeneous Poisson process (NHPP) [1]. However, since software failures follow a Poisson distribution, most related studies have been conducted based on the software reliability model, assuming NHPP. Previous studies on software reliability began when a software reliability model based on NHPP was developed by Goel and Okumoto [2]. It was assumed that software failures occur independently and do not affect each other. Subsequently, a study was conducted on an S-shaped curve model in which the number of cumulative software failures increased along the S curve [3,4,5]. In addition, a software reliability model study, assuming incomplete debugging in which the defects found in the test phase were not corrected or removed, was also conducted [6]. By expanding on this, a study on a generalized incomplete debugging defect detection rate model was conducted [7,8,9,10,11]. The defect detection rate of each proposed model is in the form of a function. In addition, since the operating environment is different for each software, it is difficult to compare them equally. Thus, the software reliability model has been studied, considering uncertain factors in the operating environment [12,13].

As software develops over time, it becomes composed of complex and intertwined structures; therefore, studies on software reliability models are being conducted, assuming that software failures occur independently [14]. A software reliability model that assumes that previous errors will continuously affect software failures if they are not well fixed has been proposed, and a dependent software reliability model that assumes an uncertain operating environment has been studied [15,16].

As research progressed, several software reliability models studied under independent and dependent assumptions developed numerous models suitable for special cases, and generalizing them has been difficult. To address this, studies on software reliability models using deep learning, which is a nonparametric method, have been performed. Among the nonparametric software reliability models using deep learning, a study on software reliability models using deep neural networks was conducted [17], and a software reliability model using deep learning that applies failure data generated through open-source software was also proposed [18]. In addition, since software failure is a sequential data characteristic, the software reliability model using the recurrent neural network (RNN) and the long short-term memory (LSTM) was studied using this characteristic [19,20,21].

The software reliability model developed by NHPP has the problem that it fits well only in special cases because it adds special mathematical assumptions. Therefore, in this study, to solve this problem, we aim at the software reliability model utilizing deep learning among machine learning methods, which is a form that relies on given failure data. Since it is a data-dependent model, it is possible to develop a generalized software reliability model that considers all failures that occur in software without relying on special cases of software failures. In addition, in the past, software reliability models utilizing deep learning have been analyzed by applying methods, but in this study, we propose to add a deeper hidden layer of deep neural networks to the recurrent neural network series. If we develop a model that utilizes 100% of the data, it may cause an overfitting problem that only fits the trained data well. As a result, we also present the results for 80% and 90% of the data sets and the resulting predictions to prove superiority using 11 criteria comparisons. Section 2 presents the theoretical background of software reliability and deep learning. Section 3 introduces numerical examples, and finally, Section 4 presents the conclusions.

2. Software Reliability Model

2.1. Generalized Software Relability Model

Software reliability refers to the probability that the software does not cause system failure for a certain period of time under specific conditions. The reliability function used to evaluate this is

$$R\left(t\right)=P\left(T>t\right)={\int }_t^{\mathrm{\infty }}f\left(u\right)du.$$

(1)

This indicates the probability of an operation over time t. The pdf f(t) assumes the failure time or lifetime of the software to be a random variable T. We obtained the reliability function by assuming a distribution for the calculation. It was assumed that it follows an exponential distribution with parameter λ. Subsequently, we approach λ as an NHPP and propose a model in which λ is not a single constant value but a mean value function m(t) that changes over time [1]. N(t) is the number of failures up to time t and is a Poisson probability density function with parameter m(t).

$$\mathrm{P}\{N\left(t\right)=n\}=\frac{\{{m\left(t\right)\}}^n}{n!}{e}^{-m\left(t\right)},n=\mathrm{0,1},2,\cdots ,t\ge 0$$

(2)

m(t) is the integral of λ(t), which is a strength function representing the number of instantaneous failures at time t and the mean value function from 0 to time t.

$$m\left(t\right)={\int }_0^t\lambda \left(s\right)ds$$

(3)

The $m\left(t\right)$ is calculated using the relationship between the number of failures a(t) at each time point and the failure detection rate b(t).

$$\frac{dm\left(t\right)}{dt}=b\left(t\right)[a\left(t\right)-m(t)]$$

(4)

The software reliability model is developed using a differential equation, which is the basic form of the above equation. The software reliability model obtained by multiplying η in Equation (4) assumes an uncertain operating environment. Here, η is a parameter for the uncertainty of the operating environment, and N is the expected number of faults that exist in the software before testing [12].

$$\frac{dm\left(t\right)}{dt}=\eta b\left(t\right)[N-m(t)]$$

(5)

In addition, if m(t) is multiplied again in Equation (4), a software reliability model can be derived assuming dependent failures, in which software failures at the previous time point affect failures at the next time point [14]. This is expressed in the following equation:

$$\frac{dm\left(t\right)}{dt}=b\left(t\right)[a\left(t\right)-m(t)]m(t)$$

(6)

If each differential equation is arranged in terms of m(t) according to the conditions in Equations (4)–(6), a software reliability model that satisfies each condition can be derived. The developed software model is presented in Table 1. Models 1 to 6 are software reliability models obtained through the most basic form of differential equations; models 7 and 8 are those assuming uncertain operating environments; model 9 assumes dependent failures; and model 10 assumes the occurrence of dependent failures in an uncertain operating environment.

Table 1. NHPP software reliability models.

No.	Model	Mean Value Function	Note
1	Goel-Okumoto (GO) [2]	$\mathrm{m}\left(t\right)=\mathrm{a}\left(1-e^{-bt}\right)$	Concave
2	Yamada et al. (DS) [3]	$m\left(t\right)=\mathrm{a}\left(1-\left(1+\mathrm{b}\mathrm{t}\right)e^{-bt}\right)$	S-shape
3	Yamada et al. (YID) [6]	$m\left(t\right)=\mathrm{a}\left(1-e^{-bt}\right)\left(1-\frac{\alpha }{b}\right)+\alpha at$	Concave
4	Pham-Zhang (PZ) [7]	$m\left(t\right)=\frac{(\left(\mathrm{c}+\mathrm{a}\right)\left[1-e^{-bt}\right]-\left[\frac{ab}{b-\alpha }\right]\left(e^{-at}-e^{-bt}\right))}{1+\beta e^{-bt}}$	Both
5	Pham et al. (PNZ) [8]	$m\left(t\right)=\frac{\mathrm{a}\left(1-e^{-bt}\right)\left(1-\frac{\alpha }{b}\right)+\alpha at}{1+\beta e^{-bt}}$	Both
6	Teng-Pham (TP) [9]	$m\left(t\right)=\frac{a}{p-q}\left[1-{\left(\frac{\beta }{\beta +\left(\mathrm{p}-\mathrm{q}\right)\ln \left(\frac{c+e^{bt}}{c+1}\right)}\right)}^{\alpha }\right]$	S-shape
7	Chang et al. (TC) [12]	$m\left(t\right)=\mathrm{N}\left[1-{\left(\frac{\beta }{\beta +{\left(at\right)}^b}\right)}^{\alpha }\right]$	Both
8	Pham (Vtub) [13]	$m\left(t\right)=\mathrm{N}\left[1-{\left(\frac{\beta }{\beta +a^{bt}-1}\right)}^{\alpha }\right]$	S-shape
9	Kim et al. (DPF) [15]	$m\left(t\right)=\frac{a}{1+\frac{a}{h}{\left(\frac{1+c}{c+e^{bt}}\right)}^a}$	S-shape, Dependent
10	Lee et al. (UDPF) [16]	$m\left(t\right)=\mathrm{N}\left[1-{\left(\frac{\beta }{\alpha +bt-\mathrm{l}\mathrm{n}\left(bt+1\right)}\right)}^{\alpha }\right]$	S-shape Dependent

2.2. Software Reliablity Model Using Deep Learning

2.2.1. Deep Neural Network

An artificial neural network is based on the biological structure of the human brain and consists of nonlinear neurons. Based on the information received by the five senses, humans think and judge last, as information moves from one neuron to the next. The neurons (nodes) present in each layer of an artificial neural network are the most basic information-processing units. The network structure consists of three layers: input, output, and hidden layers. The input layer plays the same role as the information obtained by human beings through their five senses or thoughts, whereas the output layer refers to thoughts or judgments based on this. Conversely, the hidden layer learns to think or judge while receiving signals from the previous layer and passes them to the next layer through an activation function. A deep neural network (DNN) refers to a form in which the number of hidden layers is large in an artificial neural network [22,23]. When data is inputted from the input layer, the next hidden layer moves to the next layer through an activation function after combining the input value, weight, and bias and is transmitted to the output layer. Equation (7) expresses the passing from each layer. It consists of the product of the transformed z value (input value) in the previous layer, the weight, and the sum of the biases.

$$\begin{gathered} u_{ij}=b_i+{\displaystyle \sum }_{k=1}^j{W}_{i,k}{z}_{i-1,k} \\ {z}_{ij}=f\left(u_{ij}\right) \end{gathered}$$

(7)

For each layer, the activation function when moving to the next layer mainly uses the sigmoid, hyperbolic tangent, and ReLU functions. As the activation function used when outputting the result passed to the last output layer is a continuous variable, an identity function was used.

The final predicted value was obtained using the above process. The difference between the predicted value y and the actual value t is calculated as a loss function. To minimize the difference between these values, the function is updated to have the minimum value based on the differential value for each weight and bias.

$$Loss={\displaystyle \sum }_{i=1}^n{\left(y_i-t_i\right)}^2$$

(8)

The weight update implies that the loss function is updated by the learning rate (α) based on the partial derivative value for a specific weight using the backpropagation algorithm. The deep neural network learns through a series of processes and outputs a predicted value that approximates the actual value.

$$W_{t+1}=W_t-\mathsf{\alpha }\frac{\partial }{\partial W}Loss(W)$$

(9)

2.2.2. Recurrent Neural Network

The recurrent neural network is a model specialized for sequential data structures. It has cells that can remember the information of a previous point in the hidden layer of the most basic deep neural network. Therefore, it is suitable for time-series data because it is designed to remember information from the past and influence future events. Figure 1a shows an image of the RNN model, and the formula is shown in Equation (10) [24].

$$h_t=tanh\left(W_h{h}_{t-1}+W_x{h}_t+b\right)$$

(10)

Figure 1. (a) Structure of RNN model; (b) Structure of LSTM model; (c) Structure of GRU model.

Its network structure reflects past time; however, it has a vanishing gradient problem that prevents it from accurately reflecting the information about the time that existed a long time ago as the time point becomes longer [25]. To address this problem, LSTM, which has a structure that can learn long-term dependence, was built. It contains the same hidden layer as the RNN structure; however, its internal structure differs. In LSTM, a memory cell c exists. This cell has a forget gate that determines how much previous information is reflected, an input gate that includes new information g, which indicates how much new information is reflected, and an output gate that indicates how much previous information is transferred to the next layer. The activation functions used at this time were the sigmoid function and the hyperbolic tangent. Information about the past is delivered by passing through three gates and memory cells. Through this, the gradient loss problem can be solved. Each gate follows Equation (11). Figure 1b shows the LSTM model [26].

$$\begin{gathered} {\mathrm{f}}_t=\sigma (x_t{W}_{x,\mathrm{f}}+h_{t-1}{W}_{h,\mathrm{f}}+b_{\mathrm{f}}) \\ {g}_t=\tanh \left(x_t{W}_{x,g}+h_{t-1}{W}_{h,g}+b_g\right) \\ {i}_t=\sigma (x_t{W}_{x,i}+h_{t-1}{W}_{x,i}+b_i) \\ {o}_t=\sigma (x_t{W}_{x,o}+h_{t-1}{W}_{x,o}+b_o) \\ {c}_t={\mathrm{f}}_t\circ c_{t-1}+g_t\circ i_t \\ {h}_t=o_t\circ \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(c_t\right) \end{gathered}$$

(11)

LSTM can solve the gradient loss problem in RNN; however, when going to the next step, LSTM goes through four operation processes if only one operation is performed. Processing big data may be inefficient because the number of computations increases fourfold. Complementing this, the gated recurrent unit (GRU) transmits a value to the next layer through three calculation processes in one layer. The GRU integrates the forget and input gates in the LSTM to create an update gate. In addition, it replaces the output gate with a reset gate, which determines the amount of previous information to be forgotten. The update gate determines how long the previous information should be kept. Through Equation (12), the network learns while transferring to the next layer. Figure 1c shows an image of the GRU model [27].

$$\begin{gathered} r_t=\sigma (x_t{W}_{x,\mathrm{r}}+h_{t-1}{W}_{h,r}+b_r) \\ {z}_t=\sigma (x_t{W}_{x,z}+h_{t-1}{W}_{h,z}+h_z) \\ {g}_t=tanh(x_t{W}_{x,g}+(r_t\circ h_{t-1})W_{h,g}+b_g) \\ {h}_t=\left(1-z_t\right)\circ g_t+z_t\circ h_{t-1} \end{gathered}$$

(12)

The software reliability model proposed in this study adds a deep neural network layer to add training depth to the predicted model, which reflects the past viewpoint through the hidden layer of the recurrent neural network. It proposes a learning model that goes through a deeper hidden layer to reflect the results of the past rather than just reflecting the past time of RNN series. This allows us to propose a software reliability model that can be generalized by solving problems that are only well suited to special cases of previously developed software reliability models. Among the software reliability models using deep learning, DNNs have three hidden layers, and RNNs have two. The number of nodes in the hidden layer was configured as the number of data points. Based on the given data, the number of nodes in the model should be limited to the number of data points to avoid estimating too many parameters compared to the data. The parameter optimization method used in the software reliability model using deep learning is Adam. Adam is a deep learning optimization method that combines the momentum method, which uses the acceleration of previous training, and the RMSprop method, which has a structure where the learning rate changes with each training instead of being constant. The formula is as follows:

$$\begin{gathered} u_{t+1}={{\beta }_{1}u}_t+(1-{\beta }_1)\left(\frac{\partial Loss\left(W\right)}{\partial W}\right),s_{t+1}={{\beta }_{1}s}_t+(1-{\beta }_2){\left(\frac{\partial Loss\left(W\right)}{\partial W}\right)}^2 \\ {W}_{t+1}=W_t-\mathsf{\alpha }\frac{{\widehat{u}}_t}{\sqrt{{\widehat{s}}_t+ϵ}} \end{gathered}$$

Let β₁ and β₂ be an exponential moving average of momentum and RMSprop. After obtaining the parameters u and s of momentum and RMSprop, update the weights by the learning rate α. In this study, to optimize the hyper-parameters, we set a range of hyper-parameters and iterated to find the optimal results. We want to find the optimal value by increasing the learning rate α by 5-times units from 0.0000001 to 0.01, and utilizing β₁ and β₂ of 0.9 and 0.99, respectively. Figure 2 shows the software reliability model that combines recurrent neural network types and deep neural networks among software reliability models using deep learning.

Figure 2. Software reliability model combining DNNs and type of RNN.

3. Numerical Example

3.1. Data Information

The dataset used in this study is the software failure data from PLC4X (https://plc4x.apache.org, accessed on 7 February 2022)) and Apache Camel (http://camel.apache.org, accessed on 7 February 2022). Dataset 1 is from PLC4X, which is a set of libraries for communicating with industrial programmable logic controllers (PLCs) using a variety of protocols with a shared API. The data represent the cumulative number of failures occurring between December 2017 and January 2022. Dataset 2 is from Apache Camel, which is an open-source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data. Dataset 2 presents the cumulative number of failures between April 2011 and January 2022. Table A1 and Table A2 show datasets 1 and 2. The software uses the Apache IoTDB (Database for the Internet of Things), which is a native IoT database with high performance for data management and analysis, deployable on the edge and in the cloud. Errors, such as bugs, code modifications, and database management errors, were recorded. It records defects and failures caused by bugs, new features, improvements, and tasks while operating the Apache IoTDB and is intended to estimate and predict software reliability models. Both datasets 1 and 2 used 80% and 90% of the entire data, which were utilized as training data necessary for training and parameter estimation, respectively. The remaining 20% and 10% of the data were used to compare predictions by substituting them for the training and estimated models.

3.2. Criteria

The software reliability models utilizing NHPP and deep learning were compared using 11 criteria based on the difference between the actual and predicted values. The mean squared error (MSE) and mean absolute error (MAE) were defined as the sum of the squared distances and absolute values of the distances between the estimated and actual values, divided by the difference between the number of observations and parameters [28,29].

$$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{n-m},\mathrm{M}\mathrm{A}\mathrm{E}=\frac{{\sum }_{i=1}^n\left|\widehat{m}\left(t_i\right)-y_i\right|}{n-m}$$

(13)

The predictive ratio risk (PRR) was defined by dividing the distance between the actual and predicted values by the predicted value, and the predictive power (PP) was defined by dividing the distance between the actual and predicted values by the actual value [30].

$$\mathrm{P}\mathrm{R}\mathrm{R}={\displaystyle \sum }_{i=1}^n{\left(\frac{\widehat{m}\left(t_i\right)-y_i}{\widehat{m}\left(t_i\right)}\right)}^2,\mathrm{P}\mathrm{P}={\displaystyle \sum }_{i=1}^n{\left(\frac{\widehat{m}\left(t_i\right)-y_i}{y_i}\right)}^2$$

(14)

R² is the coefficient of determination of the regression equation and determines the explanatory power by considering the number of parameters [31].

$${\mathrm{R}}^2=1-\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y_i})}^2}$$

(15)

The predicted relative variation (PRV) is the standard deviation of the prediction bias, where the bias is $\sum _{i=1}^n\left[\frac{\widehat{m}\left(t_i\right)-y_i}{n}\right]$ and defined as [32]:

$$\mathrm{P}\mathrm{R}\mathrm{V}=\sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-\widehat{m}\left(t_i\right)-\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}\right)}^2}{n-1}}.$$

(16)

The root mean square prediction error (RMSPE) estimates the closeness with which the model predicts the observations [32]:

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{P}\mathrm{E}=\sqrt{{\mathrm{V}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}^2+{\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}^2}.$$

(17)

The mean error of prediction (MEOP) sums the absolute value of the deviation between the actual data and the estimated curve and is defined as [29]:

$$\mathrm{M}\mathrm{E}\mathrm{O}\mathrm{P}=\frac{{\sum }_{i=1}^n\left|\widehat{m}\left(t_i\right)-y_i\right|}{n-m+1}.$$

(18)

The Theil statistic (TS) is the average percentage of deviation over all periods with respect to the actual values. This is defined as follows [29]:

$$\mathrm{T}\mathrm{S}=100\ast \sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-\widehat{m}\left(t_i\right)\right)}^2}{{\sum }_{i=1}^n{y_i}^2}}\%.$$

(19)

This criterion increases the penalty when a parameter is added to the model for a small sample [33].

$$\mathrm{P}\mathrm{C}=\left(\frac{n-m}{2}\right)\mathit{log}\left(\frac{\sum _{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{n}\right)+m\left(\frac{n-1}{n-m}\right)$$

(20)

The preSSE (predicted sum squared error) refers to the sum of the differences between the predicted result value and the actual value, and the degree of estimation can be identified [34].

$$\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{S}\mathrm{S}\mathrm{E}={\displaystyle \sum }_{i=\mathrm{k}+1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2$$

(21)

Based on the above criteria, we compared the software reliability model utilizing NHPP with that using deep learning. For the same comparison, the number of criterion parameters affected by the number of parameters was set to zero. The closer R² is to 1, the better the result, and the closer it is to 0 for the other 10 criteria, the better the result. The goodness of the model fit was compared by calculating the criteria using R and MATLAB R2018b.

3.3. Results

3.3.1. Results of Dataset 1

Using Dataset 1, Table 2 shows the results of the parameter estimation of the software reliability model assuming NHPP and the structure of the software reliability model using deep learning. In addition, Dataset 1 was divided into 80% and 90%, and parameter estimation and model fitting were performed on the divided dataset. The structure of the software reliability model using deep learning consisted of three hidden layers, and the number of nodes in each layer was set according to the size of the training dataset. For the software reliability model of the recurrent neural network type, two deep neural networks were added, except for the recurrent neural network hidden layer. Here, Adam was used as the optimization method. The learning rate was set to 0.005 for the DNN and 0.0001 for the RNN, LSTM, and GRU.

Table 2. Parameter estimation and structure of model using Dataset 1.

No.	Model	80%	90%
1	GO	$\widehat{a}=1961.771$$, \widehat{b}=0.00065$	$\widehat{a}=8717.730$$, \widehat{b}=0.00015$
2	DS	$\widehat{a}=193.233$$, \widehat{b}=0.02730$	$\widehat{a}=160.532$$, \widehat{b}=0.03091$
3	YID	$\widehat{a}=97.401$$, \widehat{b}=0.00523$, $\widehat{\alpha }=0.10686$	$\widehat{a}=6.230$1$, \widehat{b}=0.05319$, $\widehat{\alpha }=0.34770$
4	PZ	$\widehat{a}=431.141$$, \widehat{b}=0.04005$$, \widehat{\alpha }=16.359$, $\widehat{\beta }=27.421$$, \widehat{c}=57.233$	$\widehat{a}=88.620$$, \widehat{b}=0.07843$$, \widehat{\alpha }=9.0099$, $\widehat{\beta }=12.248$$, \widehat{c}=2.0246$
5	PNZ	$\widehat{a}=69.646$$, \widehat{b}=0.01607$, $\widehat{\alpha }=0.07992,$ $\widehat{\beta }=1.0343$	$\widehat{a}=71.272$$, \widehat{b}=0.08070$, $\widehat{\alpha }=0.00541,$ $\widehat{\beta }=10.183$
6	TP	$\widehat{a}=72.610$$, \widehat{b}=0.06011$, $\widehat{\alpha }=1.8769$$, \widehat{\beta }=0.91909$, $\widehat{c}=13.7277$$, \widehat{p}=0.46681$$, \widehat{q}=0.09829$	$\widehat{a}=177.307,$ $\widehat{b}=0.08976$, $\widehat{\alpha }=0.90527$$, \widehat{\beta }=3.2480,$ $\widehat{c}=7.5811$$, \widehat{p}=1.5258$$, \widehat{q}=0.62709$
7	TC	$\widehat{a}=0.01628$$, \widehat{b}=1.5465$, $\widehat{\alpha }=1.1396$, $\widehat{\beta }=16.708$$, \widehat{N}=1713.568$	$\widehat{a}=0.02299$$, \widehat{b}=1.5946$$,\widehat{\alpha }=4.9402$, $\widehat{\beta }=15.240$$, \widehat{N}=233.487$
8	Vtub	$\widehat{a}=1.0018$$, \widehat{b}=1.4119$$, \widehat{\alpha }=0.18384$, $\widehat{\beta }=9.0775$$, \widehat{N}=7788.528$	$\widehat{a}=1.1086$$, \widehat{b}=1.7068$$, \widehat{\alpha }=0.00730$, $\widehat{\beta }=0.36680$$, \widehat{N}=165.881$
9	DPF	$\widehat{a}=80.628$$, \widehat{b}=0.00182$, $\widehat{c}=0.41977$, $\widehat{h}=3.1673$	$\widehat{a}=76.498$$, \widehat{b}=0.00172$, $\widehat{c}=0.23803$, $\widehat{h}=3.0477$
10	UDPF	$\widehat{b}=1.3315$, $\widehat{\alpha }=39.915,$ $\widehat{\beta }=5.4535$, $\widehat{N}=729.519$	$\widehat{b}=0.10265$. $\widehat{\alpha }=2.7233,$ $\widehat{\beta }=2.2206$, $\widehat{N}=257.005$
11	DNN	α = 0.005, hidden layers = 3, optimizer = Adam, epoch = 200
12	RNN	α = 0.0001, hidden layers = 2, optimizer = Adam, epoch = 200
13	LSTM	α = 0.0001, hidden layers = 2, optimizer = Adam, epoch = 200
14	GRU	α = 0.0001, hidden layers = 2, optimizer = Adam, epoch = 200

Table 3 lists the result values of the criteria using 80% of Dataset 1. The values for MSE, PRV, RMSPE, TS, and PC in LSTM were the lowest at 2.0069, 0.1345, 1.4168, 4.9250, and 14.5574, respectively. GRU showed the lowest values of 0.6706 and 1.1032 in PRR and PP, and the lowest values of MAE and MEOP of 1.21164 and 1.186 were observed in UDPF, respectively. The R² value was the highest for LSTM and GRU at 0.9929. In LSTM, 6 out of 10 criteria showed good results. For the predictions of 10% and 20%, the DPF model with a preSSE value of 47.645 had the lowest result among the software reliability models assuming NHPP. In contrast, most software reliability models using deep learning showed values in the 20-point range, and the LSTM model showed the smallest value of 21.570.

Table 3. Comparison of all criteria using 80% of Dataset 1.

No.	Model	MSE	MAE	PRR	PP	R2	PRV	RMSPE	MEOP	TS	PC	preSSE
1	GO	21.1902	4.0601	4.7030	11.9204	0.9254	4.2944	4.6544	3.9586	16.0037	60.5184	768.422
2	DS	3.9990	1.6435	1853.84	3.9028	0.9859	1.9503	2.0240	1.6024	6.9523	28.0021	64.894
3	YID	2.7404	1.2988	26.5655	2.4337	0.9904	1.6685	1.6768	1.2663	5.7552	20.6324	454.284
4	PZ	2.8043	1.2790	15.2300	2.6449	0.9901	1.6951	1.6965	1.2471	5.8219	21.0819	8306.771
5	PNZ	2.7172	1.2864	22.5012	2.4253	0.9904	1.6634	1.6698	1.2543	5.7308	20.4666	150,716.8
6	TP	2.7168	1.2888	18.3855	2.3926	0.9904	1.6648	1.6697	1.2566	5.7303	20.4634	7839.272
7	TC	3.1185	1.4057	221.7352	3.0482	0.9890	1.7648	1.7884	1.3705	6.1394	23.1529	224.970
8	Vtub	2.8788	1.3403	108.599	2.8416	0.9899	1.6990	1.7184	1.3068	5.8987	21.5931	455.315
9	DPF	3.1625	1.3642	0.9386	1.9608	0.9889	1.7976	1.8015	1.3301	6.1826	23.4260	47.645
10	UDPF	2.5427	1.2164	0.8600	1.2500	0.9911	1.6154	1.6154	1.1860	5.5437	19.1722	876.104
11	DNN	2.4373	1.4791	0.7005	1.1789	0.9914	0.5061	1.5633	1.4421	5.4275	18.3463	25.294
12	RNN	2.1147	1.4127	0.7380	1.2844	0.9926	0.3495	1.4553	1.3774	5.0557	15.5784	22.274
13	LSTM	2.0069	1.4104	0.8014	1.4432	0.9929	0.1345	1.4168	1.3751	4.9250	14.5574	21.570
14	GRU	2.0159	1.3983	0.6706	1.1032	0.9929	0.2493	1.4204	1.3634	4.9362	14.6454	23.872

Table 4 lists the result values of the criteria using 90% of Dataset 1. MSE, RMSPE, TS, and PC were the lowest in GRU at 1.9961, 0.9948, 1.413, 4.1991, and 16.1831, respectively, and PRV was the smallest in LSTM at 0.1251. In addition, PRR and PP in UDPF showed the lowest values of 0.6386 and 0.9113, and MAE and MEOP were 1.3714 and 1.3409 in DPF, respectively. The R² value was the largest at 0.9948. In GRU, 5 out of 10 criteria showed good results. In preSSE, the TC, DS, and Vtub models showed very small values of 6.367, 6.537, and 7.340, respectively. In the software reliability model using deep learning, RNN showed the smallest value of 11.397, followed by LSTM at 11.755. The resulting values showed good results for the software reliability model, assuming NHPP.

Table 4. Comparison of all criteria using 90% of Dataset 1.

No.	Model	MSE	MAE	PRR	PP	R2	PRV	RMSPE	MEOP	TS	PC	preSSE
1	GO	22.6581	4.3155	4.7797	13.4021	0.9407	4.4490	4.8071	4.2196	14.1477	69.6287	321.655
2	DS	3.9569	1.6600	1636.27	3.7881	0.9896	1.9677	2.0112	1.6231	5.9122	31.2376	6.537
3	YID	3.5588	1.5488	58.9754	2.7164	0.9907	1.9001	1.9081	1.5144	5.6069	28.9044	11.837
4	PZ	3.1010	1.4570	31.2349	2.4008	0.9919	1.7750	1.7812	1.4246	5.2339	25.8753	2395.537
5	PNZ	3.1296	1.4638	27.1085	2.3403	0.9918	1.7798	1.7893	1.4313	5.2580	26.0772	40.741
6	TP	3.2108	1.4898	26.6591	2.3619	0.9916	1.8069	1.8125	1.4567	5.3257	26.6407	4948.058
7	TC	3.6277	1.5687	262.6646	3.1326	0.9905	1.9117	1.9263	1.5338	5.6609	29.3263	6.367
8	Vtub	3.4308	1.5331	82.7369	2.6383	0.9910	1.8662	1.8735	1.4990	5.5051	28.0984	7.340
9	DPF	3.0682	1.3714	0.8756	1.7908	0.9920	1.7676	1.7718	1.3409	5.2061	25.6412	65.275
10	UDPF	2.9984	1.3889	0.6386	0.9113	0.9922	1.7515	1.7516	1.3581	5.1465	25.1349	1709.197
11	DNN	2.5868	1.5030	0.7327	1.3150	0.9932	0.5792	1.6107	1.4696	4.7803	21.8866	13.471
12	RNN	2.0731	1.4332	0.8317	1.5417	0.9946	0.1402	1.4400	1.4013	4.2794	17.0167	11.397
13	LSTM	2.0257	1.4179	0.7584	1.3192	0.9947	0.1251	1.4234	1.3864	4.2302	16.5077	11.755
14	GRU	1.9961	1.4061	0.7593	1.331	0.9948	0.1392	1.413	1.3749	4.1991	16.1831	12.103

3.3.2. Results of Dataset 2

Using Dataset 2, Table 5 shows the results of the parameter estimation of the software reliability model assuming NHPP and the structure of the software reliability model using deep learning. Similar to Dataset 1, Dataset 2 was divided into 80% and 90%, and parameter estimation and model fitting were performed on the divided dataset. The structure of the software reliability model using deep learning consists of three hidden layers, as in Dataset 1, and the number of nodes in each layer was set according to the size of the training dataset. The learning rates were set to 0.000001 for DNN and 0.00001 for RNN, LSTM, and GRU.

Table 5. Parameter estimation and structure of model using Dataset 2.

No.	Model	80%	90%
1	GO	$\widehat{a}=15383.72$$, \widehat{b}=0.00010$	$\widehat{a}=37605.84$$, \widehat{b}=0.00005$
2	DS	$\widehat{a}=13332.14$$, \widehat{b}=0.00182$	$\widehat{a}=53880.83$$, \widehat{b}=0.00091$
3	YID	$\widehat{a}=49.407$$, \widehat{b}=0.00121$, $\widehat{\alpha }=0.66627$	$\widehat{a}=10.855$$, \widehat{b}=0.00145$, $\widehat{\alpha }=2.7890$
4	PZ	$\widehat{a}=564.590$, $\widehat{b}=0.2938$$, \widehat{\alpha }=0.35523$, $\widehat{\beta }=31.536$$, \widehat{c}=0.04628$	$\widehat{a}=3566157.77$$, \widehat{b}=0.02095$$, \widehat{\alpha }=0.14865$, $\widehat{\beta }=119107.58$$, \widehat{c}=3567.64$
5	PNZ	$\widehat{a}=166.155$, $\widehat{b}=0.01033$, $\widehat{\alpha }=0.06270,$ $\widehat{\beta }=4.2225$	$\widehat{a}=284.863$$, \widehat{b}=0.01104$, $\widehat{\alpha }=0.07397,$ $\widehat{\beta }=11.590$
6	TP	$\widehat{a}=4.1953$$, \widehat{b}=0.08837$, $\widehat{\alpha }=7.3884$$, \widehat{\beta }=2.8696$, $\widehat{c}=1.6579$$, \widehat{p}=0.04236$$, \widehat{q}=0.10460$	$\widehat{a}=49.417,$ $\widehat{b}=0.06702$, $\widehat{\alpha }=0.01198$$, \widehat{\beta }=0.02904,$ $\widehat{c}=2.4430$$, \widehat{p}=0.20147$$, \widehat{q}=0.20537$
7	TC	$\widehat{a}=0.01344$,$\widehat{b}=1.9930$, $\widehat{\alpha }=0.00754$, $\widehat{\beta }=0.39594$, $\widehat{N}=2857.101$	$\widehat{a}=0.00307$$, \widehat{b}=2.22599$, $\widehat{\alpha }=4.7647$, $\widehat{\beta }=118.747$$, \widehat{N}=73382.23$
8	Vtub	$\widehat{a}=1.0005$$, \widehat{b}=2.2792$$, \widehat{\alpha }=0.00754$, $\widehat{\beta }=0.39594$, $\widehat{N}=1562.143$	$\widehat{a}=1.0128$$, \widehat{b}=1.1078$$, \widehat{\alpha }=16.018$, $\widehat{\beta }=1060.760$, $\widehat{N}=2049.573$
9	DPF	$\widehat{a}=322.915$$, \widehat{b}=0.00018$, $\widehat{c}=0.33127$, $\widehat{h}=6.4475$	$\widehat{a}=994.419$$, \widehat{b}=0.00005$, $\widehat{c}=0.57131$, $\widehat{h}=10.012$
10	UDPF	$\widehat{b}=652.600$, $\widehat{\alpha }=21497.41,$ $\widehat{\beta }=8.9936$, $\widehat{N}=1804.06$	$\widehat{b}=13.526$, $\widehat{\alpha }=2439.458,$ $\widehat{\beta }=10.564$, $\widehat{N}=192350.3$
11	DNN	α =0.000001, hidden layers = 3, optimizer = Adam, epoch = 200
12	RNN	α = 0.00001, hidden layers = 2, optimizer = Adam, epoch = 200
13	LSTM	α = 0.00001, hidden layers = 2, optimizer = Adam, epoch = 200
14	GRU	α = 0.00001, hidden layers = 2, optimizer = Adam, epoch = 200

Table 6 shows the result values of the criteria using 80% of Dataset 2. The values of MSE, MAE, PRV, RMSPE, MEOP, TS, and PC in GRU were the lowest at 4.5494, 2.0688, 0.5218, 2.1336, 2.0489, 2.2311, and 79.0128, respectively, and R² of 0.9988 in GRU and LSTM appeared the highest. PRR showed the lowest value of 1.5342 in DNN, and PP showed the smallest value of 1.7716 in TP. In GRU, 8 out of 10 criteria showed good results. Among the software reliability models assuming NHPP, the model with the lowest result was the Vtub model, with a preSSE value of 159,551.23. In contrast, most software reliability models using deep learning showed smaller values than the existing software reliability models; among them, the RNN model showed the smallest value of 71.06.

Table 6. Comparison of all criteria using 80% of Dataset 2.

No.	Model	MSE	MAE	PRR	PP	R2	PRV	RMSPE	MEOP	TS	PC	preSSE
1	GO	514.6996	20.0326	17.3582	100.887	0.8664	21.0794	22.7819	19.8400	23.7311	322.5348	735,386.2
2	DS	13.7076	3.0740	2256.700	8.8015	0.9964	3.6874	3.7202	3.0445	3.8728	135.8146	202,541.7
3	YID	12.1406	2.8671	215.3384	7.1101	0.9968	3.4643	3.5010	2.8395	3.6447	129.5631	184,160.8
4	PZ	10.0437	2.5378	3299.707	9.6331	0.9974	3.1828	3.1847	2.5134	3.3150	119.7981	174,920.2
5	PNZ	9.6155	2.4483	7.2913	1.9345	0.9975	3.1155	3.1160	2.4248	3.2436	117.5539	1,643,848
6	TP	11.0679	2.6448	5.4184	1.7716	0.9971	3.3366	3.3430	2.6194	3.4800	124.7987	2,037,721
7	TC	13.4688	3.0404	2453.543	9.2252	0.9965	3.6077	3.6872	3.0112	3.8389	134.9098	189,842.4
8	Vtub	10.7640	2.7480	5303.130	7.7007	0.9972	3.2707	3.2966	2.7216	3.4318	123.3649	159,551.2
9	DPF	16.6392	3.3789	5.4085	88.826	0.9957	4.0398	4.0985	3.3464	4.2668	145.7959	305,246.7
10	UDPF	22.5801	3.8694	99.4808	9.7126	0.9941	4.6880	4.7742	3.8322	4.9705	161.5192	274,796.2
11	DNN	5.8181	2.1792	1.5342	4.0310	0.9985	1.0391	2.4143	2.1583	2.5231	91.6808	355.63
12	RNN	4.8223	2.0739	2.1101	9.9452	0.9987	0.7255	2.1971	2.0539	2.2970	82.0123	71.06
13	LSTM	4.6712	2.0805	2.3802	16.1958	0.9988	0.5882	2.1621	2.0605	2.2608	80.3730	290.41
14	GRU	4.5494	2.0688	2.1001	9.9841	0.9988	0.5218	2.1336	2.0489	2.2311	79.0128	347.92

Table 7 shows the result values for the criteria using 90% of Dataset 2. MSE, PRV, RMSPE, TS, and PC showed the smallest values of 8.8655, 0.9189, 2.9787, 2.3336, and 1237.5573, respectively, in GRU, and R² showed the largest value of 0.9988 in the same model. MAE and MEOP had the smallest values of 2.8168 and 2.7928, respectively, in RNN. PRR was 1.6236 in DNN, and PP was the smallest at 1.4301 in Vtub. In GRU, 6 out of 10 criteria showed good results. Regarding the preSSE value, the GRU model showed the smallest value of 468.03 among the software reliability models using deep learning.

Table 7. Comparison of all criteria using 90% of Dataset 2.

No.	Model	MSE	MAE	PRR	PP	R2	PRV	RMSPE	MEOP	TS	PC	preSSE
1	GO	1278.733	30.0077	23.6240	173.2001	0.8256	33.3020	35.8928	29.7512	28.0264	415.9016	426,981.8
2	DS	98.7794	7.0277	2204.006	8.9742	0.9865	9.8441	9.9807	6.9676	7.7895	267.3789	114,300.6
3	YID	97.1781	6.9453	840.7242	8.4733	0.9867	9.7596	9.8995	6.8859	7.7261	266.4310	113,140.1
4	PZ	27.9916	4.2381	677.6398	67.5837	0.9962	5.3037	5.3136	4.2019	4.1466	194.2418	26,776.2
5	PNZ	39.4936	4.7960	18.9956	3.8443	0.9946	6.2657	6.3113	4.7550	4.9254	214.2075	1,165,855
6	TP	13.6426	2.9286	4.0108	1.6448	0.9981	3.6944	3.7095	2.9035	2.8948	152.5569	1,390,597
7	TC	64.3763	6.6342	19,228.33	14.4663	0.9912	7.8678	8.0567	6.5775	6.2884	242.5466	65,012.80
8	Vtub	26.1124	3.8598	3.7169	1.4301	0.9964	5.1293	5.1322	3.8268	4.0050	190.2112	23,868.05
9	DPF	50.3622	6.2521	9.2044	251.4521	0.9931	7.0064	7.1264	6.1987	5.5620	228.3074	28,590.91
10	UDPF	36.7890	4.7296	4.2437	47.5652	0.9950	6.0896	6.0917	4.6891	4.7537	210.0929	33,114.75
11	DNN	11.4930	2.8455	1.6236	4.2855	0.9984	1.8509	3.3945	2.8211	2.6570	142.612	680.76
12	RNN	10.3925	2.8168	2.9032	22.1128	0.9986	1.5746	3.2271	2.7928	2.5266	136.7744	1122.65
13	LSTM	11.7785	2.8708	2.9508	40.5778	0.9984	1.8888	3.4365	2.8463	2.6898	144.0352	1251.45
14	GRU	8.8655	2.8335	2.8749	19.0924	0.9988	0.9189	2.9787	2.8092	2.3336	127.5573	468.03

Figure 3 and Figure 4 are graphs that combine the estimated and trained values for 80% and 90% of the data above with the predicted values for the remaining 20% and 10%. The points before the black line are estimated values, and the points after the black line are predicted values.

Figure 3. (a) Estimated and predicted value for 80% of Dataset 1; (b) Estimated and predicted value for 90% of Dataset 1.

Figure 4. (a) Estimated and predicted value for 80% of Dataset 2; (b) Estimated and predicted value for 90% of Dataset 2.

3.4. Confidence Interval

In this study, GRU, the software reliability model that showed the best results among the proposed models, was used to obtain the confidence interval. In addition, the difference in the distribution between the predicted and actual data values of the other models was compared. The confidence interval follows Equation (22). Since software failures follow a Poisson distribution, we take the mean and variance as $\widehat{m}\left(t\right)$. $z_{\alpha }$ is defined as the 100($1-\alpha$)/2 percentile of the standard normal distribution [35]. A 95% confidence interval was used.

$$\widehat{m}\left(t\right)+z_{\alpha }\sqrt{\widehat{m}\left(t\right)}$$

(22)

Table A3 and Table A4 show the predicted values and corresponding confidence intervals for the remaining 20% and 10% of the data based on the estimated and trained models for 80% and 90% of Dataset 1. The bolded values indicate that the actual data values do not fall within the prediction intervals. For 80% of Dataset 1, the other NHPP software reliability models, except for PNZ, PZ, and Vtub, and the software reliability models using deep learning, predicted well within the 95% confidence interval. For 90% of Dataset 1, the other NHPP software reliability models (except PNZ, PZ, TP, and UDPF) and the software reliability models using deep learning performed well within the 95% confidence interval.

Table A5 and Table A6 show the predicted values and corresponding confidence intervals for the remaining 20% and 10% of the data based on the estimated and trained models for 80% and 90% of Dataset 2. For 80% of Dataset 2, the GO, PNZ, and TP models showed results where the actual data was not within the prediction confidence interval at all, while the DS model showed results where the actual data was within the 95% prediction confidence interval up to the 108th time, the YID, TC, DPF, and UDPF models up to the 109th time, and the PZ, Vtub model up to the 110th time. For 90% of Dataset 2, the NHPP software reliability models showed that the PZ, Vtub, DPF, and UDPF models resulted in the actual data falling within the 95% prediction confidence interval up to the 118th time. Except for the PZ, Vtub, DPF, and UDPF models, none of the NHPP software reliability models had the actual data fall within the predicted confidence interval. The software reliability models using deep learning performed well, with predictions within the 95% confidence interval. Except for the models that do not fall within the 95% confidence intervals shown in Table A3, Table A4, Table A5 and Table A6, the remaining models showed good results in following the data trend, and the software reliability models using deep learning showed good estimation and prediction results in following the data trend.

4. Conclusions

In this study, the software reliability models using NHPP and deep learning were compared. Regarding the software reliability model using deep learning, a model composed of deep neural networks and recurrent neural networks (RNN, LSTM, and GRU) that have been applied to time-series data characteristics was used. It was constructed by including the hidden layer of the neural network. Using Datasets 1 and 2, it was confirmed that this model showed better estimation and predictive power than existing software reliability models. Among them, the software reliability of the recurrent neural network series showed better results, and as a result of fitting with 80% and 90% datasets, the software reliability model with the deep neural network added to the GRU showed the best results. In addition, the software reliability model of the recurrent neural network showed good results when the predicted values were compared with the remaining datasets. The NHPP software reliability model showed results that relatively fit the data well with a small number of time points; however, it showed a significant increase as the number of time points increased. On the other hand, the software neural network using deep learning showed results that fit the data trend well, even when the time point was large.

However, software reliability models using deep learning always face the problem of overfitting. Therefore, to address this, we plan to conduct research on software reliability models based on mathematical and statistical assumptions by grafting them together with the NHPP software reliability models. Through this, we aim to create a stable, data-dependent, and complementary model with a more mathematical basis.