A Software Reliability Model Considering a Scale Parameter of the Uncertainty and a New Criterion

Abstract

It is becoming increasingly common for software to operate in various environments. However, even if the software performs well in the test phase, uncertain operating environments may cause new software failures. Traditional proposed software reliability models under uncertain operating environments suffer from the problem of being well-suited to special cases due to the large number of assumptions involved. To improve these problems, this study proposes a new software reliability model that assumes an uncertain operating environment. The new software reliability model is a model that minimizes assumptions and minimizes the number of parameters that make up the model, so that the model can be applied to general situations better than the traditional proposed software reliability models. In addition, various criteria based on the difference between the predicted and estimated values have been used in the past to demonstrate the superiority of the software reliability models. Also, we propose a new multi-criteria decision method that can simultaneously consider multiple goodness-of-fit criteria. The multi-criteria decision method using ranking is useful for comprehensive evaluation because it does not rely on individual criteria alone by ranking and weighting multiple criteria for the model. Based on this, 21 existing models are compared with the proposed model using two datasets, and the proposed model is found to be superior for both datasets using 15 criteria and the multi-criteria decision method using ranking.

1. Introduction

Software is a set of programs organized by an algorithm to perform a specific task. With the development of technology, many fields have become increasingly dependent on software. As software is used in many areas, there has been significant interest in its reliability. Software reliability is a measurement of how long software can be used without failure. Because software is utilized in many fields, if it fails or is unreliable, it will cause significant problems, highlighting the importance of software reliability.

Considerable research has been conducted on software reliability. Various methods have been used to estimate and predict software failures, such as error-seeding models, failure rate models, curve-fitting models, time series models, and nonhomogeneous Poisson processes (NHPPs). The NHPP software reliability model (SRM) is the most representative of these research methodologies. The NHPP SRM was developed by Goel and Okumoto in 1979. They proposed an SRM assuming that the occurrence of failures increases exponentially over time [1,2]. Based on this, an SRM that assumes that the cumulative failure of software increases in an S-shaped curve was developed [3,4,5,6], and research on SRMs that reflects the various efforts put into the testing phase to measure software reliability was conducted. In addition, SRMs that assume incomplete debugging, that is, where failures and defects that occur may not be corrected, have been proposed [7], and many SRMs that assume generalized incomplete debugging have been studied [8,9,10,11,12].

Software structures have become increasingly diverse and complex because they are actively used in many industries. Because of the complex organization of software, the failures that occur are not independent, and software failures can have dependent effects on other software failures [13,14]. Kim et al. [15] and Lee et al. [16] proposed an SRM that assumed that software failures occur in a dependent manner.

Because software is used in many different ways, its roles have become very diverse, and the environment provided to each piece of software has also become very diverse. Many studies have been conducted on SRMs that assume uncertainty in operating environments [17,18,19,20,21,22,23,24,25,26]. Here, uncertain operating environments refer to the environments used by actual consumers. To assume uncertain operating environments, Teng and Pham [18] and Pham [19] proposed an SRM utilizing a gamma distribution, and Song et al. [20] assumed the uncertain operating environments to be exponential and applied the failure detection rate function as a monotonically increasing function to present a new model for uncertain operating environments. Many researchers have proposed an SRM that considers uncertain factors [21,22,23,24]. Chang et al. [25] proposed an SRM that considers testing coverage, which indicates whether sufficient tests have been performed in uncertain operating environments. Lee et al. [26] proposed an uncertain operating environment SRM that assumes dependent failures to reflect a complex software structure.

In this study, we propose a new NHPP SRM that considers the uncertain operating environments of software. Traditional NHPP SRMs that consider uncertain operating environments include several mathematical assumptions, which make the model very complicated and suffer from the problem that it only fits well in special cases with assumptions. Our proposed new NHPP SRM minimizes the mathematical assumptions and can be better applied to general situations than traditional SRMs. In addition, while past software reliability studies judged the excellence of a model based on various criteria such as the difference between the predicted and estimated values of the developed SRM, we propose a multi-criteria decision method using ranking (MCDMR) that considers multiple criteria simultaneously through ranking. Section 2 introduces the SRM and the new NHPP model that considers uncertain operating environments. Section 3 introduces integrated criteria enabling the consideration of multiple criteria simultaneously, and Section 4 explains the numerical example results. Finally, Section 5 concludes this paper.

2. A New Software Reliability Model

2.1. Software Reliability Model

Software reliability assesses how long a piece of software can be used without failure. The reliability function $\mathrm{R}\left(\mathrm{t}\right)$ used to evaluate the software reliability is given by the following equation:

$$\mathrm{R}\left(\mathrm{t}\right)=\mathrm{P}(\mathrm{T}>\mathrm{t})={\int }_{\mathrm{t}}^{\mathrm{\infty }}\mathrm{f}\left(\mathrm{u}\right)\mathrm{d}\mathrm{u}$$

It is defined as the probability that the system is still operating after time $\mathrm{t}$. The reliability function $\mathrm{R}\left(\mathrm{t}\right)$ is derived from the probability density function $\mathrm{f}\left(\mathrm{t}\right)$ by assuming the lifetime or failure time to be a random variable $\mathrm{T}$. To calculate the reliability function $\mathrm{R}\left(\mathrm{t}\right)$, the number of failures per unit time is assumed to be Poisson distributed with mean $\mathsf{\lambda }$. However, the mean number of failures per unit time is a constant and does not depend on time. To improve this, many traditional SRMs follow a non-homogeneous Poisson process (NHPP) with a time-dependent mean value function $\mathrm{m}\left(\mathrm{t}\right)$ per unit time. The NHPP is represented by the following equation:

$$\mathrm{Pr}\{\mathrm{N}\left(\mathrm{t}\right)=\mathrm{n}\}=\frac{\{{\mathrm{m}\left(\mathrm{t}\right)\}}^{\mathrm{n}}}{\mathrm{n}!}{\mathrm{e}}^{-\mathrm{m}\left(\mathrm{t}\right)}, \mathrm{n}=0,1,2,\cdots ,\mathrm{t}\ge 0$$

where $\mathrm{N}\left(\mathrm{t}\right)$ is the total number of events in a given time and $\mathrm{m}\left(\mathrm{t}\right)$ is the mean value function from time $0$ to time $\mathrm{t}$. The $\mathrm{m}\left(\mathrm{t}\right)$ is derived as follows:

$$m\left(\mathrm{t}\right)={\int }_0^{\mathrm{t}}\mathsf{\lambda }\left(\mathrm{s}\right)\mathrm{d}\mathrm{s}$$

$\mathrm{m}\left(\mathrm{t}\right)$ is the integral of the intensity function $\mathsf{\lambda }\left(\mathrm{t}\right)$, which is the instantaneous number of failures at time $\mathrm{t}$, and the mean number of failures from time $0$ to time $\mathrm{t}$. From this relationship, the reliability function $\mathrm{R}\left(\mathrm{t}\right)$ is derived as follows.

$$\mathrm{R}\left(\mathrm{t}\right)={\mathrm{e}}^{-{\int }_0^{\mathrm{t}}\mathsf{\lambda }\left(\mathrm{s}\right)\mathrm{d}\mathrm{s}}={\mathrm{e}}^{-\mathrm{m}\left(\mathrm{t}\right)}$$

The reliability function $\mathrm{R}\left(\mathrm{t}\right)$ is calculated from the intensity function $\mathsf{\lambda }\left(\mathrm{t}\right)$ and $\mathrm{m}\left(\mathrm{t}\right)$ to improve software reliability.

2.2. Proposed NHPP SRM

The mean value function $\mathrm{m}\left(\mathrm{t}\right)$ through the NHPP is calculated as a differential equation, where $\mathrm{a}\left(\mathrm{t}\right)$ is the number of failures at each time point and $\mathrm{b}\left(\mathrm{t}\right)$ is the failure detection rate. The most basic differential equation is shown in Equation (1).

$$\frac{\mathrm{d}\mathrm{m}(\mathrm{t})}{\mathrm{d}\mathrm{t}}=\mathrm{b}\left(\mathrm{t}\right)[\mathrm{a}\left(\mathrm{t}\right)-\mathrm{m}(\mathrm{t})]$$

(1)

It assumes that the software has the same operating environment and that software failures occur independently. Traditional SRMs define $\mathrm{a}\left(\mathrm{t}\right)$ and $\mathrm{b}\left(\mathrm{t}\right)$ as in Equation (1) and solve the differential equation to obtain the SRM in Equation (2).

$$\mathrm{m}(\mathrm{t})=e^{-{\int }_{t_0}^t\mathrm{b}\left(\tau \right)d\tau }[m_0+{\int }_{t_0}^t\mathrm{a}\left(\tau \right)b\left(\tau \right)e^{-{\int }_{t_0}^t\mathrm{b}\left(\tau \right)d\tau }d\tau ]$$

(2)

where $m_0$ is the initial condition, and $t_0$ is the start time of the debugging process.

However, the environment in which the software operates varies significantly depending on the function or environment of the software. For example, the operating system and computer specifications of users of the developed software vary widely, but it is not practical for companies to release tests that consider all situations; therefore, the errors that may occur in different environments vary widely. Therefore, considering the uncertainty of the environment in which the software is operated can increase software reliability. In this study, we propose an NHPP SRM that considers uncertain operating environments. Furthermore, our proposed NHPP SRM that considers uncertain operating environments follows the following assumptions.

⯀

The initial condition of the $\mathrm{m}\left(\mathrm{t}\right)$ is $\mathrm{m}\left(0\right)=0;$

⯀

$\mathrm{a}\left(\mathrm{t}\right)=\mathrm{N}$ is the expected number of software faults before testing;

⯀

$\mathsf{\eta }$ follows a gamma distribution with $\mathsf{\alpha }$ and $\mathsf{\beta }$;

⯀

$\mathsf{\beta }$ is a parameter containing the failure detection rate.

Our proposed SRM considers uncertain operating environments by extending the differential equations that derive the traditional SRM to assume uncertain operating environments. The differential equation for the model is derived by adding the uncertain operating environment parameter $\mathsf{\eta }$, as in Equation (3):

$$\frac{\mathrm{d}\mathrm{m}\left(\mathrm{t}\right)}{\mathrm{d}\mathrm{t}}=\mathsf{\eta }\mathrm{b}\left(\mathrm{t}\right)\left[\mathrm{a}\left(\mathrm{t}\right)-\mathrm{m}\left(\mathrm{t}\right)\right]$$

(3)

where the parameter $\mathsf{\eta }$ follows a gamma distribution with parameters $\mathsf{\alpha }$ and $\mathsf{\beta }$, and the scale parameter, $\mathsf{\beta }$, contains the fault detection rate of the software.

The proposed model uses $\mathsf{\alpha }$ and $\mathsf{\beta }$ drawn from the gamma distribution rather than the other parameters that make up the traditional $\mathrm{a}\left(\mathrm{t}\right)$ and $\mathrm{b}\left(\mathrm{t}\right)$. The parameter $\mathsf{\eta }$ in Equation (3) follows a gamma distribution with parameters $\mathsf{\alpha }$ and $\mathsf{\beta }$, where $\mathrm{a}\left(\mathrm{t}\right)$ is $\mathrm{a}\left(\mathrm{t}\right)=\mathrm{N}$ and $\mathrm{b}\left(\mathrm{t}\right)$ is $\mathrm{b}\left(\mathrm{t}\right)=\frac{{\mathsf{\beta }}^2\mathrm{t}}{\mathsf{\beta }\mathrm{t}+1}$. $\mathrm{N}$ is the total expected number of failures, and $\mathsf{\alpha }$ and $\mathsf{\beta }$ are the shape and scale parameters for parameter $\mathsf{\eta }$ in uncertain operating environments. Substituting this into Equation (3) gives Equation (4):

$$\mathrm{m}\left(\mathrm{t}\right)={\mathrm{N}\left(1-\frac{\mathsf{\beta }}{\mathsf{\alpha }+{\int }_0^{\mathrm{t}}\mathrm{b}\left(\mathrm{s}\right)\mathrm{d}\mathrm{s}}\right)}^{\mathsf{\alpha }}$$

(4)

In Equation (4), because $\mathrm{b}\left(\mathrm{t}\right)$ is $\mathrm{b}\left(\mathrm{t}\right)=\frac{{\mathsf{\beta }}^2\mathrm{t}}{\mathsf{\beta }\mathrm{t}+1}$, through substitution and summary, the NHPP SRM assumes uncertain operating environments expressed in the final form of Equation (5). Traditional NHPP SRMs for uncertain operating environments have proposed models with multiple assumptions, often consisting of four or more parameters [18,19,20,25,26]. However, these models make a number of assumptions that may be appropriate for particular situations. The proposed NHPP SRM contains fewer parameters than the previously proposed uncertain operating environment SRM. It does not contain many assumptions and can be judged as a model for general situations rather than for special situations.

$$\mathrm{m}\left(\mathrm{t}\right)={\mathrm{N}\left(1-\frac{\mathsf{\beta }}{\mathsf{\alpha }+{\int }_0^{\mathrm{t}}\frac{{\mathsf{\beta }}^2\mathrm{t}}{\mathsf{\beta }\mathrm{t}+1}\mathrm{d}\mathrm{s}}\right)}^{\mathsf{\alpha }}={\mathrm{N}\left(1-\frac{\mathsf{\beta }}{\mathsf{\alpha }+\mathsf{\beta }\mathrm{t}-\mathrm{l}\mathrm{n}\left(\mathsf{\beta }\mathrm{t}+1\right)}\right)}^{\mathsf{\alpha }}$$

(5)

Table 1. SRMs.

No.	Model	Mean Value Function	Note
1	Goel-Okumoto (GO) [1]	$\mathrm{m}\left(t\right)=\mathrm{a}\left(1-e^{-bt}\right)$	Concave
2	Hossain-Dahiya (HDGO) [2]	$m\left(t\right)=\log \left[\frac{\left(e^a-\mathrm{c}\right)}{\left(e^{a{e}^{-bt}}-\mathrm{c}\right)}\right]$	Concave
3	Yamada et al. (DS) [3]	$m\left(t\right)=\mathrm{a}\left(1-\left(1+\mathrm{b}\mathrm{t}\right)e^{-bt}\right)$	S-Shape
4	Ohba (IS) [4]	$m\left(t\right)=\frac{\mathrm{a}\left(t1-e^{-bt}\right)}{1+\beta e^{-bt}}$	S-Shape
5	Yamada et al. (YE) [6]	$m\left(t\right)=\mathrm{a}\left(1-e^{-\gamma \alpha \left(1-e^{-\beta t}\right)}\right)$	Concave
6	Yamada et al. (YR) [6]	$m\left(t\right)=\mathrm{a}\left(1-e^{-\gamma \alpha \left(1-e^{-\beta t^2/2}\right)}\right)$	S-Shape
7	Yamada et al. (YID 1) [7]	$m\left(t\right)=\frac{\mathrm{a}b}{\alpha +b}\left(e^{\alpha t}-e^{-bt}\right)$	Concave
8	Yamada et al. (YID 2) [7]	$m\left(t\right)=\mathrm{a}\left(1-e^{-bt}\right)\left(1-\frac{\alpha }{b}\right)+\alpha at$	Concave
9	Pham-Zhang (PZ) [8]	$m\left(t\right)=\frac{(\left(\mathrm{c}+\mathrm{a}\right)\left[1-e^{-bt}\right]-\left[\frac{ab}{b-\alpha }\right]\left(e^{-at}-e^{-bt}\right))}{1+\beta e^{-bt}}$	Both
10	Pham et al. (PNZ) [9]	$m\left(t\right)=\frac{\mathrm{a}\left(1-e^{-bt}\right)\left(1-\frac{\alpha }{b}\right)+\alpha at}{1+\beta e^{-bt}}$	Both
11	Pham (IFD) [11]	$m\left(t\right)=\mathrm{a}\left(1-e^{-bt}\right)\left(1+\left(\mathrm{b}+\mathrm{d}\right)t+bd{t}^2\right)$	Concave
12	Roy et al. (RMD) [12]	$m\left(t\right)=a\alpha \left[1-e^{-bt}\right]-\left[\frac{ab}{b-\beta }\left(e^{-\beta t}-e^{-bt}\right)\right]$	Concave
13	Zhang et al. (ZFR) [5]	$m\left(t\right)=\frac{a}{p-\beta }\left[1-{\left(\frac{\left(1+\alpha \right)e^{-bt}}{1+{\alpha e}^{-bt}}\right)}^{\frac{c}{b}\left(p-\beta \right)}\right]$	S-Shape
14	Kapur et al. (KSRGM) [10]	$m\left(t\right)=\frac{A}{1-\alpha }\left[1-{\left(\left(1+bt+\frac{b^2{t}^2}{2}\right)e^{-bt}\right)}^{p(1-\alpha )}\right]$	S-Shape
15	Chang et al. (TC) [25]	$m\left(t\right)=\mathrm{N}\left[1-{\left(\frac{\beta }{\beta +{\left(at\right)}^b}\right)}^{\alpha }\right]$	Both
16	Teng-Pham (TP) [18]	$m\left(t\right)=\frac{a}{p-q}\left[1-{\left(\frac{\beta }{\beta +\left(\mathrm{p}-\mathrm{q}\right)\ln \left(\frac{c+e^{bt}}{c+1}\right)}\right)}^{\alpha }\right]$	S-Shape
17	Song et al. (3P) [20]	$m\left(t\right)=\mathrm{N}\left[1-\left(\frac{\beta }{\beta -\frac{a}{b}\ln \left(\frac{\left(1+c\right)e^{-bt}}{1+c{e}^{-bt}}\right)}\right)\right]$	S-Shape
18	Pham (Vtub) [19]	$m\left(t\right)=\mathrm{N}\left[1-{\left(\frac{\beta }{\beta +a^{bt}-1}\right)}^{\alpha }\right]$	S-Shape
19	Kim et al. (DPF1) [15]	$m\left(t\right)=\frac{a}{1+\frac{a}{h}{\left(\frac{1+c}{c+e^{bt}}\right)}^a}$	Concave, Dependent
20	Lee et al. (DPF2) [16]	$m\left(t\right)=\frac{a}{1+\frac{a}{h}{\left(\frac{b+c}{c+b{e}^{bt}}\right)}^{\frac{a}{b}}}$	Concave, Dependent
21	Lee et al. (UOP) [26]	$m\left(t\right)={N\left(1-\frac{\beta }{\alpha +\frac{c}{b}\mathrm{l}\mathrm{n}\left(\frac{\alpha +e^{bt}}{1+\alpha }\right)}\right)}^{\alpha }$	S-Shape, Dependent
22	Proposed Model	$m\left(t\right)={N\left(1-\frac{\beta }{\alpha +\beta t-\mathrm{l}\mathrm{n}\left(\beta t+1\right)}\right)}^{\alpha }$	S-Shape

shows the mean value functions of the traditional SRM and the proposed model. Models 1–14 are SRMs with independent assumptions, models 15–18 are SRMs considering uncertain operating environments, models 19 and 20 are SRMs considering dependent failures, model 21 is an SRM considering dependent failure models and uncertain operating environments, and model 22 is the proposed SRM considering uncertain operating environments. For detailed construction information on models 1 to 21, see the papers on the proposed models.

3. Multi-Criteria Decision Method Using Ranking

To demonstrate the reliability of the software, multiple criteria are used based on the distance between the actual and estimated values. We propose a new measure that integrates the proposed multiple-criteria measures using multi-criteria decision-making (MCDM). The traditional MCDM method integrates qualitative measures of quantitative analysis to reflect the variability of the indicators. In MCDM, the most important issue is determining weights based on these criteria. Singh [27] recommended an approach using Monte Carlo simulation, and Saxena et al. [28], Kumar et al. [29], and Grag et al. [30] used the entropy method to determine the criteria weights, and a technique for order preference by similarity to an ideal solution (TOPSIS) method was used to rank the criteria of SRMs. Our proposed integrated MCDMR reflects the ranking of each suitability among the methods to determine the weights.

First, matrix $C$ is constructed using the SRMs to be compared, and the criteria are calculated by the models. The ranking of each model for each criterion is then organized in matrix $R$:

$$C=\left(\begin{array}{ccc}\begin{array}{cc}{C}_{11}& C_{12}\\ C_{21}& C_{22}\end{array}& \cdots & \begin{array}{c}{C}_{1k}\\ C_{2k}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{C}_{s1}& C_{s1}\end{array}& \cdots & C_{sk1}\end{array}\right), R=\left(\begin{array}{ccc}\begin{array}{cc}{R}_{11}& R_{12}\\ R_{21}& R_{22}\end{array}& \cdots & \begin{array}{c}{R}_{1k}\\ R_{2k}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{R}_{s1}& R_{s1}\end{array}& \cdots & R_{sk1}\end{array}\right)$$

Based on this, a new $y_{ij}$ is created by normalizing each fitness by $y_{ij}=\frac{C_{ij}}{{\sum }_{i=1}^s{C}_{ij}}(i=1,2,\cdots ,m,j=1,2,\cdots ,n)$ for objective criteria, and the ranking of each fitness by $w_{ij}=\frac{r_{ij}}{{\sum }_{i=1}^m{r}_{ij}}(j=1,2,\cdots ,n)$ is transformed to create a weight $w_{ij}$. Matrix $V$ is created by multiplying the new normalized $y_{ij}$ by weight $w_{ij}$.

$$V=\left(\begin{array}{ccc}\begin{array}{cc}{V}_{11}& V_{12}\\ V_{21}& V_{22}\end{array}& \cdots & \begin{array}{c}{V}_{1k}\\ V_{2k}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{V}_{s1}& V_{s1}\end{array}& \cdots & V_{sk1}\end{array}\right)=\left(\begin{array}{ccc}\begin{array}{cc}{w}_{11}{y}_{11}& w_{12}{y}_{12}\\ w_{21}{y}_{21}& w_{22}{y}_{22}\end{array}& \cdots & \begin{array}{c}{w}_{1k}{y}_{1k}\\ w_{2k}{y}_{2k}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{w}_{s1}{y}_{s1}& w_{s2}{y}_{s2}\end{array}& \cdots & w_{sk}{y}_{sk}\end{array}\right)$$

The MCDMR defines $V_1$, $\cdots$, $V_s$ as the sum of all components of each row in a matrix $V$. For example, $V_1$ for the first row is the sum of the components from $V_{11}$ to $V_{1k}$.

$$MCDMR=\left(\begin{array}{c}{V}_{11}+V_{12}+\cdots +V_{1k}\\ \begin{array}{c}{V}_{21}+V_{22}+\cdots +V_{2k}\\ ⋮\end{array}\\ V_{s1}+V_{s2}+\cdots +V_{sk}\end{array}\right)=\left(\begin{array}{c}{V}_1\\ \begin{array}{c}{V}_2\\ ⋮\end{array}\\ V_s\end{array}\right)$$

After computing the MCDMR for each $V_1$ to $V_{\mathrm{s}}$, the smaller the value displayed, the better the corresponding SRGM. This is useful for a comprehensive judgment because it not only relies on individual criteria but also considers criteria with multiple features simultaneously. Therefore, our proposed MCDMR simultaneously considers multiple fits by ranking.

4. Numerical Example

4.1. Data Information

Two datasets were used in this study. The first dataset comprises failure data from software developed by Bell Labs [31]. The software consists of 21,700 instructions written by nine programmers. Failure data were observed for 12 h, and 104 failures were observed. The second dataset was obtained by testing an online software package developed by IBM [32]. The software, which consists of 40,000 lines of code, was tested for 21 days, and 46 cumulative failures were recorded. We demonstrated that the proposed SRM performs well in estimating software failures using these two datasets.

4.2. Criteria

We compared the proposed SRM with several existing SRMs using 15 criteria. The 15 criteria are derived from the difference between the actual and estimated values;

Table 2. Criteria for model comparisons.

No.	Criteria	Formula
1	MSE	$\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{n-m}$
2	RMSE	$\sqrt{\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{n-m}}$
3	PRR	${\displaystyle \sum _{i=1}^n}{\left(\frac{\widehat{m}\left(t_i\right)-y_i}{\widehat{m}\left(t_i\right)}\right)}^2$
4	PP	${\displaystyle \sum _{i=1}^n}{\left(\frac{\widehat{m}\left(t_i\right)-y_i}{y_i}\right)}^2$
5	$R^2$	$1-\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{{\sum }_{i=1}^n{\left(y_i-\overline{y_i}\right)}^2}$
6	${adj\_R}^2$	$1-\frac{\left(1-R^2\right)\left(n-1\right)}{n-m-1}$
7	MAE	$\frac{{\sum }_{i=1}^n\left|\widehat{m}\left(t_i\right)-y_i\right|}{n-m}$
8	AIC	$-2\log L+2m$
9	BIC	$-2\log L+m\log n$
10	PRV	$\sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-\widehat{m}\left(t_i\right)-{\sum }_{i=1}^n\left[\frac{\widehat{m}\left(t_i\right)-y_i}{n}\right]\right)}^2}{n-1}}$
11	RMSPE	$\sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-\widehat{m}\left(t_i\right)-{\sum }_{i=1}^n\left[\frac{\widehat{m}\left(t_i\right)-y_i}{n}\right]\right)}^2}{n-1}+{{\displaystyle \sum _{i=1}^n}\left[\frac{\widehat{m}\left(t_i\right)-y_i}{n}\right]}^2}.$
12	MEOP	$\frac{{\sum }_{i=1}^n\left|\widehat{m}\left(t_i\right)-y_i\right|}{n-m+1}$
13	TS	$100\ast \sqrt{\frac{{\sum }_{i=1}^n{\left(y_i-\widehat{m}\left(t_i\right)\right)}^2}{{\sum }_{i=1}^n{y_i}^2}}$
14	PIC	${\displaystyle \sum _{i=1}^n}{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2+m\left(\frac{n-1}{n-m}\right)$
15	PC	$\frac{n-m}{2}\ast \mathit{log}\left(\frac{{\sum }_{i=1}^n{\left(\widehat{m}\left(t_i\right)-y_i\right)}^2}{n}\right)+\mathrm{m}\left(\frac{n-1}{n-m}\right)$

lists the criteria for each criterion. In Table 2, $\widehat{m}(t)$ is the estimated value of the model $m\left(t\right)$, $y_i$ is the actual value, $n$ is the number of observations, and $m$ is the number of parameters in each model.

Mean squared error (MSE) is a measure of the sum of the squares of the differences between the estimated and true values, considering the number of observations and parameters [33]. Root mean squared error (RMSE) is the square root of the mean squared error [33,34]. Predictive ratio risk (PRR) and predictive power (PP) are defined as the difference between the actual and predicted values divided by the predicted and actual values with respect to the model estimation [35]. $R^2$ is the coefficient of determination of the regression equation, and $adj\_R^2$ is the modified coefficient of determination of the regression equation that considers the number of parameters to determine the explanatory power [36]. Mean absolute error (MAE) is the sum of the absolute values of the difference between the estimated and true values, considering the number of observations and parameters [37,38]. Akaike’s information criterion (AIC) is used to compare likelihood function maximization. It aims to maximize the Kullback–Leibler level of agreement between the model and probability distribution of the data. The Bayes information criterion (BIC) is a fit that increases the sum of the squared residuals and the number of effects in the model and is a modified form of the penalty of AIC [39,40]. It penalizes the model more strongly than AIC. Predicted relative variation (PRV) is the standard deviation of the prediction bias [41]. Root mean square prediction error (RMSPE) provides an estimate of the closeness of each model’s predictions [41,42]. Mean error of prediction (MEOP) is defined as the sum of the absolute values of the deviations between the true and estimated values [37,43]. The Theil statistic (TS) is the average percentage deviation of all the time points from the true value [37]. Pham’s information criterion (PIC) considers the trade-off between the model and the number of parameters included in the model by slightly increasing the penalty for each additional parameter added to the model when the sample is fairly small [44], and Pham’s criteria (PC) is defined to consider more penalties than PIC [45]. Based on the above scale, we compared the existing NHPP SRM with our proposed model. The closer $R^2$ and $adj\_R^2$ are to 1 and the closer the other 13 scales are to 0, the better the fit. In addition, we utilized the MCDMR proposed in Section 3 to comprehensively evaluate the various criteria proposed in the literature to determine the superiority of the model by comparing the comprehensive criteria. Using R and MATLAB, we estimated the parameters of each model using the least square estimator method [46], which estimates parameters based on the difference between the model-estimated values and the actual number of failures, and calculated the criteria to compare their superiority.

4.3. Results on Dataset 1

Table 3. Parameter estimation of model from dataset 1.

No	Model	$\widehat{\mathit{a}}$	$\widehat{\mathit{b}}$	$\widehat{\mathit{\alpha }}$	$\widehat{\mathit{\beta }}$	$\widehat{\mathit{N}}$	$\widehat{\mathit{\gamma }}$	$\widehat{\mathit{c}}$	$\widehat{\mathit{p}}$	$\widehat{\mathit{q}}$	$\widehat{\mathit{d}}$	$\widehat{\mathit{h}}$
1	GO	103.9957	0.25072
2	HDGO	103.9956	0.25072					0.003999
3	DS	94.42989	0.64986
4	IS	103.9919	0.250789		0.000336
5	YE	135.4096		0.079955	0.118615		22.03046
6	YR	99.87051		0.30301	0.081458		8.609839
7	YID1	80.07877	0.368316	0.026422
8	YID2	77.60045	0.381731	0.033943
9	PZ	94.9835	4.728741	0.203146	3.469121			13.93669
10	PNZ	77.57743	0.382112	0.033973	0.000898
11	IFD	11.11251	0.830547								1.07 × 10−0.7
12	RMD	20.84912	0.289209	5.46394	0.069861
13	ZFR	16.75903	0.009765	0.010496	0.158331			1.571965	0.319499
14	KSRGM	50.34612	67.63962	0.504539					0.008162
15	TC	0.123963	0.887039	2.764066	1.760686	125.3202
16	TP	273.867	0.001876	78.32247	0.094677			15.26608	3.2414	0.611472
17	3P	7.009842	5.44 × 10−8		0.131806	137.9787		234.0264
18	Vtub	1.050673	0.921887	1.257807	0.251814	127.3638
19	DPF1	99.2143	0.00489					0.060524				28.30202
20	DPF2	99.21582	0.000271					0.058441				28.30183
21	UOP	0.959106	0.248168	1.021567	0.927368	133.764
22	NEW			2.750654	1.406831	127.7906

shows the estimated parameter values for each model obtained from dataset 1. Our proposed model has $\alpha$, $\beta$, and $N$, which are denoted as $\widehat{\alpha }=2.7507$, $\widehat{\beta }=1.4068$, and $\widehat{N}=127.7906$. Figure 1 shows a graphical representation of the cumulative number of failures and the estimated values of each time point in dataset 1. The black dots represent the actual data, and the solid red line represents the estimated value of the proposed SRM. The thick black dashed line represents the 95% confidence interval of the proposed model, and the thin black dashed line represents the 99% confidence interval of the proposed model.

Since it is not easy to judge how well the model fits from Figure 1 alone, we interpret the results in conjunction with the information in

Table 4. Comparison of all criteria from dataset 1.

Criteria	GO	HDGO	DS	IS	YE	YR	YID1	YID2	PZ	PNZ	IFD
MSE	6.474	7.193	44.306	7.196	5.728	87.848	3.994	4.040	5.413	4.546	291.502
RMSE	2.544	2.682	6.656	2.683	2.393	9.373	1.999	2.010	2.327	2.132	17.073
PRR	0.037	0.037	1.205	0.037	0.019	3.155	0.012	0.011	0.005	0.011	3.110
PP	0.029	0.029	0.323	0.029	0.016	0.505	0.011	0.011	0.005	0.011	0.900
R2	0.990	0.990	0.928	0.990	0.993	0.886	0.994	0.994	0.994	0.994	0.574
adjR2	0.987	0.986	0.912	0.986	0.988	0.821	0.992	0.992	0.989	0.991	0.414
MAE	2.274	2.526	5.566	2.526	2.510	8.872	1.948	1.963	2.396	2.210	18.217
AIC	67.156	69.156	115.306	69.160	67.385	158.805	61.394	61.667	68.850	63.665	100.809
BIC	68.126	70.610	116.275	70.614	69.325	160.745	62.848	63.121	71.275	65.605	102.264
PRV	2.412	2.412	6.230	2.412	2.036	7.823	1.807	1.817	1.856	1.817	14.116
RMSPE	2.425	2.425	6.337	2.425	2.041	7.979	1.808	1.818	1.856	1.818	15.337
MEOP	2.067	2.274	5.060	2.274	2.231	7.886	1.753	1.766	2.097	1.965	16.395
TS	2.953	2.953	7.725	2.953	2.484	9.729	2.200	2.213	2.259	2.213	18.797
PC	10.627	11.251	20.244	11.253	10.860	21.781	8.604	8.656	11.881	9.935	27.910
PIC	66.939	68.405	445.261	68.429	51.327	708.287	39.612	40.030	45.746	41.864	2627.181
MCDMR	0.01486	0.02002	0.10848	0.02181	0.01322	0.18699	0.00254	0.00414	0.01381	0.00577	0.00904
Criteria	RMD	ZFR	KSRGM	TC	TP	3P	Vtub	DPF1	DPF2	UOP	NEW
MSE	6.760	10.797	6.542	5.166	12.833	5.352	5.192	10.760	10.760	4.798	3.834
RMSE	2.600	3.286	2.558	2.273	3.582	2.314	2.279	3.280	3.280	2.190	1.958
PRR	0.028	0.037	0.062	0.006	0.037	0.009	0.006	0.026	0.026	0.005	0.005
PP	0.023	0.029	0.043	0.006	0.029	0.009	0.006	0.030	0.030	0.005	0.005
R2	0.991	0.990	0.992	0.994	0.990	0.994	0.994	0.986	0.986	0.995	0.994
adjR2	0.986	0.977	0.987	0.989	0.971	0.989	0.989	0.978	0.978	0.990	0.992
MAE	2.656	3.790	2.495	2.425	4.528	2.531	2.458	3.254	3.254	2.326	1.878
AIC	68.697	75.164	61.969	67.149	77.048	67.226	66.973	81.712	81.710	67.235	63.825
BIC	70.636	78.074	63.908	69.573	80.443	69.650	69.398	83.651	83.649	69.660	65.280
PRV	2.208	2.413	2.152	1.813	2.401	1.845	1.818	2.797	2.797	1.747	1.771
RMSPE	2.217	2.426	2.179	1.813	2.414	1.846	1.818	2.797	2.797	1.747	1.771
MEOP	2.361	3.249	2.218	2.122	3.773	2.215	2.150	2.893	2.893	2.036	1.691
TS	2.699	2.954	2.655	2.207	2.940	2.246	2.212	3.405	3.405	2.127	2.156
PC	11.522	16.058	11.391	11.718	19.591	11.842	11.736	13.382	13.381	11.459	8.420
PIC	59.581	75.779	57.836	44.021	79.566	45.323	44.203	91.581	91.579	41.444	38.176
MCDMR	0.03436	0.01289	0.00987	0.39357	0.01903	0.03164	0.01472	0.03334	0.03262	0.00745	0.00253

. Table 4 shows the results for the criteria calculated from the estimates of each model, based on the parameter estimates obtained from Table 3. $MSE$, $RMSE$, $Adj\_R^2$, $MAE$, $MEOP$, $PC$, and $PIC$ are the best, with values of $3.834$, $1.958$, $0.992$, $1.878$, $1.878$, $1.691$, $8.420$, and $38.176$, whereas $R^2$, $PRV$, $RMSPE$, and $TC$ are the second best, with values of $0.994$, 1.771, $1.771$, and $2.156$. This is also among the best fits of the other fits. The newly proposed MCDMR, which considers all 15 fits together, is also the best at $0.00253$.

4.4. Results on Dataset 2

Table 5. Parameter estimation of model from dataset 2.

No	Model	$\widehat{\mathit{a}}$	$\widehat{\mathit{b}}$	$\widehat{\mathit{\alpha }}$	$\widehat{\mathit{\beta }}$	$\widehat{\mathit{N}}$	$\widehat{\mathit{\gamma }}$	$\widehat{\mathit{c}}$	$\widehat{\mathit{p}}$	$\widehat{\mathit{q}}$	$\widehat{\mathit{d}}$	$\widehat{\mathit{h}}$
1	GO	258479.7	8.27 × 10−0.6
2	HDGO	709.7827	0.003083					0.462394
3	DS	77.25299	0.096622
4	IS	59.28546	0.16839		8.278172
5	YE	4709.105		2.411867	0.000372		0.507832
6	YR	77.83392		2.677162	0.004541		0.522922
7	YID1	19614.07	8.39 × 10−0.5	0.030908
8	YID2	1.491036	0.306843	1.745694
9	PZ	17.01691	0.153619	0.16534	5.82304			44.56855
10	PNZ	11.71105	0.195293	0.198144	1.382584
11	IFD	8.684959	0.138925								0.023631
12	RMD	104.218	0.033806	1.146429	0.149771
13	ZFR	38.40169	0.193953	5.65856	0.17475			0.178089	0.753317
14	KSRGM	32.74804	0.98946	0.821748					0.108743
15	TC	0.019064	1.567033	839.154	221.1735	78.78594
16	TP	102.844	0.173628	1.206102	1.133787			16.86387	1.556427	0.089124
17	3P	0.49233	0.168722		0.240659	61.55869		116.1341
18	Vtub	1.970059	0.689159	0.292767	19.85291	87.25193
19	DPF1	51.44718	0.004767					0.028006				2.634778
20	DPF2	51.45985	5.03 × 10−5					0.010786				2.633591
21	UOP	0.467491	0.042319	1.537043	1.497623	226.1843
22	NEW			33.62065	4.901007	167.9314

shows the estimated parameter values for each model obtained from dataset 2. The estimated parameter values of the proposed model are $\widehat{\alpha }=33.6027$, $\widehat{\beta }=4.9010$, and $\widehat{N}=167.9314$. Figure 2 shows a graphical representation of the cumulative number of failures and the estimated values at each time point in dataset 2. The black dots represent the actual data, and the solid red line represents the estimated values of the proposed SRM. The thick black dashed line represents the 95% confidence interval of the proposed model, and the thin black dashed line represents the 99% confidence interval of the proposed model.

Also, since it is not easy to judge how well the model fits from Figure 2 alone, we interpret the results in conjunction with the information in

Table 6. Comparison of all criteria from dataset 2.

Criteria	GO	HDGO	DS	IS	YE	YR	YID1	YID2	PZ	PNZ	IFD
MSE	6.568	7.728	1.637	1.395	7.559	2.420	2.936	1.701	1.564	1.646	2.104
RMSE	2.563	2.780	1.279	1.181	2.749	1.556	1.714	1.304	1.251	1.283	1.451
PRR	0.805	0.863	26.321	0.679	0.817	55.688	0.356	3.064	0.843	0.968	0.451
PP	1.852	2.041	1.208	0.297	1.889	1.588	0.526	0.608	0.331	0.369	0.351
R2	0.972	0.969	0.993	0.994	0.972	0.991	0.988	0.993	0.994	0.994	0.992
adjR2	0.969	0.964	0.992	0.993	0.964	0.989	0.986	0.992	0.993	0.992	0.990
MAE	2.232	2.525	1.107	0.973	2.545	1.477	1.542	1.170	1.104	1.157	1.305
AIC	77.325	79.554	78.118	76.699	81.386	83.031	79.127	78.662	80.793	79.560	78.284
BIC	79.414	82.688	80.207	79.832	85.564	87.209	82.261	81.796	86.016	83.738	81.417
PRV	2.325	2.459	1.224	1.120	2.371	1.375	1.606	1.236	1.118	1.183	1.372
RMSPE	2.490	2.629	1.246	1.120	2.527	1.432	1.625	1.237	1.119	1.183	1.376
MEOP	2.120	2.392	1.052	0.921	2.403	1.395	1.461	1.109	1.039	1.093	1.236
TS	9.047	9.552	4.516	4.058	9.181	5.195	5.887	4.481	4.052	4.284	4.984
PC	19.035	20.350	5.834	4.940	20.103	10.423	11.639	6.726	7.655	7.146	8.642
PIC	126.890	142.437	33.200	28.438	133.214	45.851	56.180	33.948	31.280	32.690	41.213
MCDMR	0.09265	0.11500	0.03568	0.00512	0.10967	0.08481	0.04813	0.02263	0.01316	0.01963	0.03299
Criteria	RMD	ZFR	KSRGM	TC	TP	3P	Vtub	DPF1	DPF2	UOP	NEW
MSE	1.616	1.671	1.850	1.723	1.794	1.569	1.544	2.002	2.001	1.573	1.387
RMSE	1.271	1.293	1.360	1.313	1.339	1.253	1.243	1.415	1.415	1.254	1.178
PRR	3.112	0.797	36.310	6.012	0.707	0.681	0.561	0.336	0.336	0.271	0.375
PP	0.611	0.321	1.174	0.760	0.303	0.297	0.270	0.583	0.582	0.193	0.231
R2	0.994	0.994	0.993	0.994	0.994	0.994	0.995	0.992	0.992	0.994	0.995
adjR2	0.992	0.992	0.991	0.992	0.991	0.993	0.993	0.991	0.991	0.993	0.994
MAE	1.150	1.179	1.239	1.207	1.257	1.095	1.094	1.204	1.203	1.131	0.998
AIC	80.090	82.790	83.337	82.566	84.738	80.702	80.602	78.795	78.792	80.563	76.587
BIC	84.268	89.057	87.515	87.788	92.049	85.924	85.824	82.973	82.970	85.786	79.720
PRV	1.170	1.119	1.245	1.169	1.120	1.120	1.110	1.300	1.300	1.122	1.117
RMSPE	1.172	1.119	1.254	1.174	1.121	1.120	1.111	1.304	1.304	1.122	1.117
MEOP	1.086	1.105	1.170	1.136	1.173	1.030	1.030	1.137	1.136	1.065	0.946
TS	4.244	4.054	4.542	4.252	4.059	4.058	4.025	4.725	4.723	4.063	4.047
PC	6.987	9.326	8.140	8.425	11.252	7.679	7.549	8.810	8.805	7.700	4.891
PIC	32.171	33.061	36.160	33.810	35.114	31.356	30.952	38.740	38.719	31.423	28.301
MCDMR	0.01999	0.02143	0.05685	0.03090	0.02850	0.01373	0.00941	0.03148	0.02953	0.01523	0.00271

. Table 6 shows the results for the criteria calculated from the estimates of each model, based on the parameter estimates obtained from Table 5. $MSE$, $RMSE$, $R^2$, $Adj\_R^2$, $AIC$, $PC$, and $PIC$ have the best results with $1.387$, $1.178$, $0.995$, $0.994$, $76.587$, $4.891$, and $28.301$, whereas $PP$, $MAE$, $BIC$, $PRV$, $PMSPE$, $MEOP$, and $TC$ are the second best with values of $0.231$, $0.998$, $79.720$, $1.117$, $1.117$, $0.946$, and $4.047$. The newly proposed MCDMR, which considers all 15 fits together, is also the best at 0.00271.

5. Conclusions and Remark

In this paper, we proposed a new NHPP SRM that considers uncertain operating environments. Unlike the existing NHPP SRMs that consider uncertain operating environments, the proposed model minimizes assumptions and fits general situations with fewer parameters. In addition, to demonstrate the superiority of the proposed model, a new MCDMR incorporating multiple criteria for comparing the models has been proposed. The proposed NHPP SRM, which considers uncertain operating environments, was compared with several traditional NHPP SRMs using two datasets, and its superiority was demonstrated.

The development of traditional SRMs is based on limited circumstances and fixed assumptions. However, as the software industry develops, software structures become increasingly complex, and various assumptions are added to predict software failures. This complicates the structure of SRMs for predicting software failures and causes problems because they work well only in special cases. To improve this, there is a significant need for research that minimizes these assumptions. In addition, it is very difficult for traditional NHPP SRMs to model data generated in real time on the fly. Therefore, research is being conducted on SRMs using data-dependent machine learning algorithms [47]. Kumar and Singh [48] and Jaiswal and Malhotra [49] applied well-known machine learning methods such as artificial neural networks, support vector machines, cascade correlation neural networks, decision trees, and fuzzy inference systems to predict software reliability. Wang et al. [50] and Chen et al. [51] predicted software failures using recurrent neural networks that exploit the characteristics of sequential data as a method for predicting SRMs and showed good performance. Zainuddin et al. [52] and Wang et al. [53] proposed a model for predicting software failures using long short-term memory (LSTM) and gated recurrent units (GRU), which are extensions of recurrent neural networks. Research is required on new SRMs that extend the model proposed in this study and assume general rather than specialized situations using machine learning and deep learning.