Modeling and Assessing Software Reliability in Open-Source Projects

Abstract

One of the key components of the software quality model is reliability. Its importance has grown with the increasing use and reuse of open-source components in software development. Software reliability growth models are commonly employed to address this aspect by predicting future failure rates and estimating the number of remaining defects throughout the development process. This paper investigates two software reliability growth models derived from the Verhulst model, with a particular focus on a structural property known as Hausdorff saturation. We provide analytical estimates for this characteristic and propose it as an additional criterion for model selection. The models are evaluated using four open-source datasets, where the Hausdorff saturation metric supports the conclusions drawn from standard goodness-of-fit measures. Furthermore, we introduce an interactive software reliability assessment tool that integrates with GitHub, enabling expert users to analyze real-time issue-tracking data from open-source repositories. The tool facilitates model comparison and enhances practical applicability. Overall, the proposed approach contributes to more robust reliability assessment by combining theoretical insights with actionable diagnostics.

1. Introduction

Reliability is one of the key attributes of a quality model according to ISO/IEC 25010:2023 (Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—Product quality model) [1]. Moreover for software it is a critical quality metric that reflects a system’s resistance to failure during operation. Traditional reliability assessment methods often require extensive testing or production deployment data. One basic approach of software reliability prediction is using software reliability growth models (SRGMs). Their main aim is supporting the reliability estimation in the process of software development, especially in the phases of testing and validation.

The development of new software reliability growth models constitutes an important topic in the scientific literature. Each of them is based on distinct assumptions concerning the software fault detection process. For instance, some models assume perfect debugging, where faults are immediately and completely removed without introducing new errors. However, this is often unrealistic, as the difficulty of correcting faults tends to increase as testing progresses. In contrast, other models incorporate imperfect debugging, acknowledging that the fault correction process may be incomplete and may introduce additional ones. Some new models for imperfect debugging can be found in [2,3,4,5,6,7].

Open-source software (OSS) projects are developed collaboratively by geographically dispersed developers without centralized management. Unlike traditional software, the testing phase is not clearly demarcated; it is carried out in parallel with coding. It relies on users to identify defects rather than dedicated test teams, so all bugs cannot be fully detected. Given the dynamic and distributed nature of OSS development, assessing its reliability is essential, particularly as such software is increasingly used in new systems. Studying OSS reliability helps ensure quality despite the absence of formal testing phases. Our objective is to investigate the reliability of open-source projects by applying an SRGM. While this is not a novel technique, it remains a meaningful approach for understanding and predicting the fault-finding process. Other advanced methods and applications of the methodology are extensively explored in various studies, such as [8,9,10,11,12].

Most software reliability growth models assume that software faults follow a nonhomogeneous Poisson process (NHPP). They are characterized by a model-specific mean value function $m\left(t\right)$ used to estimate the expected number of faults detected over time. Based on $m\left(t\right)$, the reliability function for a NHPP model can be expressed as the conditional probability $R\left(x\right|t)$, which represents the probability that no software failure will occur in the time interval $[t,t+x]$, where $t\ge 0$ and $x>0$. Formally, this can be written as

$$R\left(x\right|t)=exp(-[m(t+x)-m\left(t\right)\left]\right).$$

Table 1. Summary of some basic software reliability models.

Model Name	Mean Value Function	Model Type
Goel–Okumoto [13]	$m\left(t\right)=a(1-exp(-bt))$	Concave
Delayed S-shaped [14]	$m\left(t\right)=a(1-(1+bt)exp(-bt))$	S-shaped
Inflection S-shaped [15]	${\displaystyle m\left(t\right)=\frac{a(1-(1+bt)exp(-bt))}{1+\beta exp(-bt)}}$	S-shaped
Yamada linear imperfect debugging model [16]	$m\left(t\right)=a(1-exp(-bt))\left(1-{\displaystyle \frac{\alpha }{b}}\right)+\alpha at$	Concave
Yamada exponential imperfect debugging model [16]	${\displaystyle m\left(t\right)=\frac{ab}{\alpha +b}\left(exp\left(\alpha t\right)-exp(-bt)\right)}$	Concave
Pham–Nordmann–Zhang [17]	${\displaystyle m\left(t\right)=\frac{a}{1+\beta exp(-bt)}\left((1-exp(-bt))\left(1-\frac{\alpha }{b}\right)+\alpha t\right)}$	S-shaped and Concave
Pham–Zhang [18]	${\displaystyle m\left(t\right)=\frac{\left({\displaystyle (c+a)(1-exp(-bt))-\frac{ab}{b-\alpha }(exp(-\alpha t)-exp(-bt))}\right)}{1+\beta exp(-bt)}}$	S-shaped and Concave
Dependent-parameter model [19]	$m\left(t\right)=\alpha (1+\beta t)(\beta t+exp(-\beta t)-1)$	Concave

presents summary of some basic software reliability growth models including the definitions of their mean value functions and their corresponding types (concave or S-shaped).

In 2018, Pham [20] proposed estimating the mean value function $m\left(t\right)$ using a logistic growth model, also known as the Verhulst model, defined by

$${\displaystyle \frac{dm\left(t\right)}{dt}}=b\left(t\right)m\left(t\right)\left(1-{\displaystyle \frac{m\left(t\right)}{a\left(t\right)}}\right),m\left(0\right)\ne 0,$$

(1)

where $a\left(t\right)$ is fault content function and $b\left(t\right)$ is software fault detection rate per unit of time.

Haque and Ahmad [21] proposed the software reliability model that takes into account the improvement of the testing team’s skills over time. Its mean value function is expressed by

$$m_{HA}\left(t\right)={\displaystyle \frac{a}{1+\beta exp\left({\displaystyle -\frac{b{t}^{\alpha }}{\alpha }}\right)}}.$$

(2)

They used (1) with $a\left(t\right)=a$, $b\left(t\right)=b{t}^{\alpha -1}$ and ${\displaystyle m\left(0\right)=\frac{a}{1+\beta }}$. Function $b\left(t\right)$ is the same as in the Generalized Goel Model and represents an initial increase followed by a decrease in the failure rate.

Song, Kim and Chang [22] consider a software reliability model that considers the number of finite faults and dependent faults with the mean value function defined by

$$m_{SKC}\left(t\right)={\displaystyle \frac{a}{1+\left({\displaystyle \frac{a}{k}-1}\right)(1+bt)exp(-bt)}}.$$

(3)

It is obtained from (1) with $a\left(t\right)=a$, ${\displaystyle b\left(t\right)=\frac{b^{2}t}{bt+1}}$ and $m\left(0\right)=k$. This model is assumed to be used for open-source software.

Some other SRGM models obtained by the logistic growth model can be found in [23,24,25].

In this study, we have two main aims. First, we introduce an intrinsic characteristic known as Hausdorff saturation for software reliability models. It provides a quantitative measure of how close the software is to completing its failure discovery process. In Section 2 we present a detailed analysis for two software reliability models—Haque–Ahmad $m_{HA}$ and Song–Kim–Chang $m_{SKC}$. We prove precise estimates of Hausdorff distance to the horizontal asymptote. Additionally, we utilize four datasets from an open-source platforms to perform a comparative analysis between three classical and two proposed software reliability growth models. We apply standard goodness-of-fit tests and Hausdorff saturation to determine the most appropriate approximation model. Our second goal is assessing the reliability of open-source software (OSS) projects by analyzing the patterns of issue creation and resolution over time. By focusing on the trends in opened and closed issues, we aim to determine the maturity and reliability of OSS projects. Section 3 describes the development of a software reliability assessment tool designed for this purpose. The proposed tool implements five software reliability growth models within a dynamic Wolfram Language environment that directly interfaces with the GitHub API. The presented process enables real-time reliability assessment and provides comparative analysis of model fits to determine the most appropriate for a given software project.

2. Approximation Results

2.1. Hausdorff Saturation

The main aim of this subsection is investigation of the intrinsic characteristic named Hausdorff saturation. It presents the distance between the mean value function $m\left(t\right)$ of software reliability growth model and a $\mathsf{\Gamma }$-shaped curve connecting its endpoints. This metric can be applied to models with a horizontal asymptote, which typically represents the theoretical maximum number of defects that can be detected over time. By evaluating the proximity of the model’s predictions to this asymptote, Hausdorff saturation provides insight into the maturity and completeness of the software testing process. This characteristic can aid in selecting the most suitable software reliability model for approximating a given dataset.

The advantages of the Hausdorff metric for approximating functions and point sets in the plane, compared to the uniform metric, have been highlighted in numerous publications and monographs over the past 70 years. Recently, this classical theory has been applied in the study of dynamic models in various fields, including growth theory such as investigations into epidemic models, particularly those related to the COVID-19 pandemic [26], the synthesis and analysis of antenna patterns and filters [27], the approximation of data related to computer virus propagation [28], debugging and test theory [29,30,31,32,33].

We consider Euclidean space with the maximum norm, namely, for the points $A=(t_A,x_A)$ and $B=(t_B,x_B)$ in ${\mathbb{R}}^2$, we have $\parallel A-B\parallel =max\left(\right|t_A-t_B|,|x_A-x_B\left|\right)$. Hence we can define the Hausdorff distance [34,35].

Definition 1.

The Hausdorff distance (the H-distance) $\rho (f,g)$ between two interval functions $f,g$ on $\mathsf{\Omega }\subseteq \mathbb{R}$ is the distance between their completed graphs $F\left(f\right)$ and $F\left(g\right)$ considered as closed subsets of $\mathsf{\Omega }\times \mathbb{R}$. More precisely,

$$\rho (f,g)=max\{\underset{A\in F\left(f\right)}{sup}\underset{B\in F\left(g\right)}{inf}\parallel A-B\parallel ,\underset{B\in F\left(g\right)}{sup}\underset{A\in F\left(f\right)}{inf}\parallel A-B\parallel \}.$$

Definition 2.

The saturation of a software reliability model on $[a,b)$, $-\infty <a<b\le \infty$ is the Hausdorff distance between the completed graph of the mean value function $m(·)$ with asymptote h and the Γ-shaped curve consisting of two lines—one vertical between the points $(a,0)$ and $(a,h)$ and another horizontal between $(a,h)$ and $(b,m(b\left)\right)$.

Corollary 1.

The saturation of an SRGM model with mean value function $m\left(t\right)$ to the horizontal asymptote h at the “median level” satisfies the following nonlinear equation:

$$m(t_0+d)=h-d,$$

(4)

where ${\displaystyle m\left(t_0\right)=\frac{1}{2}h}$.

Proof. 

The proof follows immediately from Definitions 1 and 2. Equation (4) has a unique root since the function $f\left(d\right)=m(t_0+d)-h+d$ is increasing and continuous. □

According to Definitions 1 and 2, we have that Hausdorff saturation d is the smallest square (box) ball touching the graph of the mean value function of the software reliability model. The Hausdorff metric is an approximation error between the sigmoidal (mean value) function and a step function. Figure 1 presents a graphical representation of the problem.

A small Hausdorff distance indicates that the model’s predicted failure counts are approaching the theoretical upper bound, suggesting that the majority of detectable defects have been identified and resolved. This reflects a mature testing process and a high level of software reliability. Conversely, a large Hausdorff distance implies a significant gap between current failure predictions and the model’s asymptote, indicating that the software remains in the reliability growth phase and is not yet suitable for high-assurance deployment.

The next theorem gives some useful estimates for Hausdorff saturation of the software reliability model.

Theorem 1.

The Hausdorff saturation d of Haque–Ahmad software reliability model $m_{HA}$ defined by (2) satisfies

$$d_l={\displaystyle \frac{a}{{\mathsf{\Delta }}_1}}<d<{\displaystyle \frac{aln\left({\mathsf{\Delta }}_1\right)}{{\mathsf{\Delta }}_1}}=d_r,$$

where

$${\mathsf{\Delta }}_1=2.1\ast \left(1+{\displaystyle \frac{1}{4}}a\alpha log\left(\beta \right){\left({\displaystyle \frac{b}{\alpha log\left(\beta \right)}}\right)}^{1/\alpha }\right),$$

(5)

with ${\mathsf{\Delta }}_1>e^{1.05}$.

Proof of Theorem 1.

First we note that ${lim}_{t\to \infty }{m}_{HA}\left(t\right)=a$ and ${\displaystyle m_{HA}\left(t_0\right)=\frac{1}{2}a}$ for $t_0={\left({\displaystyle \frac{b}{\alpha log\left(\beta \right)}}\right)}^{-1/\alpha }$. We apply Corollary 1 as we consider the function

$$f_1\left(d\right)=m_{HA}(t_0+d)-a+d.$$

We apply the Taylor series for function

$$f\left(d\right)=m\left(t_0\right)-a+(1+m^{\prime }\left(t_0\right))d+\mathcal{O}\left(d^2\right)=-{\displaystyle \frac{a}{2}}+(1+m^{\prime }\left(t_0\right))d+\mathcal{O}\left(d^2\right).$$

With simple calculations, we have

$$m_{HA}^{\prime }\left(t\right)={\left({\displaystyle \frac{a}{1+\beta exp\left(-\frac{b{t}^{\alpha }}{\alpha }\right)}}\right)}^{\prime }={\displaystyle \frac{ab\beta t^{\alpha -1}{e}^{-\frac{b{t}^{\alpha }}{\alpha }}}{1+{\left(\beta e^{-\frac{b{t}^{\alpha }}{\alpha }}\right)}^2}}.$$

Hence, ${\displaystyle m_{HA}^{\prime }\left(t_0\right)=\frac{{\mathsf{\Delta }}_1}{2.1}-1}$. From here we can conclude that $g_1\left(d\right)-f_1\left(d\right)=\mathcal{O}\left(d^2\right)$, where

$$g_1\left(d\right)=-{\displaystyle \frac{a}{2}}+{\displaystyle \frac{{\mathsf{\Delta }}_1}{2.1}}d$$

with ${\mathsf{\Delta }}_1$ defined by (5). Therefore function $g_1\left(d\right)$ approximates $f_{!}\left(d\right)$ for $d\to 0$ with $\mathcal{O}\left(d^2\right)$. Function $g_1\left(d\right)$ is also monotonically increasing since $g_1^{\prime }\left(d\right)>0$. If ${\mathsf{\Delta }}_1>e^{1.05}$, then $g\left(d_l\right)<0$ and $g\left(d_r\right)>0$, which completes the proof. □

Theorem 2.

The Hausdorff saturation d of the Song–Kim–Chang software reliability model $m_{SKC}$ defined by (3) satisfies

$$d_l={\displaystyle \frac{a}{{\mathsf{\Delta }}_2}}<d<{\displaystyle \frac{aln\left({\mathsf{\Delta }}_2\right)}{{\mathsf{\Delta }}_2}}=d_r,$$

where

$${\mathsf{\Delta }}_2=2.1\ast \left(1+{\displaystyle \frac{ab\left(-a{e}^{W_{-1}\left(\frac{k}{e(k-a)}\right)+1}+k{e}^{W_{-1}\left(\frac{k}{e(k-a)}\right)+1}+k\right)}{4k}}\right),$$

(6)

with ${\mathsf{\Delta }}_2>e^{1.05}$, and $W_k\left(z\right)$ is the Lambert-W function.

Proof of Theorem 2.

Regarding analogues, first we compute ${lim}_{t\to \infty }{m}_{SKC}\left(t\right)=a$. Moreover ${\displaystyle m_{SKC}\left(t_0\right)=\frac{1}{2}a}$ for $t_0=-{\displaystyle \frac{W_{-1}\left(\frac{k}{e(k-a)}\right)+1}{b}}$. We apply Corollary 1 as we consider the function

$$f_2\left(d\right)=m_{SKC}(t_0+d)-a+d.$$

We consider the analogue from the Taylor series, and

$$m_{SKC}^{\prime }\left(t\right)={\displaystyle \frac{a{b}^{2}kt(a-k)e^{bt}}{{\left((a-k)(bt+1)+k{e}^{bt}\right)}^2}},$$

and we obtain that ${\displaystyle m_{SKC}^{\prime }\left(t_0\right)=\frac{{\mathsf{\Delta }}_2}{2.1}-1}$. Hence $g_2\left(d\right)-f_2\left(d\right)=\mathcal{O}\left(d^2\right)$, where

$$g_2\left(d\right)=-{\displaystyle \frac{a}{2}}+{\displaystyle \frac{{\mathsf{\Delta }}_2}{2.1}}d$$

with ${\mathsf{\Delta }}_2$ defined by (6). Therefore function $g_2\left(d\right)$ approximates $f_2\left(d\right)$ for $d\to 0$ with $\mathcal{O}\left(d^2\right)$. We have analogue proof of $g_2\left(d_l\right)<0$ and $g_1\left(d_r\right)>0$, which ends the proof. □

2.2. Numerical Examples

In this subsection, we present two examples involving the fitting of software reliability growth models (SRGMs) to an open-source dataset containing detected software faults. We constructed a nonlinear regression model based on an SRGM using the NonlinearModelFit function in the CAS Wolfram Mathematica [36]. Let us note that this function estimates the parameters of nonlinear models via the least-squares method, under the assumption that the residuals are independent and normally distributed. It returns a FittedModel object, which provides access to a wide range of diagnostic information. A key advantage of this model-fitting framework is the ability to retrieve detailed results and diagnostics without refitting the model. The available properties include parameter estimates, standard errors, confidence intervals, residuals and goodness-of-fit statistics.

Example 1.

The GNOME (GNU’s Not Unix Network Object Model Environment) project is an open-source platform that has a free and complete set of user friendly applications and tools. We consider the dataset (

Table 2. Detected faults in GNOME 2.2 official public releases.

week	1	2	3	4	5	6	7	8	9	10	11	13	15	16	17	18	22	24
Cumulative detected Failures	5	9	14	19	28	33	35	36	38	41	43	44	48	49	50	51	52	54

) from version GNOME 2.2 [37]. The first row is time in weeks, and the second one is for cumulative detected faults.

We propose a comparison between software reliability models that have horizontal asymptote $h=a$, namely, the Goel–Okumoto, delayed S-shaped, inflection S-shaped, Haque–Ahmad and Song–Kim–Chang models. Values of estimated parameters with four digits after the decimal point for the considered models are presented in

Table 3. Parameter estimation for GNOME 2.2 dataset.

Model	a	b	$\mathit{\alpha }$	$\mathit{\beta }$	k
Goel–Okumoto	$57.0362$	$0.1224$	−	−	−
Delayed S-shaped	$50.7748$	$0.3325$	−	−	−
Inflection S-shaped	$53.0419$	$0.2078$	−	$0.9808$	−
Haque–Ahmad	$56.2411$	$1.2771$	$0.1531$	$50661.1$	−
Song–Kim–Chang	$49.5002$	$0.5413$	−	−	$8.9540$

.

Figure 2 shows a graphical representation of the cumulative number of failures (with black dots) and estimated mean value functions of the proposed software reliability models.

We evaluate the goodness-of-fit of the considered software reliability models using standard criteria, available as properties of the FittedModel object. Namely, we use the Akaike information criterion (AIC), Bayesian information criterion (BIC), coefficient of determination ($R^2$ or R-squared), and Adjusted R-squared. Let us recall that, among a set of approximation models, the preferred model is the one with the minimum AIC and BIC values and the maximum R-squared and adjusted R-squared values. The numerical results are presented in

Table 4. Comparison of criteria values for GNOME 2.2 dataset.

Model	Goel–Okumoto	Delayed S-Shaped	Inflection S-Shaped	Haque–Ahmad	Song–Kim–Chang
AIC	78.8995	81.7299	76.1154	71.5129	97.8165
BIC	81.5706	84.4010	79.6769	75.9648	101.378
R-Squared	0.9978	0.9974	0.9983	0.9988	0.9944
Adjusted R-Squared	0.9975	0.9971	0.9980	0.9985	0.9933

. We can see that Haque–Ahmad software reliability model $m_{HA}$ has the best criteria values.

Let us consider the Hausdorff saturation of obtained approximation models with parameters from Table 3. For Haque–Ahmad software reliability model $m_{HA}$ defined by (2) for $a=56.2411$ $b=1.2771$, $\alpha =0.1531$ and $\beta =50661.1$, the value of Hausdorff distance is $d=8.72883$ with $t_0=5.51209$. The values of upper and lower estimates $d_l=5.12019$ and $d_r=12.2703$ are computed using Theorem 1. According to Theorem 2 for the Song–Kim–Chang software reliability model, we have $d=5.01575$ for $t_0=5.28609$ with corresponding estimates $d_l=3.95241$ and $d_r=9.99032$. Figure 3 graphically presents the obtained results. It is obvious that Haque–Ahmad software reliability model $m_{HA}$ is better for the GNOME 2.2 dataset.

The residuals from a fitted model are defined as the differences between the observed and predicted values. The fitting residuals data is distributed according to the Normal distribution. The residual vs. predictor plot of fitting residuals for all five models is presented in Figure 4. Also Figure 5 presents the quantile plot of fitting residuals for Haque–Ahmad and Song–Kim–Chang software reliability models.

Also we tested the hypothesis that fitting residuals for all five compared models follow a normal distribution. It is not rejected at the 5 percent significance level according to the Jarque–Bera ALM, Mardia Combined, Mardia Kurtosis, Mardia Skewness or Shapiro–Wilk standard goodness-of-fit tests.

We propose comparing different types of software reliability models for approximating the GNOME 2.2 dataset. According to the standard test of goodness-of-fit and Hausdorff saturation, we can conclude that Haque–Ahmad software reliability model $m_{HA}$ is the best for predicting the number of faults in the future. Figure 6 presents actual and predicted number of software faults. The solid black points show the faults that were actually observed during testing, while the open (unfilled) points show the predicted values from our model. This kind of plot helps us assess how stable the fault detection process is and gives us a sense of whether the software is reaching a reliable state before release.

Example 2.

Ansible is an open-source automation tool that supports provisioning, configuration, application deployment and infrastructure management following the Infrastructure as Code (IaC) paradigm. It uses a declarative language to define system states and operates across Unix-based systems and Windows. As a widely adopted DevOps tool, Ansible plays a crucial role in automating complex tasks and ensuring efficient, repeatable infrastructure operations.

We consider a dataset (

Table 5. Detected faults in Ansible official public releases.

months	1	2	3	4	5	6	7	8	9	10	11	12
Cumulative detected Failures	240	404	540	792	932	1118	1285	1504	1706	1982	2005	2454
months	13	14	15	16	17	18	19	20	21	22	23	24
Cumulative detected Failures	3152	3766	4441	5147	5629	6201	6827	7275	8375	8991	9525	9851

) from Ansible software used in [38]. The time period is from 1 January 2016 to 31 December 2017 as for each month we take the cumulative detected faults.

Table 6. Parameter estimation for Ansible software dataset.

Model	a	b	$\mathit{\alpha }$	$\mathit{\beta }$	k
Goel–Okumoto	$4.3910\times {10}^6$	$7.8016\times {10}^{-5}$	−	−	−
Delayed S-shaped	158,948	$0.0170$	−	−	−
Inflection S-shaped	$\mathrm{15,564.2}$	$0.1649$	−	$27.0258$	−
Haque–Ahmad	$\mathrm{12,873.3}$	$0.1601$	$1.0968$	$34.4353$	−
Song–Kim–Chang	$\mathrm{12,207.3}$	$0.2740$	−	−	$526.564$

presents values of estimated parameters with four digits after the decimal point for the considered models. Figure 7 shows a graphical representation of the cumulative number of failures (with black dots) and estimated mean value functions of the proposed software reliability models.

Table 7. Comparison of criteria values for Ansible software dataset.

Model	Goel–Okumoto	Delayed S-Shaped	Inflection S-Shaped	Haque–Ahmad	Song–Kim–Chang
AIC	408.901	341.398	329.312	319.08	320.008
BIC	412.435	344.932	334.024	324.97	324.72
R-Squared	0.954636	0.997276	0.998493	0.999101	0.998977
Adjusted R-Squared	0.950513	0.997028	0.998277	0.998921	0.998831

presents the values of standard criteria for evaluation the goodness-of-fit. Although the Haque–Ahmad software reliability model $m_{HA}$ has the best values for three of the proposed criteria, both the Haque–Ahmad and Song–Kim–Chang models yield extremely similar results. Therefore, we can conclude that both models are suitable for application to the considered dataset.

The Hausdorff saturation is calculated for obtained parameters from Table 6. We obtain the value of Hausdorff distance $d=27.52$ with $t_0=18.2982$ for Haque–Ahmad software reliability model $m_{HA}$. By Theorem 1 we have estimates $d_l=8.96625$ and $d_r=65.1797$. Regarding the analogue for Song–Kim–Chang software reliability model $m_{SKC}$, we have $d=25.3841$ for $t_0=17.7701$ with corresponding estimates $d_l=8.36737$ and $d_r=9.99032$ by Theorem 2. The obtained Hausdorff saturation values are highly similar, particularly when normalized with respect to the horizontal asymptote. This suggests that both models are equally appropriate for application to the given dataset.

Based on the fault detection data for the Ansible software over a two-year period, both the Haque–Ahmad and Song–Kim–Chang models demonstrate comparable performance in terms of goodness-of-fit and Hausdorff saturation. Therefore, both models can be effectively used for fault prediction in this context. As shown in Figure 8, the actual number of faults (solid black points) is followed by predicted values (open, unfilled points), with both models closely capturing the observed trend. This consistency indicates that either model can be reliably applied for fault prediction and reliability assessment of the software.

Example 3.

Electron is an open-source project under the OpenJS Foundation maintained by an active community of contributors. It is designed for developing cross-platform desktop applications with HTML, CSS and JavaScript. Electron is built on Node.js and Chromium, which makes it a suitable choice for web developers.

Electron frequently releases major versions in sync with every other Chromium release. We consider the issue reports from GitHub for version 31.0.0 during its Stable phase.

Table 8. Detected faults in Electron official public releases.

weeks	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Cumulative Failures	1	4	9	15	17	23	29	37	42	62	74	82	89	97	113
weeks	16	17	18	19	20	21	22	23	24	25	26	27	28	29
Cumulative Failures	127	139	152	160	163	167	168	172	178	180	181	182	185	186

presents the cumulative number of detected faults over a period of 29 weeks, from 10 April 2024 to 30 October 2024.

Table 9. Parameter estimation for Electron 31.0.0 dataset.

Model	a	b	$\mathit{\alpha }$	$\mathit{\beta }$	k
Goel–Okumoto	$9417.43$	$0.0007$	−	−	−
Delayed S-shaped	$250.494$	$0.1015$	−	−	−
Inflection S-shaped	$181.97$	$0.2871$	−	$37.8345$	−
Haque–Ahmad	$189.496$	$0.4169$	$0.8222$	$68.2193$	−
Song–Kim–Chang	$184.097$	$0.3527$	−	−	$9.6001$

presents the estimated parameters for the models under consideration, with values rounded to four decimal places. Figure 9 provides a graphical representation of the cumulative number of failures (indicated by black dots) alongside the estimated mean value functions of the proposed software reliability models. A quantitative evaluation of the models’ performance, based on standard goodness-of-fit criteria, is detailed in

Table 10. Comparison of criteria values for Electron software dataset.

Model	Goel–Okumoto	Delayed S-Shaped	Inflection S-Shaped	Haque–Ahmad	Song–Kim–Chang
AIC	243.408	211.389	171.699	158.868	171.148
BIC	247.51	215.491	177.168	165.705	176.618
R-Squared	0.986373	0.995483	0.998931	0.999362	0.998951
Adjusted R-Squared	0.985364	0.995148	0.998807	0.999260	0.998830

.

Using the parameters estimated in Table 9, we calculated the Hausdorff saturation for the Haque–Ahmad software reliability model ($m_{HA}$). The Hausdorff distance is $d=11.1591$ with $t_0=13.1706$. Consistent with Theorem 1, the lower and upper bounds were estimated as $d_l=6.68992$ and $d_r=22.3695$, respectively. Similarly, an analogous calculation for the Song–Kim–Chang software reliability model ($m_{SKC}$) yielded a Hausdorff distance of $d=9.56518$ at $t_0=13.1212$. The corresponding estimates, as established by Theorem 2, are $d_l=6.1097$ and $d_r=20.8072$.

The analysis of Hausdorff saturation demonstrates the stability of the models under consideration. Based on the combined results of standard goodness-of-fit tests and the Hausdorff saturation, we can conclude that the Haque–Ahmad software reliability model $m_{HA}$ is the most suitable for forecasting future fault occurrences. This is illustrated in Figure 10.

Example 4.

Eclipse Foundation platform is a major open-source software ecosystem. It is primarily Java-based with a plugin architecture supporting multiple programming languages such as C, C++, PHP, Perl, JavaScript, Python and others.

Table 11. Detected faults in Eclipse.

months	1	2	3	4	5	6	7	8	9
Cumulative Failures	177	328	472	594	726	872	975	1070	1213
months	10	11	12	13	14	15	16	17	18
Cumulative Failures	1306	1409	1537	1609	1698	1809	1848	1876	1927
months	19	20	21	22	23	24	25	26	27
Cumulative Failures	1946	1955	1968	1978	1981	1990	1991	2000	2004

contains real defect reports submitted by users and developers during routine software operation. The defect reports were obtained from the public Eclipse Bugzilla repository. It covers the period of 27 months (from 01.2023 to 03.2025), during which ten releases (4.26–4.35) were issued.

The estimated parameters of the analyzed models are given in

Table 12. Parameter estimation for Eclipse dataset.

Model	a	b	$\mathit{\alpha }$	$\mathit{\beta }$	k
Goel–Okumoto	$2355.22$	$0.0838$	−	−	−
Delayed S-shaped	$2041.94$	$0.2340$	−	−	−
Inflection S-shaped	$2074.6$	$0.1886$	−	$1.9766$	−
Haque–Ahmad	$2057.26$	$0.4349$	$0.7453$	$13.5707$	−
Song–Kim–Chang	$1980.13$	$0.3988$	−	−	$318.35$

.

Table 13. Comparison of criteria values for Eclipse software dataset.

Model	Goel–Okumoto	Delayed S-Shaped	Inflection S-Shaped	Haque–Ahmad	Song–Kim–Chang
AIC	309.725	307.907	274.875	267.41	300.433
BIC	313.612	311.795	280.059	273.889	305.616
R-Squared	0.998173	0.998292	0.999535	0.999674	0.998802
Adjusted R-Squared	0.998027	0.998155	0.999477	0.999618	0.998652

summarizes the quantitative performance assessment based on standard goodness-of-fit metrics. A visual comparison is shown in Figure 11, where the cumulative observed failures (black dots) are plotted alongside the estimated mean value functions of the proposed software reliability models.

The corresponding values of the Hausdorff saturation for the Haque–Ahmad software reliability model $m_{HA}$ are $d=21.453$ with $t_0=7.45499$ with estimates $d_l=7.25165$ and $d_r=40.9566$, respectively. Analogous for the Song–Kim–Chang software reliability model $m_{SKC}$ we have Hausdorff distance $d=14.5305$ at $t_0=7.65121$ with corresponding estimates $d_l=6.2984$ and $d_r=36.2197$.

Based on the Haque–Ahmad software reliability model, Figure 12 gives the predicted with open (unfilled) points and actual number solid black points of software faults.

3. Software Reliability Assessment Tool

Although software reliability growth models can be implemented manually using specialized mathematical software, this process is often time-consuming and demands a high level of expertise. Most software engineers are reluctant to implement these models due to limited awareness, a lack of understanding of the underlying mathematical concepts or time constraints. The development of software engineering tools has been driven by the objective of bridging the gap between theoretical academic research and its practical application. In [39] Xiao, Okamura and Dohi present a review of over 40 software reliability assessment tools. A review of five open-source tools, SFRAT (https://github.com/LanceFiondella/srt.core, accessed on 27 July 2025), SRATS (https://github.com/SwReliab/SRATS2010, accessed on 27 July 2025), RSRAT (https://github.com/SwReliab/Rsrat, accessed on 27 July 2025), PHSRM (https://github.com/okamumu/phsrm, accessed on 27 July 2025) and SMERFS (https://www.slingcode.com/smerfs/, accessed on 27 July 2025), can be found in [40]. Some examples of other tools are SFRAT [41], C-SFRAT [42] and C SFRAT [43]. The basic difference among the tools lie in how they collect data, which software reliability models are used and the programming languages used in their development.

Several tools support the full process of reliability analysis. For instance, STRAIT [44] is a free, open-source tool that collects issue report data directly from GitHub and supports nine software reliability models. It was developed using Java and R. Another example is RAAF-OSS [45], a reliability automatic assessment framework designed for open-source software and compatible with Jira. It is implemented in Python and supports six software reliability models.

We present a computational tool for assessing software reliability through the analysis of issue tracking patterns in GitHub repositories. The tool supports the full reliability analysis process and is designed to be accessible to users without expert knowledge.

It is a dynamic Wolfram Language environment that directly interfaces with GitHub’s API. To the best of the authors’ knowledge, this is the first tool of its kind.

The tool processes five main steps: data collection, data normalization, model fitting, visualization and evaluation.

Data Collection: In open-source software (OSS) projects, public repositories serve both as development platforms and as sources for issue data collection. Issues are retrieved from the GitHub repository, and their creation and closure timestamps are captured. The dataset used comprises closed issues from GitHub repositories, obtained via the GitHub REST API v3. Data points are organized as $\{t,N(t\left)\right\}$ pairs, where t is the time in days since the first reported issue and $N\left(t\right)$ is the cumulative number of issues up to time t. This transformed dataset emulates software reliability growth testing data, where the time of discovery of software faults (represented by issues) is modeled over calendar time. This modeling is performed in the absence of formal test-phase data, which is often unavailable in open-source projects. Our tool automates the collection and storage of issue reports directly from GitHub using the project URL. Users only need to provide the repository owner and name.

Data Normalization: Timestamps are normalized to analyze the progression of issues over time. The raw timestamps undergo the following processing steps: 1. conversion to absolute time values; 2. normalization relative to the creation date of the first issue; and 3. binning into daily intervals for consistent time series analysis. In the interactive interface, the user has opportunity to choose amount of data to include, choosing either all issue reports or only a portion using a dynamic slider. Another option allows the user to set the maximum number of iterations to be used during the model fitting process.

Accurate parameter estimation is essential for the effective application of software reliability growth models. Since software failures exhibit nonlinear behavior, merely fitting test data to models is insufficient. Instead, optimization techniques are required to identify the most suitable parameters that ensure reliable prediction accuracy. Our tool allows users to select the most appropriate optimization method for parameter estimation, including

• “ConjugateGradient”—nonlinear conjugate gradient;
• “Gradient” —gradient descent;
• “LevenbergMarquardt”—Gauss–Newton method for least squares;
• “Newton”—Newton method;
• “QuasiNewton” —quasi-Newton BFGS;
• “InteriorPoint”—interior point method;
• “NMinimize”—optimization that always attempts to find a global minimum;
• “Automatic”—automatic default method.

Model Fitting: Various reliability growth models, such as Goel–Okumoto, delayed S-shaped, inflection S-shaped, Haque–Ahmad and Song–Kim–Chang, are applied to the issue data. Each model can be selected by the user. We use CAS Wolfram Mathematica to fit data with high precision. The model fitting function NonlinearModelFit [36] returns a FittedModel object, which provides convenient access to both model predictions and diagnostic information without the need to refit the model. One of the key advantages is its ability to support direct evaluation at specific input values, as well as queries for parameter estimates, residuals and other diagnostic metrics. Once a model is chosen, the parameter estimations are displayed in a table view. The table presents the best-fit parameter information, including values for “Estimate,” “Standard Error,” “t-Statistic” and “p-Value.”

Visualization: Plots are generated to visualize how the selected models fit the actual issue data. These plots allow for analysis of issue trends and provide a clear graphical representation of model performance, helping users choose the most appropriate model.

Evaluation: Criteria values for model comparison present some properties that measure the goodness-of-fit of the considered models in table view. The presented criteria are as follows: “AIC” (Akaike information criterion), “BIC” (Bayesian information criterion), “R-Squared” (coefficient of determination $R^2$) and “Adjusted R-Squared” ($R^2$ adjusted for the number of model parameters). In each row of the table, the most suitable value is highlighted in blue for the selected software reliability models—specifically, the minimum values for AIC and BIC, and the maximum values for R-squared and adjusted R-squared. This facilitates the identification of the model that demonstrates the best fit to the observed data.

The main advantages provided by the interactive interface of our tool are as follows:

• Dynamic model fitting with adjustable sample sizes;
• Support for multiple optimization algorithms;
• Visual comparison of model fits;
• Parameter estimates with statistical significance;
• Ease of use, even for users without expert knowledge.

We demonstrate capabilities of our tool by using the MPAndroidChart repository as a case study (see Figure 13). MPAndroidChart is a powerful and user-friendly charting library for Android https://github.com/PhilJay/MPAndroidChart (accessed on 27 July 2025). This open-source project belongs to Philipp Jahoda and is licensed under the Apache License, Version 2.0. Its repository issue tracker is dedicated exclusively to bug reports and feature requests.

Analysis of the MPAndroidChart repository revealed a decreasing trend in opened issues over time, suggesting a reduction in the introduction of new defects. Also we observe a high closure rate. A significant proportion of opened issues have been resolved and closed, indicating active maintenance and responsiveness from the development community.

In our software reliability tool, we provide a comparative analysis of all five models. Based on parameter estimations and standard goodness-of-fit tests, we conclude that the Song–Kim–Chang software reliability model offers the most accurate predictions for the number of future faults.

The decreasing number of opened issues, coupled with a high closure rate, suggests that the MPAndroidChart project is maturing and becoming more reliable over time. This aligns with findings from previous studies indicating that OSS projects exhibit reliability growth patterns similar to closed source projects. Moreover, the active engagement of the community in resolving issues contributes to the project’s reliability. As more users and developers participate in identifying and fixing bugs, the software becomes more robust.

By analyzing issue tracking data, we can gain valuable insights into the reliability and maturity of OSS projects. The trends observed in the MPAndroidChart repository demonstrate that a decreasing rate of new issues and a high resolution rate are indicative of a reliable and well-maintained project.

4. Discussion and Conclusions

Open-source projects have become integral to modern software development, offering transparency and collaborative improvement. However, assessing the their reliability remains a challenge due to the decentralized nature of their development. The paper presents an analysis of the so-called Hausdorff saturation characteristic for two software reliability models based on the logistic growth model. The main goal is to propose an additional criterion for model selection. Thus, this distance acts as a geometric indicator of software reliability maturity, helping assess how close we are to a stable, dependable system. This methodology can be successfully applied to other models as well. We considered four numerical examples using fault detection datasets from two different open-source projects. In the presented two examples, the proposed metric supports the results obtained from the previous metrics, including standard goodness-of-fit tests. The question of the usefulness of the proposed Hausdorff saturation criterion remains open and will be the subject of future research, particularly in the context of large datasets. We present a new dynamic tool for evaluating the reliability of open-source software through the analysis of issue tracking dynamics. The tool supports the complete reliability assessment process in real time and is designed for use by individuals without specialized expertise. To demonstrate its applicability, we include a case study based on an open-source project. Furthermore, the tool is easily extensible, allowing for the integration of additional software reliability growth models.

The applicability of SRGMs to real-world projects is heavily dependent on the characteristics of software repositories and the adopted development methodology. For OSS in particular, development typically follows agile practices rather than classical waterfall models. This approach, characterized by short iterations, continuous feedback and the parallel execution of coding and testing, aligns perfectly with the decentralized nature of OSS communities. Consequently, the reliability assessment of OSS must account for the heterogeneity of the reported data, which includes not only genuine defects but also duplicates, enhancement requests and rejected reports. This reality distinguishes OSS reliability analysis from that of traditional systems, where testing is a more formalized and managed process. Some recent studies [46,47,48] emphasize the need to integrate repository mining and empirical software engineering into SRGM-based analysis, while foundational works [49] continue to provide the guiding theoretical framework. Future research may explore several enhancements to improve software reliability assessment. We can consider incorporating additional metrics. While this study focuses on issue opening and closing trends, integrating other indicators such as pull request activity, code churn, commit frequency and test coverage could offer a more comprehensive view of software reliability and overall project health. One promising direction is analyzing the roles and behaviors of contributors to gain insight into how community dynamics impact the evolution of reliability in open-source projects. Another avenue involves developing a real-time monitoring dashboard that continuously tracks issue dynamics and estimates software reliability, thereby supporting maintainers and contributors in evaluating project status on an ongoing basis. Another promising direction involves correlating issue trends with external quality indicators, such as user ratings, download counts or real-world usage in production systems, to strengthen the link between internal repository data and perceived reliability. Lastly, predictive modeling based on historical issue data could enable the forecasting of future reliability trends, issue arrival rates and resolution times, ultimately aiding in resource planning and risk assessment allocation for OSS projects.