OSS Project Stability Assessment Support Tool Considering EVM Based on Wiener Process Models

Abstract

The method of earned value management is traditionally applied to the progress assessment of actual software projects in many IT companies. Also, Open Source Software (OSS) are used under the various situations because of cost reduction, standardization, and quick delivery. In particular, many OSS are developed and managed by using fault big data recorded on the bug tracking systems. Several research papers in terms of OSS reliability assessment have been published in the past. The fault is caused by the poor handling of effort control. Therefore, we can make a stable and safety operation for OSS system if the OSS project is appropriately managed by using the software effort. The OSS effort estimation model by using a conventional stochastic differential equation model is discussed in this paper. Then, we propose the optimal maintenance problem based on the earned value requirement. In particular, we develop the OSS project stability support tool. Furthermore, several performance illustrations of the developed software tool are shown by using the effort data under actual OSS project.

1. Introduction

Traditionally, the method of Earned Value Management (EVM) [1] is used in the progress assessment of actual software projects under various IT companies. Also, many Open Source Software (OSS) are managed in OSS project by using fault big data recorded on the bug tracking systems. Recently, OSS are used under the various situations, because OSS are useful for many users to make cost reduction, standardization, and quick delivery. Moreover, the methods for reliability assessment of OSS have been proposed by several researchers [2,3,4]. Also, our research group has proposed and discussed the method of reliability assessment for the proprietary software and OSS [5,6,7,8]. However, the researches focused on the software effort expenditures of OSS have been not proposed.

Moreover, the effective methods assisting dynamic testing management for OSS development paradigm as typified by the open source project are few in number [9,10,11]. Also, many OSS are used in various area such as cloud computing, mobile cloud, and server software. For example, several research papers [12,13,14,15,16] have been proposed in the area of cloud computing. However, these papers focus on the security, service optimization, secure control, resource allocation technique, etc. The research papers in terms of project management for clouds by using OSS are few in number.

In particular, it is important to appropriately manage the quality according to the progress status of OSS project, because the software failure is caused by the poor handling of effort control as shown in Figure 1. Figure 1 shows the difference in the data sets between the proposed method and the existing reliability assessment one. In particular, our method can totally assess the development phase by using the effort data. Also, the appropriate control of management effort for OSS will indirectly link to the quality, reliability, and cost. Moreover, it is useful for OSS project managers to decide the maintenance time.

In this paper, we discuss the OSS effort estimation model by using the stochastic differential equation model. Also, this paper discusses a useful method of OSS project management based on the earned value analysis considering the irregular fluctuation of performance resulting from the characteristics of OSS development and management. Moreover, we formulate the optimal maintenance problem based on the proposed effort estimation model and extend it to the optimization problem with requirements specification based on the proposed model by stochastic differential equation. Then, we find the optimum maintenance time by minimizing the total expected software maintenance effort by using the cost performance index. In particular, we develop the OSS project stability support tool considering EVM. Furthermore, several performance illustrations of the OSS project stability support tool are shown by using the effort data in actual OSS project.

2. Wiener Process Models for Effort Estimation

Considering the characteristic of the operation phase in OSS projects, the time-dependent effort expenditure phenomenon of operation phase keeps an irregular state, because there is variability among the levels of developer and skill through version-upgrade, and the OSS are developed and maintained by several developers and users. Then, the time-dependent effort expenditure phenomenon of maintenance phase becomes unstable. Thus, the operation phases of many OSS projects are influenced from external factors by triggers such as the difference of skill of developers and users, time lags of development and maintenance activities. From above points, we discus the stochastic differential equation modeling for the OSS project. Then, let $\mathsf{\Psi }\left(t\right)$ be the cumulative maintenance effort up to operational time $t(t\ge 0)$ in the OSS project. Suppose that $\mathsf{\Psi }\left(t\right)$ takes on continuous real values. Since the estimated maintenance effort are observed during the operational phase of the OSS project, $\mathsf{\Psi }\left(t\right)$ gradually increases as the operational procedures go on. Based on the software reliability growth modeling approach [5,6,7,8], the following linear differential equation in terms of maintenance effort can be formulated:

$$\frac{d\mathsf{\Psi }\left(t\right)}{dt}=\beta \left(t\right)\{\alpha -\mathsf{\Psi }\left(t\right)\},$$

(1)

where $\beta \left(t\right)$ is the increase rate of maintenance effort at operational time t and a non-negative function, and $\alpha$ means the estimated maintenance effort required until the end of operation.

Therefore, we extend Equation (1) to the following stochastic differential equation with Brownian motion [17]:

$$\frac{d\mathsf{\Psi }\left(t\right)}{dt}=\{\beta \left(t\right)+\sigma \nu \left(t\right)\}\{\alpha -\mathsf{\Psi }\left(t\right)\},$$

(2)

where $\sigma$ is a positive constant representing a magnitude of the irregular fluctuation, and $\nu \left(t\right)$ a standardized Gaussian white noise. By using Itô’s formula [18], we can obtain the solution of Equation (2) under the initial condition $\mathsf{\Psi }\left(0\right)=0$ as follows:

$$\mathsf{\Psi }\left(t\right)=\alpha [1-\exp \{-{\int }_0^t\beta \left(s\right)ds-\sigma \omega \left(t\right)\}],$$

(3)

where $\omega \left(t\right)$ is one-dimensional Wiener process which is formally defined as an integration of the white noise $\nu \left(t\right)$ with respect to time t. Moreover, we define the increase rate of maintenance effort in case of $\beta \left(t\right)$ defined as [19]:

$${\int }_0^t\beta \left(s\right)ds\doteq \frac{\frac{d{F}_{*}\left(t\right)}{dt}}{\alpha -F_{*}\left(t\right)}.$$

(4)

In this paper, we assume the following equations based on software reliability models $F_{*}\left(t\right)$ as the cumulative maintenance effort function of the proposed model:

$$\begin{array}{}\left(5\right)& \hfill F_e\left(t\right)& \equiv & \alpha (1-e^{-\beta t}),\hfill \left(6\right)& \hfill F_s\left(t\right)& \equiv & \alpha \{1-(1+\beta t)e^{-\beta t}\},\hfill \end{array}$$

where ${\mathsf{\Psi }}_e\left(t\right)$ means the cumulative maintenance effort for the exponential software reliability growth model with $F_e\left(t\right)$. Similarly, ${\mathsf{\Psi }}_s\left(t\right)$ is the cumulative maintenance effort for the delayed S-shaped software reliability growth model with $F_s\left(t\right)$.

Therefore, the cumulative maintenance effort up to time t is obtained as follows:

$$\begin{array}{}\left(7\right)& \hfill {\mathsf{\Psi }}_e\left(t\right)& =& \alpha [1-\exp \{-\beta t-\sigma \omega \left(t\right)\}],\hfill \left(8\right)& \hfill {\mathsf{\Psi }}_s\left(t\right)& =& \alpha [1-(1+\beta t)\exp \{-\beta t-\sigma \omega \left(t\right)\}].\hfill \end{array}$$

In this model, we assume that the parameter $\sigma$ depends on several noises by external factors from several triggers in OSS projects. Then, the expected cumulative maintenance effort spent up to time t are respectively obtained as follows:

$$\begin{array}{}\left(9\right)& \hfill \mathrm{E}\left[{\mathsf{\Psi }}_e\left(t\right)\right]& =& \alpha [1-\exp \{-\beta t+\frac{{\sigma }^2}{2}t\}],\hfill \left(10\right)& \hfill \mathrm{E}\left[{\mathsf{\Psi }}_s\left(t\right)\right]& =& \alpha [1-(1+\beta t)\exp \{-\beta t+\frac{{\sigma }^2}{2}t\}].\hfill \end{array}$$

Similarly, the sample path of fixing effort required for OSS maintenance until the end of operation time t is obtained as follows:

$$\begin{array}{}\left(11\right)& \hfill {\mathsf{\Psi }}_{re}\left(t\right)& =& \alpha \exp \{-\beta t-\sigma \omega \left(t\right)\},\hfill \left(12\right)& \hfill {\mathsf{\Psi }}_{rs}\left(t\right)& =& \alpha (1+\beta t)\exp \{-\beta t-\sigma \omega \left(t\right)\}.\hfill \end{array}$$

Then, the expected fixing effort required for OSS maintenance until the end of operation time t is respectively obtained as follows:

$$\begin{array}{}\left(13\right)& \hfill \mathrm{E}\left[{\mathsf{\Psi }}_{re}\left(t\right)\right]& =& \alpha \exp \{-\beta t+\frac{{\sigma }^2}{2}t\},\hfill \left(14\right)& \hfill \mathrm{E}\left[{\mathsf{\Psi }}_{rs}\left(t\right)\right]& =& \alpha (1+\beta t)\exp \{-\beta t+\frac{{\sigma }^2}{2}t\}.\hfill \end{array}$$

3. Assessment Measures for EVM

The method of EVM has been used for the project management under various IT companies. Generally, the EVM is applied to commonly software development projects [20]. However, it is difficult to directly apply the EVM to the actual OSS project, because the development cycle of OSS project is different from the traditional software development paradigm. Considering the characteristic of the OSS operation phase, the OSS development project is managed by using the bug tracking system. The method of EVM will be helpful for OSS managers to assess QCD (Quality, Cost, Delivery) of OSS [21]. This paper proposes the method of earned value analysis for OSS projects by using the data sets obtained from bug tracking system.

Considering the earned value analysis for OSS, we assume the following terms as the EVM for OSS:

▸

Actual Cost (AC): Cumulative maintenance effort up to operational time t considering the reporter and assignee.

▸

Earned Value (EV): Cumulative maintenance effort up to operational time t considering the reporter.

▸

Cost Variance (CV): Fixing effort required for OSS maintenance up to operational time t, $\mathrm{E}\left[{\mathsf{\Theta }}_e\left(t\right)\right]$ and $\mathrm{E}\left[{\mathsf{\Theta }}_s\left(t\right)\right]$.

▸

Cost Performance Index (CPI): $\mathrm{E}\left[CP{I}_e\left(t\right)\right]$ and $\mathrm{E}\left[CP{I}_s\left(t\right)\right]$ obtained from AC and EV.

▸

Estimate at Completion (EAC): $\mathrm{E}\left[EA{C}_e\left(t\right)\right]$ and $\mathrm{E}\left[EA{C}_s\left(t\right)\right]$ obtained from AC, EV, CPI, and BAC.

▸

Estimate to Completion (ETC): $\mathrm{E}\left[ET{C}_e\left(t\right)\right]$ and $\mathrm{E}\left[ET{C}_s\left(t\right)\right]$ obtained from AC, EV, CPI, BAC, and EAC.

▸

Budget at Completion (BAC): Planned Value (PV) in the end point as the specified goal of OSS project.

Then, the expected fixing effort required for OSS maintenance up to operational time t in case of ${\mathsf{\Psi }}_e\left(t\right)$ and ${\mathsf{\Psi }}_s\left(t\right)$ can be formulated as:

$$\begin{array}{ccc}\hfill \mathrm{E}\left[{\mathsf{\Theta }}_e\left(t\right)\right]& =& \mathrm{E}\left[{\mathsf{\Psi }}_{re}^r\left(t\right)\right]-\mathrm{E}\left[{\mathsf{\Psi }}_{re}^{ra}\left(t\right)\right],\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill \mathrm{E}\left[{\mathsf{\Theta }}_s\left(t\right)\right]& =& \mathrm{E}\left[{\mathsf{\Psi }}_{rs}^r\left(t\right)\right]-\mathrm{E}\left[{\mathsf{\Psi }}_{rs}^{ra}\left(t\right)\right],\hfill \end{array}$$

(16)

where $\mathrm{E}\left[{\mathsf{\Psi }}_{re}^{ra}\left(t\right)\right]$ and $\mathrm{E}\left[{\mathsf{\Psi }}_{rs}^{ra}\left(t\right)\right]$ are the maintenance effort considering the reporter and assignee until the end of operation. Also, $\mathrm{E}\left[{\mathsf{\Psi }}_{re}^r\left(t\right)\right]$ and $\mathrm{E}\left[{\mathsf{\Psi }}_{rs}^r\left(t\right)\right]$ mean the maintenance effort considering the reporter until the end of operation.

Similarly, the sample path of fixing effort required for OSS project maintenance up to operational time t in case of ${\mathsf{\Psi }}_e\left(t\right)$ and ${\mathsf{\Psi }}_s\left(t\right)$ are given by

$$\begin{array}{ccc}\hfill {\mathsf{\Theta }}_e\left(t\right)& =& {\mathsf{\Psi }}_{re}^r\left(t\right)-{\mathsf{\Psi }}_{re}^{ra}\left(t\right),\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\mathsf{\Theta }}_s\left(t\right)& =& {\mathsf{\Psi }}_{rs}^r\left(t\right)-{\mathsf{\Psi }}_{rs}^{ra}\left(t\right).\hfill \end{array}$$

(18)

The zero point of fixing effort $\mathrm{E}\left[{\mathsf{\Theta }}_e\left(t\right)\right]$ and $\mathrm{E}\left[{\mathsf{\Theta }}_s\left(t\right)\right]$ mean the starting point of surplus effort. On the other hand, the zero point of fixing effort $\mathrm{E}\left[{\mathsf{\Theta }}_e\left(t\right)\right]$ and $\mathrm{E}\left[{\mathsf{\Theta }}_s\left(t\right)\right]$ is the end point of effort required in OSS project. Therefore, the OSS project managers will be able to judge the necessity of fixing effort and stability of OSS from the starting point of surplus effort.

Moreover, we can obtain the CPI by using the following equations:

$$\begin{array}{ccc}\hfill \mathrm{E}\left[CP{I}_e\left(t\right)\right]& =& \frac{\mathrm{E}\left[{\mathsf{\Psi }}_{re}^r\left(t\right)\right]}{\mathrm{E}\left[{\mathsf{\Psi }}_{re}^{ra}\left(t\right)\right]},\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill \mathrm{E}\left[CP{I}_s\left(t\right)\right]& =& \frac{\mathrm{E}\left[{\mathsf{\Psi }}_{rs}^r\left(t\right)\right]}{\mathrm{E}\left[{\mathsf{\Psi }}_{rs}^{ra}\left(t\right)\right]}.\hfill \end{array}$$

(20)

Similarly, the sample path of CPI in case of ${\mathsf{\Psi }}_e\left(t\right)$ and ${\mathsf{\Psi }}_s\left(t\right)$ are given by

$$\begin{array}{ccc}\hfill CP{I}_e\left(t\right)& =& \frac{{\mathsf{\Psi }}_{re}^r\left(t\right)}{{\mathsf{\Psi }}_{re}^{ra}\left(t\right)},\hfill \end{array}$$

(21)

$$\begin{array}{ccc}\hfill CP{I}_s\left(t\right)& =& \frac{{\mathsf{\Psi }}_{rs}^r\left(t\right)}{{\mathsf{\Psi }}_{rs}^{ra}\left(t\right)}.\hfill \end{array}$$

(22)

In particular, the CPI is very important for OSS project managers to assess the stability of OSS project.

4. Optimal Maintenance Time Based on Wiener Process Models Considering CPI as Requirement Specification

Several optimal software release problems based on software reliability models have been proposed by several researchers [22]. Considering the characteristics of the effort phenomena in OSS project, it is interesting for the software developers to predict and estimate the time when we should stop the operation in order to OSS version upgrade efficiently.

In the operation of OSS, we discuss the optimal maintenance problem by minimizing the maintenance effort. Then, we define the following effort rate parameters:

$e_1$:

the importance rate of maintenance effort needed to operate OSS,

$e_2$:

the operation effort per unit time during the operation,

$e_3$:

the importance rate of maintenance effort after the upgrade task such as major version upgrade.

Then, the expected maintenance in the operation of OSS can be formulated as:

$$E_1\left(t\right)=e_1\mathrm{E}\left[{\mathsf{\Psi }}_{*}^{ra}\left(t\right)\right]+e_{2}t.$$

(23)

Also, the expected software maintenance effort after the maintenance of OSS is represented as follows:

$$E_2\left(t\right)=e_3\left\{\alpha -\mathrm{E}\left[{\mathsf{\Psi }}_{*}^{ra}\left(t\right)\right]\right\}.$$

(24)

Consequently, from Equations (23) and (24), the total expected software maintenance effort during the specified period such as the specified version is given by

$$E\left(t\right)=E_1\left(t\right)+E_2\left(t\right).$$

(25)

The optimum maintenance time $t^{*}$ is obtained by minimizing $E\left(t\right)$ in Equation (25).

It is important for the OSS managers to find the optimal maintenance time considering the total expected software maintenance effort simultaneously with requirement specification such as CPI. This section focuses on the optimal maintenance problem with requirement specification [23]. Also, we consider CPI as the requirement specification. Then, the optimal maintenance problem with requirement specification is defined as follows:

$$\begin{array}{c}\underset{t}{\min } E\left(t\right)\hfill \\ subject to CP{I}_{*}\left(t\right)\ge 1.0\hfill \end{array}$$

(26)

Generally, the trend of stability in OSS project is unstable if the estimated value of CPI is smaller than 1.0. On the other hand, the trend of stability in OSS project becomes stable if the estimated value of CPI is larger than 1.0.

5. Framework of OSS Project Stability Support tool Considering EVM

The specification requirements of the OSS project stability support tool are shown as follows:

▸

This tool should be operated by clicking the mouse button and typing on the keyboard to input the data through GUI system. In particular, the user experience design is adopted as the important element of our tool.

▸

Statistical computing R for statistical computing and ggplot2 library of R package should be implemented to the source program. This tool is developed as a stand–alone of multi–platform. Also, this tool operates as Web application via the Web server.

▸

The method of maximum-likelihood is used as the estimation of unknown parameters in our model. Moreover, the optimum maintenance time is estimated by using the method of general–purpose optimization based on Nelder–Mead.

▸

This tool treats the proposed effort estimation model based on stochastic differential equation, and illustrate the cumulative maintenance effort at arbitrary time t, the CPI, and the total software maintenance effort.

It is well known the following keywords in terms of user experience design:

▸

Visual Design

▸

Information Architecture

▸

Information

▸

Structuring, Organization and Labeling

▸

Finding and Managing

▸

Interaction Design

▸

Usability

▸

Accessibility

▸

Human-Computer Interaction

We focus on the “Visual Design” and “Interaction Design” as the user experience design. “Visual Design” and “Interaction Design” can easily implement by using CSS3 and JavaScript programming language, because CSS3 and JavaScript includes the various effect components. We develop the OSS project stability support tool based on Visual Design” and “Interaction Design” by using the animation effects of CSS3 and JavaScript.

Moreover, the plotly R library (Plotly R Library) is used in the developed tool. In particular, the developed application software is implemented as the stand-alone application of multi-platform by using the latest technology of NW.js.

Furthermore, the procedures of EVM built into the developed tool are shown as follows:

• This tool processes the data file in terms of the effort data in the OSS project.
• Using the effort data obtained from the actual OSS project, we analyze the data for input data.
• This tool estimates the unknown parameters included in our model. Then, the unknown parameters are estimated by using the method of maximum–likelihood.
• This tool illustrates the estimated cumulative maintenance effort, fixing effort required for OSS maintenance until the end of operation, total expected software maintenance effort, and CPI as EVM measures.
• We focus on the maintenance time based on Wiener process models considering CPI as requirement specification. In particular, the total software effort and the optimal software maintenance time minimizing the effort for our model are plotted in our tool.

6. Performance Illustrations Based on Developed Tool

This section focuses on the Apache Tomcat known as the OSS developed under Apache Software Foundation [24]. The project data used in this paper is obtained from the bug tracking system on the website of Apache Software Foundation. In particular, the effort data of version 7 after the year 2011 are applied in order to understand the maintenance effort in Apache Tomcat.

Figure 2 is the main screen of the developed OSS project stability support tool. Moreover, Figure 3 and Figure 4 show the estimated cumulative maintenance effort up to time t in cases of ${\mathsf{\Psi }}_e\left(t\right)$, $\mathrm{E}\left[{\mathsf{\Psi }}_e\left(t\right)\right]$, ${\mathsf{\Psi }}_s\left(t\right)$, and $\mathrm{E}\left[{\mathsf{\Psi }}_s\left(t\right)\right]$. From Figure 3 and Figure 4, we found that the estimated maintenance effort required at the end of operation in cases of ${\mathsf{\Psi }}_e\left(t\right)$ and $\mathrm{E}\left[{\mathsf{\Psi }}_e\left(t\right)\right]$ are negatively estimated than the case of ${\mathsf{\Psi }}_s\left(t\right)$ and $\mathrm{E}\left[{\mathsf{\Psi }}_s\left(t\right)\right]$. Similarly, Figure 5 and Figure 6 show the estimated maintenance effort required until the end of operation in cases of ${\mathsf{\Psi }}_{re}\left(t\right)$, $\mathrm{E}\left[{\mathsf{\Psi }}_{re}\left(t\right)\right]$, ${\mathsf{\Psi }}_{rs}\left(t\right)$, and $\mathrm{E}\left[{\mathsf{\Psi }}_{rs}\left(t\right)\right]$. In particular, the noise in case of S-shaped type model becomes small according to the operation procedures go on. We found that the OSS project condition is stable after 2000 days.

Furthermore, we show several numerical examples based on the optimal maintenance problems which are discussed in Section 4. Figure 7 and Figure 8 show the estimated total expected software maintenance effort in cases CPI $CP{I}_e$ and $CP{I}_s$ as requirement specification, respectively. For example, we find that the optimum maintenance time with stability requirement is derived as $t^{*}=11.2$ years in Figure 8. Then, we found that the estimated CPI is larger than 1.0 at the time of $t^{*}=11.2$ years. These results show that this OSS has need to make a major-upgrade after 11.2 years from release. Thus, our tool can support the stability and effort control considering the operational environment of OSS. The developed tool will be helpful as the assessment tool of the progress of the OSS project in operation phase.

7. Conclusions

In the past, several conventional methods of OSS reliability assessment have been proposed. Usually, the fault data sets are used to assess the software reliability in the testing phase. However, the software fault is the effect generated by many fault-prone causes. In other words, the effort data sets become a cause of fault data sets. Therefore, it is difficult for the conventional methods to totally assess the development phase by using the effort data, because the conventional methods focus on the reliability assessment based on fault data. We can control the OSS project considering the project stability and reliability by using the software effort data sets. Thereby, the proposed method will lead to assess the safety of OSS systems developed under various OSS projects. Also, the appropriate control of management effort for OSS will indirectly link to the quality, safety, reliability, and cost reduction of OSS.

In this paper, we have discussed the method of earned value analysis based on the stochastic differential equation model. Moreover, the optimal maintenance problem based on maintenance effort for OSS have been discussed. In particular, we have defined the optimal maintenance problems considering the CPI as the stability and requirement specification. Also, the software tool based on the optimal maintenance problem considering the OSS project stability in order to decide the optimal maintenance time has been developed in this paper. Furthermore, a set of actual OSS effort data has been analyzed to show several performance illustrations of the software tool in terms of the progress analysis and optimum maintenance time considering stability for an OSS project. Then, we found that our method and tool can assess the stability and effort control considering the operational environment of OSS. The developed tool will be helpful as the assessment tool of the progress of the OSS project in operation phase.