Improving Software Defect Prediction in Noisy Imbalanced Datasets

Abstract

Software defect prediction is a popular method for optimizing software testing and improving software quality and reliability. However, software defect datasets usually have quality problems, such as class imbalance and data noise. Oversampling by generating the minority class samples is one of the most well-known methods to improving the quality of datasets; however, it often introduces overfitting noise to datasets. To better improve the quality of these datasets, this paper proposes a method called US-PONR, which uses undersampling to remove duplicate samples from version iterations and then uses oversampling through propensity score matching to reduce class imbalance and noise samples in datasets. The effectiveness of this method was validated in a software prediction experiment that involved 24 versions of software data in 11 projects from PROMISE in noisy environments that varied from 0% to 30% noise level. The experiments showed a significant improvement in the quality of datasets pre-processed by US-PONR in noisy imbalanced datasets, especially the noisiest ones, compared with 12 other advanced dataset processing methods. The experiments also demonstrated that the US-PONR method can effectively identify the label noise samples and remove them.

1. Introduction

Defects in software systems often lead to software errors and losses [1,2]. With the development of software technology, the scale and complexity of software have grown rapidly, and software testing to discover and correct software defects has become increasingly expensive. To optimize software testing schemes, software defect prediction (SDP) is used to identify modules in software systems that may contain defects and to improve the efficiency and performance of software testing. In recent years, with the rapid development of machine learning technology, many researchers have applied machine learning to software defect prediction [3] and have achieved remarkable results [4,5]. For machine learning to predict software defects, software defect metrics first need to be defined in order to characterize the distribution of defects. At present, CK metrics [6], process metrics [7,8,9,10], and network metrics [11,12,13,14,15] are widely used for this task.

The quality of the software defect dataset is one of the most important factors that affect the performance of SDP. In real-world software projects, defects usually exist in very few modules, so the defect samples in the software defect dataset are far smaller than the clean samples, resulting in serious class imbalance problems and reducing the performance of defect prediction [16]. Many scholars have proposed methods to solve the class imbalance problem, but these rebalancing methods are still limited in their application and performance [17]. Most of the existing dataset rebalancing methods are based on undersampling and oversampling [18]. Undersampling can effectively solve the class imbalance problem, but the instances discarded during undersampling may contain information useful or important for predicting defects [19]. In SDP, oversampling is often more effective than undersampling, and minority class sample generation methods such as SMOTE and MAHAKIL [20,21] can effectively alleviate the class imbalance problem, but these oversampling methods generate overfitting noise that can degrade the predictive performance of the model [22].

Another problem with software defect datasets involves the way labels on software samples are manually marked by testers or developers [23]. Samples may be erroneously labeled due to insufficient knowledge of the defects, the time difference between defect introduction and defect discovery, or new defects caused by defect correction [24]; this mislabeling introduces noise in the dataset, which deteriorates its quality and reduces the performance of defect prediction. At present, there are few studies on the noise of software defect datasets in the SDP field [25,26]. To reduce the noise in the dataset, using propensity score matching (PSM), a noise reduction method used in machine learning [27], may be a good idea, being a statistical method that achieves noise reduction by addressing the difference between the distribution of the observed data and the overall distribution.

This paper proposes a novel method for predicting software defects using data pre-processing. Named US-PONR—for Undersampling, Propensity-Score-Matching-Based Oversampling and Noise Reduction—this method aims to improve the quality of software defect datasets by addressing deficiencies caused by class imbalance and dataset noise. The method uses undersampling in data pre-processing to remove duplicate samples that result from multiple versions of the data. And the method uses propensity score matching (PSM) to oversample and reduce noise in the dataset, which alleviate the introduction of overfitting noise. To predict defects, an aggregated multi-classifier trained under a cross-validation (CV) scoring method was built to construct the predictor. We conducted experiments that used US-PONR to predict defects under different noise environment; the results show that the proposed method is better than the benchmark and SOTA data pre-processing methods, and they further demonstrate the ability of the proposed method to identify and remove label noise samples.

This paper demonstrates how US-PONR can make the following contributions to the field of software defect prediction:

(1)

US-PONR offers a new PSM-based method for data oversampling and noise reduction, which reduces the introduction of overfitting noise caused by the minority class sample generation.

(2)

It also offers a new method of data pre-processing for software defect prediction that uses a combination of undersampling, oversampling, and noise reduction.

(3)

US-PONR additionally performs SOTA in software defect prediction experiments under different noise environment settings, and the experiments demonstrate that the method can effectively identify and label noise samples and remove them.

The rest of this paper is organized as follows: Section 2 presents background information about software defect prediction; Section 3 presents the methodology the authors used to develop US-PONR; Section 4 describes the experiment the authors conducted to test the method, including the research questions and experiment settings; Section 5 reports the experiment’s results; Section 6 discusses the method’s limitations; and Section 7 spells out major conclusions.

2. Background

2.1. Software Defect Prediction (SDP) and Metrics

SDP is an increasingly important subject in software reliability research. Using historical data to link software metrics and defects, SDP supports the designing and testing of software by determining the defect tendency of software modules [28]. At present, SDP mainly uses machine learning algorithms to make binary classification judgments on whether a module is defective, and it is applied to software at different stages, including data acquisition, data processing, model training, and model evaluation. And a major focus of data collection is to characterize software features through metrics.

The earliest software metric applied to SDP measured software code and its complexity. In 1991, Chidamber and Kmerer proposed the famous CK metric element [6], which became one of the standard metric tuples in the SDP field. From the perspective of object-oriented design, the CK metric tuple comprehensively considers the factors affecting software, such as the number of code lines, the degree of class cohesion, and the relationship between classes. Researchers also proposed applying to SDP the metrics of the code development process [7,8,9,10] in order to address the macro-integrity of a software program and how its elements interact. Researchers also provided network metrics to measure codes and thus build a software network that can characterize software features. Many studies on network metrics have been conducted [11,12,13,14,15]. Later, Jin proposed a distance metric based on cost-sensitive learning for reducing class imbalance and better differentiation of defect/clean samples [29].

Besides software metrics, there are many studies that have applied machine learning for software defect prediction. Smoya removed clean samples surrounding defect samples using a filtering technique to enhance the performance of predicting software defects using SVMs [30]. Xu represented the source code as an augmented code property graph and trained a software defect predictor with a graph neural network [31]. Hanif trained a code pre-trained language model using defect datasets as the corpus for code defect domain tasks [32].

2.2. Class Imbalance

In machine learning, the quality of the dataset is one of the most important factors affecting performance. In the SDP field, dataset quality is mainly compromised by widespread serious imbalance and the introduction of noise caused by various factors.

Class imbalance is as a key problem in machine learning and data mining [33]. It refers to an imbalance in the proportion of a dataset’s different classes of instances. Since imbalanced defect datasets reduce the model’s ability to learn defects [1], database rebalancing methods are often used to process the original dataset. There are various methods for rebalancing datasets.

One method is undersampling, which balances the dataset by reducing samples from the majority class. The simplest form of undersampling is random undersampling, which balances the dataset by randomly reducing the samples in the majority class. A. Guzmán-Ponce proposed a two-stage under-sampling algorithm combining the DBSCAN clustering algorithm and graph-based procedure to face the class imbalance [34]. The disadvantage of undersampling is that it may cause the training data to lose important information from the majority of class samples [19].

Another method for rebalancing datasets is oversampling. Oversampling balances a dataset by adding a minority class of sample data. Random oversampling (ROS) randomly replicates minority class samples, but this can lead to severe overfitting. Most current research instead balances the dataset by generating minority class samples. This method of oversampling often develops sample generation strategies based on the common assumption in the machine learning community that closer instances are more similar than more distant ones [35]. One sample generation strategy, MAHAKIL, determines Mahalanobis distance to generate samples [21], but this method does not work when the number of instances in a few classes is less than their dimensionality, as this generates overly diverse data that reduces the model’s ability to find defects [36]. Another strategy, COSTE, generates samples based on complexity [37], but further investigation is needed to validate its assumption that there is less information about defects in complex samples. Mateusz discussed the impact of class imbalance in few-shot learning and pointed out the effectiveness of oversampling to rebalance datasets [38].

The SMOTE algorithm proposed by Chawla et al. in 2002 is one of the most commonly used oversampling methods in academia [20]; it generates minority class samples using k-NN spatial distance. However, this method may increase the risk of overfitting and also increase the false positive rate of prediction results [39]. Researchers have proposed variants of SMOTE to minimize these drawbacks [40]. Paria proposed a range-controlled SMOTE to alleviate the increasing of overlaps between different classes around the class boundaries caused by SMOTE [41]. Batista et al. [42] proposed combining SMOTE with TomekLinks (SMOTE_TomekLinks) and with the edited nearest neighbor rule (SMOTE_ENN). Han et al. [43] proposed Borderline-SMOTE. He et al. [44] added different weights for different minority instances (ADASYN). Douzas et al. applied k-means clustering [45] and self-organizing maps [46] to the SMOTE method (kmeans-SMOTE and SOMO). Lee et al. [47] added Gaussian random variables to the SMOTE synthesis sample process (Gaussian-SMOTE). Barua et al. [48] gave the minority sample weight based on its distance to the nearest majority sample (MWMOTE). Recently, Agrawl et al. [36] proposed an algorithm to automatically optimize the parameter combination in SMOTE: k = number of neighbors, m = number of synthetic examples to create, and r = power parameter for the Minkowski distance (SMOTUNED).

2.3. Noise Reduction

There are two main types of noise in software defect prediction: noise existing in the dataset itself and noise introduced into the dataset by data pre-processing techniques such as oversampling. Reducing both kinds of noise has become an important area of research [49]. Some researchers perform noise reduction by adding filters after oversampling or by changing the sample generation method. Hu et al. [50] changed the strategy of selecting nearest neighbor samples, which caused a denoising effect (MSMOTE). Luengo et al. [51] combined IPF filtering with SMOTE (SMOTE-IPF). Koziarski et al. [52] cleaned up the decision boundary and guided synthetic oversampling samples (CCR). In 2012, Ramentol et al. [53] applied rough set theory to SMOTE (SMOTE-RSB). In 2016, Ramentol et al. [54] applied fuzzy rough set theory to SMOTE (SMOTE-FRST-2T). Khoshgoftaar et al. [55] applied filters to SDP for noise reduction. In 2017, Rivera [27] introduced propensity score matching (PSM) to the balancing and noise reduction methods.

3. Materials and Methods

3.1. Framework

This section introduces the framework of US-PONR (or Undersampling, Propensity Score Matching-Based Oversampling, and Noise Reduction), which is illustrated in Figure 1. The US-PONR consists of three main steps: undersampling, PSM-based oversampling and noise reduction, and predictor construction. An overview of each step appears below, while a more in-depth discussion of each step follows in Section 3.2, Section 3.3 and Section 3.4.

In our method, after obtaining the defect datasets, the CK [6], process [7], and network metrics [11,13] were selected to characterize the source code samples, which contain 80 metrics.

Table 1. CK metrics.

Name	Description
WMC	Weighted methods per class
DIT	Depth of inheritance tree
NOC	Number of children
CBO	Coupling between object classes
RFC	Response for a class
LCOM	Lack of cohesion in methods
LOC	Lines of code

,

Table 2. Process metrics.

Name	Description
REVISIONS	Number of revisions of a file.
AUTHORS	Number of distinct authors that checked a file into the repository.
LOC_ADDED	Sum over all revisions of the lines of code added to a file.
MAX_LOC_ADDED	Maximum number of lines of code added for all revisions.
AVE_ LOC_ADDED	Average lines of code added per revision.
LOC_DELETED	Sum over all revisions of the lines of code deleted from a file.
MAX_LOC_DELETED	Maximum number of lines of code deleted for all revisions.
AVE_LOC_DELETED	Average lines of code deleted per revision.
CODECHURN	Sum of (added lines of code—deleted lines of code) over all revisions.
MAX_CODECHURN	Maximum CODECHURN for all revisions.
AVE_CODECHURN	Average CODECHURN per revision.

and

Table 3. Network metrics.

Name	Description	Name	Description
Funcount	The number of internal functions of the class node.	Katz_centrality	The relative influence of a node within a network.
Indegree	The total number of connections it points to other nodes.	Load_centrality	The fraction of all shortest paths that pass through that node.
Outdegree	The total number of connections other nodes point to it.	PageRank	A ranking of the nodes in the graph G based on the structure of the incoming links.
Insidelinks	The total number of connections within the internal functions of the node.	Average_neighbor_degree	The average of the neighborhood of each node.
Out_degree_centrality	The fraction of nodes its outgoing edges are connected to.	Number_of_cliques	The number of maximal cliques for each node.
In_degree_centrality	The fraction of nodes its incoming edges are connected to.	Core_number	The largest value k of a k-core containing that node.
Degree_centrality	The fraction of nodes it is connected to.	Brokerage	The number of pairs not directly connected.
Closeness_centrality	The reciprocal of the sum of the shortest path distances from v to all other nodes.	EffSize	Effective size of network.
Betweenness_centrality	The sum of the fraction of all-pair shortest paths that pass through node v.	Constraint	Measures how strongly a module is constrained by its neighbors.
Eccentricity	The maximum distance from v to all other nodes in G.	Hierarchy	Measures how the constraint measure is distributed across neighbors.
Communicability_centrality	A broader measure of connectivity, which assumes that information could flow along all possible paths between two nodes.	TwoStepReach	The percentage of nodes that are two steps away.

shows some of the metrics used, while the entire meta dataset can be found at https://github.com/buaaSoftwareReliabilityGroup/US-PONR (accessed on 18 September 2023). The first step in US-PONR is undersampling (US). After the source code is represented using metrics, the software data are undersampled to pre-adjust the degree of imbalance in the dataset, which alleviates the problem of class imbalance and overfitting of the dataset by reducing the number of repeated non-defective samples in the dataset. At this point in the process, the ratio of undersampling also needs to be determined. A more detailed account of undersampling in US-PONR is provided in Section 3.2.

The second step in US-PONR is PSM-based oversampling and noise reduction (PONR). Both of these tasks are performed using the PSM technique to obtain the final dataset. PONR is performed on the undersampled dataset by calculating the propensity score of each sample in the dataset and then by obtaining the nearest neighbor sample set of each sample based on propensity score matching. According to the distribution of the PSM-based nearest neighbor sample set, oversample synthesis is performed to balance the dataset. After oversampling, PSM is performed again to judge whether a sample is noise and to exclude noise samples. Section 3.3 covers the specific calculation for the propensity score of each sample and the detailed steps of PONR.

The third step in US-PONR is predictor construction. To obtain the defect predictor, an ensemble model aggregating multi-ML classifiers was trained by the cross-validation (CV) method, in order to select the most appropriate ML model for different datasets. Section 3.4 describes how to perform cross-validation and to aggregate the model in detail.

3.2. Undersampling

In datasets, the number of non-defective samples often far exceeds the number of defective samples. These datasets typically include duplicate non-defective samples brought by non-defective code files that have not been modified in each version.

In such cases, the dataset needs to be undersampled in order to reduce the degree of class imbalance and reduce the over-fitting of the model caused by repeated samples.

The initial dataset can be expressed by the following formula:

$${\mathrm{X}}_{\mathrm{origin}}=\left\{\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)|\mathrm{i}\in \left(1,\mathrm{n}\right),{\mathrm{y}}_{\mathrm{i}}\in \left\{0,1\right\},{\mathrm{x}}_{\mathrm{i}}={\left[{\mathrm{x}}_{\mathrm{i},1},{\mathrm{x}}_{\mathrm{i},2},\dots ,{\mathrm{x}}_{\mathrm{i},\mathrm{m}}\right]}^{\mathrm{T}}\right\}$$

(1)

where x_i is the feature vector of the i-th sample, y_i is the defect label of the i-th sample, n is the number of samples, and m is the feature dimension.

In the method we are proposing, undersampling is introduced to alleviate the imbalance in a dataset caused by duplicate codes in multiple versions. The process of undersampling is represented by Algorithm 1:

Algorithm 1: Undersampling

input$\mathrm{original} \mathrm{dataset} {\mathrm{X}}_{\mathrm{origin}}$$, \mathrm{imbalance} \mathrm{degree} {\mathrm{r}}_{\max }={\mathrm{y}}_{\mathrm{negetive}}/{\mathrm{y}}_{\mathrm{positive}}$$, \mathrm{step} {\mathrm{s}}_{\mathrm{us}}$

output$\mathrm{undersample} \mathrm{dataset} {\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$$[ ][ ] \mathrm{contains} {\mathrm{X}}_{\mathrm{us}}$$\mathrm{with} \mathrm{diff} {\mathrm{r}}_{\mathrm{us}}$

1. ${\mathrm{r}}_{\mathrm{us}}$: undersample ratio, initialize to 1

2. ${\mathrm{X}}_{\max }$, ${\mathrm{X}}_{\mathrm{d}}$, ${\mathrm{X}}_{\min }$ initialize to null

3. for ${\mathrm{x}}_{\mathrm{i}}$ in ${\mathrm{X}}_{\mathrm{origin}}$

4.     if ${\mathrm{y}}_{\mathrm{i}}=0$

5.         then if ${\mathrm{x}}_{\mathrm{i}}$ in ${\mathrm{X}}_{\max }$

6.             then ${\mathrm{X}}_{\mathrm{d}}$ ← ${\mathrm{x}}_{\mathrm{i}}$

7.         ${\mathrm{X}}_{\max }$ ← ${\mathrm{x}}_{\mathrm{i}}$

8.         end if

9.     else

10.        ${\mathrm{X}}_{\min }$ ← ${\mathrm{x}}_{\mathrm{i}}$

11.     end if

12. end for

13. do {

14.     r = ${\mathrm{the} \mathrm{number} \mathrm{of} \mathrm{samples} \mathrm{in} \mathrm{X}}_{\max }$/${\mathrm{the} \mathrm{number} \mathrm{of} \mathrm{samples} \mathrm{in} \mathrm{X}}_{\min }$

15.     while r ≤ ${\mathrm{r}}_{\mathrm{us}}$

16.         ${\mathrm{X}}_{\max }$ ← xj (random select from ${\mathrm{X}}_{\mathrm{d}}$)

17.     end while

18.     ${\mathrm{X}}_{\mathrm{us}}$ = ${\mathrm{X}}_{\max }$ + ${\mathrm{X}}_{\min }$

19.     ${\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$ ← ${\mathrm{X}}_{\mathrm{us}}$ and ${\mathrm{r}}_{\mathrm{us}}$

20.     ${\mathrm{r}}_{\mathrm{us}}$ += ${\mathrm{s}}_{\mathrm{us}}$

21. }while (${\mathrm{r}}_{\mathrm{us}}$ ≤ ${\mathrm{r}}_{\max }$)

22. return ${\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$

where ${\mathrm{r}}_{\max }$ is the imbalance degree of the dataset, ${\mathrm{X}}_{\max }$ are samples with defect-free labels in ${\mathrm{X}}_{\mathrm{origin}}$, ${\mathrm{X}}_{\min }$ are samples with defect labels in ${\mathrm{X}}_{\mathrm{origin}}$, ${\mathrm{X}}_{\mathrm{d}}$ are duplicate defect-free samples, the undersampling parameter ${\mathrm{r}}_{\mathrm{us}}$ is the ratio of non-defective samples to defective samples after undersampling, ${\mathrm{X}}_{\mathrm{us}}$ is the dataset undersampled under certain ${\mathrm{r}}_{\mathrm{us}}$, and ${\mathrm{s}}_{\mathrm{us}}$ is the search step for the undersampling ratio.

The algorithm searches through the steps in [1,${\mathrm{r}}_{\max }$] and then deletes the duplicated data in the dataset under each ${\mathrm{r}}_{\mathrm{us}}$. Setting different ${\mathrm{r}}_{\mathrm{us}}$ will alleviate the class imbalance problem to varying degrees, but setting small ${\mathrm{r}}_{\mathrm{us}}$ will lead to few samples and the loss of important data information. Therefore, it is necessary to optimize ${\mathrm{r}}_{\mathrm{us}}$. The optimization method for ${\mathrm{r}}_{\mathrm{us}}$ is introduced in Section 3.4.

After undersampling, the under sampled dataset ${\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$ is obtained, which contains the different ${\mathrm{r}}_{\mathrm{us}}$ values and their corresponding undersampled dataset ${\mathrm{X}}_{\mathrm{us}}$.

3.3. PONR

After undersampling, a dataset with repeated sample removal is obtained. However, undersampling greatly reduces the sample size in the dataset. To keep the data volume sufficient, the dataset cannot be balanced only with undersampling.

At present, the most popular dataset balancing method in machine learning is SMOTE and its derivative methods. But the synthetic data strategy of this method creates the problems of overfitting and noise amplification. In order to overcome overfitting and dataset noise, propensity score matching (PSM) is introduced to generate synthetic samples and reduce dataset noise. The theoretical framework of the PSM method is to solve the difference between the observed sampling data distribution and the overall distribution. The systematic deviation uses the propensity score to measure the difference in high-dimensional data in the feature space.

PSM is used for nearest neighbor search, which generates synthetic samples and reduces dataset noise. PSM is also used to reduce the complexity and cost of the algorithm while effectively denoising the dataset and the synthesis of minority samples, oversampling the samples closest to the center of the minority class, and improving the quality of the software defect dataset.

This subsection calculates the propensity score of each sample in ${\mathrm{X}}_{\mathrm{us}}$ by solving the weight vector of ${\mathrm{X}}_{\mathrm{us}}$ and adds it as an additional feature to ${\mathrm{X}}_{\mathrm{us}}$ for providing a basis for the subsequent PSM-based noise removal and oversampling process.

First, use the logistic function to characterize the distribution of data:

$$\mathrm{f}\left(\mathrm{x}\right)=\frac{1}{1+{\mathrm{e}}^{-\mathrm{x}}}$$

(2)

Next, define an m-dimensional (m is the dimension of feature) weight vector $\beta$ to characterize the similarity of samples. Also, define a constant ${\beta }_0$ and perform a minimum initialization, allowing each element of ${\beta }_0$ be a random decimal value close to 0 to ensure that $-{\mathrm{x}}_{\mathrm{i}}{}^{\mathrm{T}}\beta +{\beta }_0\ne 0$:

Now, define the maximum log-likelihood probability of the dataset as

$$\ln \mathrm{L}={\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{y}}_{\mathrm{i}}\ln \left(\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\beta \right)\right)+\left(1-{\mathrm{y}}_{\mathrm{i}}\right)\ln \left(1-\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}{}^{\mathrm{T}}\beta \right)\right)\right)}$$

(3)

Solve the β that maximizes the maximum log-likelihood probability of the dataset:

$$\arg {\max }_{\beta }\ln \mathrm{L}={\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{y}}_{\mathrm{i}}\ln \left(\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\beta \right)\right)+\left(1-{\mathrm{y}}_{\mathrm{i}}\right)\ln \left(1-\mathrm{f}\left({\mathrm{x}}_{\mathrm{i}}{}^{\mathrm{T}}\beta \right)\right)\right)}$$

(4)

Substitute the obtained weight vector $\beta$ and the feature vector of each sample for the logistic function to obtain the propensity score ${\mathrm{f}}_{\beta }\left({\mathrm{x}}_{\mathrm{i}}\right)$ of each sample:

$${\mathrm{f}}_{\beta }\left({\mathrm{x}}_{\mathrm{i}}\right)=\frac{1}{1+{\mathrm{e}}^{-{\mathrm{x}}_{\mathrm{i}}{}^{\mathrm{T}}\beta +{\beta }_0}}$$

(5)

Add the propensity score of each sample to ${\mathrm{X}}_{\mathrm{us}}$ as the (m + 1) th dimensional feature of the sample to obtain the propensity score dataset ${\mathrm{X}}_{\beta }$. For each sample ${\mathrm{x}}_{\mathrm{i}}$, traverse the other samples in ${\mathrm{X}}_{\beta }$ to calculate the Euclidean distance between each sample:

$${\mathrm{d}}_{\mathrm{i}\mathrm{j}}=({\displaystyle \sum _{\mathrm{z}=1}^{\mathrm{m}+1}|{\mathrm{x}}_{\mathrm{i}}^{\mathrm{z}}-{\mathrm{x}}_{\mathrm{j}}^{\mathrm{z}}{|}^2{)}^{1/2}}\mathrm{i}\mathrm{f}\mathrm{f} {\mathrm{x}}_{\mathrm{i}}\ne {\mathrm{x}}_{\mathrm{j}} \mathrm{and} {\mathrm{x}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{j}}\in {\mathrm{X}}_{\beta }$$

(6)

where ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{z}}$ is the z-th dimension feature of ${\mathrm{x}}_{\mathrm{i}}$. Then, find the k samples whose Euclidean distance is closest to ${\mathrm{x}}_{\mathrm{i}}$ to form the k nearest neighbor sample set ${\mathrm{XK}}_{\mathrm{i}}$ of ${\mathrm{x}}_{\mathrm{i}}$:

$${\mathrm{XK}}_{\mathrm{i}}=\left\{\left({\mathrm{x}}_{\mathrm{i}\mathrm{j}},{\mathrm{y}}_{\mathrm{i}\mathrm{j}}\right)|\mathrm{j}\ne \mathrm{i},\mathrm{j}\in \left(1,\mathrm{k}+1\right),\min \left({\mathrm{d}}_{\mathrm{i}\mathrm{j}}\right) \mathrm{in} {\mathrm{X}}_{\beta }\right\}$$

(7)

Then, oversampling the data, the number of synthetic samples to be generated is

$${\mathrm{N}}_{\mathrm{new}}={\mathrm{r}}_{\mathrm{os}}\times \left({\mathrm{num}}_{{\mathrm{y}}_{\mathrm{i}} = 0}-{\mathrm{num}}_{{\mathrm{y}}_{\mathrm{i}} = 1}\right),\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)\mathrm{i}\mathrm{n} {\mathrm{X}}_{\mathrm{us}}$$

(8)

where parameter ${\mathrm{r}}_{\mathrm{os}}$ is the weight of the number of samples generated and is set to 1 by default. Traverse each defect sample ${\mathrm{x}}_{\mathrm{i}}$ in the noise reduction dataset ${\mathrm{XK}}_{\mathrm{i}}$, randomly select one defect sample ${\mathrm{x}}_{\mathrm{i}\mathrm{j}}$ in its ${\mathrm{XK}}_{\mathrm{i}}$ for each defect sample, and randomly synthesize a new defect sample between the two defect sample lines in the eigenspace by (9):

$${\mathrm{x}}_{\mathrm{new}}={\mathrm{x}}_{\mathrm{i}}+\mathrm{c}\left({\mathrm{x}}_{\mathrm{i}\mathrm{j}}-{\mathrm{x}}_{\mathrm{i}}\right),{\mathrm{y}}_{\mathrm{new}}=1, {\mathrm{x}}_{\mathrm{i}\mathrm{j}}\in {\mathrm{X}}_{\mathrm{ik}}$$

(9)

where c represents a randomly generated constant between 0 and 1. Repeat the above process until the number of synthesized instances reaches ${\mathrm{n}}_{\mathrm{new}}$. Add all synthesized defective instances to ${\mathrm{X}}_{\beta }$ for obtaining ${\mathrm{X}}_{\mathrm{os}}$. After oversampling, the dataset still needs noise reduction. For each ${\mathrm{x}}_{\mathrm{i}}$, count the number of different class samples in ${\mathrm{XK}}_{\mathrm{i}}$ by (10):

$$\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}={\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{k}}|{\mathrm{y}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}\mathrm{j}}|}$$

(10)

If $\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}⩾$ the noise discrimination threshold t, consider ${\mathrm{x}}_{\mathrm{i}}$ to be a noise sample, and remove it from ${\mathrm{X}}_{\mathrm{os}}$. In this method, set the nearest neighbor number parameter k = 5 and the noise discrimination threshold t = 3 for PONR; the value of k is set according to the density of the dataset used, and the value of t is set to ensure that the data in which more than half of the k-neighbors belong to different categories are treated as noise. Repeating the above traversal process until traversing the entire ${\mathrm{X}}_{\mathrm{os}}$ without discriminating noise, the dataset after noise reduction ${\mathrm{X}}_{\mathrm{final}}$ is obtained. At this point, the complete data processing flow is completed.

3.4. Predictor Construction

The third step of the US-PONR method involves building a software defect predictor that uses a simple method of machine learning aggregation based on cross-validation (CV) scoring. It also provides a scheme for simple balanced parameter optimization.

In the SDP field, data distribution is significantly diverse: different software projects, different metrics, and different types of defects all exhibit different data distribution characteristics. Therefore, it is difficult for a single ML algorithm to comprehensively predict software defects. Inspired by the ensemble learning used in SDP [56], in this paper, multiple ML models are aggregated and trained through a CV scoring method to build an aggregate defect prediction model. Instructions for researchers are as follows:

First, set the parameter ${\mathrm{N}}_{\mathrm{k}}$ of the StratifiedKFold random grouping and the model score error threshold ${\mathrm{T}}_{\mathrm{dev}}$. StratifiedKFold random grouping refers to randomly dividing the dataset into ${\mathrm{N}}_{\mathrm{k}}$ groups and keeping the proportion of defective samples and non-defective samples in each group equal to the original dataset. The ${\mathrm{X}}_{\mathrm{final}}$ obtained in the previous step is randomly StratifiedKFold grouped to obtain a set of ${\mathrm{N}}_{\mathrm{k}}$ groups datasets:$\left\{{\mathrm{X}}_{\mathrm{final},\mathrm{k}}|\mathrm{k}=1\dots {\mathrm{N}}_{\mathrm{k}}\right\}$. In each loop, select a group of the dataset as ${\mathrm{x}}_{\mathrm{validate}}$ and the remaining groups as ${\mathrm{x}}_{\mathrm{train}}$:

$$\left\{{\mathrm{X}}_{\mathrm{validate}}={\mathrm{X}}_{\mathrm{final},\mathrm{i}},{\mathrm{X}}_{\mathrm{train}}=\mathrm{set}\left({\mathrm{X}}_{\mathrm{final},\mathrm{j}}\right),\mathrm{i}\in \left(1,{\mathrm{N}}_{\mathrm{k}}\right),\mathrm{j}\ne \mathrm{i}\in \left(1,{\mathrm{N}}_{\mathrm{k}}\right)\right\}$$

(11)

Then, the CV-based algorithm showed in Algorithm 2 was used to predict results:

Algorithm 2: CV-based aggregation

input StratifiedKFold Grouped Dataset $\left\{{\mathrm{X}}_{\mathrm{final},\mathrm{k}}|\mathrm{k}=1\dots {\mathrm{N}}_{\mathrm{k}}\right\}$,

undersampling ratio ${\mathrm{r}}_{\mathrm{us}}$, Classifers, Model Score Error Threshold ${\mathrm{T}}_{\mathrm{dev}}$

output defect prediction result, best undersampling ratio ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$

1.  for ${\mathrm{r}}_{\mathrm{us}}$ in ${\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$:

2.      for ${\mathrm{X}}_{\mathrm{final},\mathrm{k}}$ in $\left\{{\mathrm{X}}_{\mathrm{final},\mathrm{k}}|\mathrm{k}=1\dots {\mathrm{N}}_{\mathrm{k}}\right\}$:

3.          ${\mathrm{X}}_{\mathrm{train}}$ ← ${\mathrm{X}}_{\mathrm{final},\mathrm{k}}$

4.          ${\mathrm{X}}_{\mathrm{validate}}$ ← $\left\{{\mathrm{X}}_{\mathrm{final},\mathrm{j}}|\mathrm{j}\in \left(1,{\mathrm{N}}_{\mathrm{k}}\right),\mathrm{j}\ne \mathrm{k}\right\}$

5.          for classifier i in Classifiers

6.              if classifier i need hyperparameter-optimal or threshold-optimal:

7.                  optimize hyperparameter or threshold

8.              end if

9.              ${\mathrm{AUC}}_{\mathrm{i}}$ ← train predictor by ${\mathrm{X}}_{\mathrm{train}}$ and ${\mathrm{X}}_{\mathrm{validate}}$

10.              if ${\mathrm{AUC}}_{\max }$ < ${\mathrm{AUC}}_{\mathrm{i}}$:

11.                  ${\mathrm{AUC}}_{\max }$ = ${\mathrm{AUC}}_{\mathrm{i}}$

12.              end if

13.                  ${\mathrm{AUC}}_{\mathrm{classifiers}}$ ← ${\mathrm{AUC}}_{\mathrm{i}}$

14.          end for

15.           ${\mathrm{AUC}}_{\max \_{\mathrm{N}}_{\mathrm{k}}}$ += ${\mathrm{AUC}}_{\max }$

16.           ${\mathrm{AUC}}_{\mathrm{classifiers}\_{\mathrm{N}}_{\mathrm{k}}}$ += ${\mathrm{each} \mathrm{AUC}}_{\mathrm{i}}{ \mathrm{was} \mathrm{summed} \mathrm{separately} \mathrm{in} \mathrm{AUC}}_{\mathrm{classifiers}}$

17.      end for

18.      $\overline{{\mathrm{AUC}}_{\max }}$ = ${\mathrm{AUC}}_{\max \_{\mathrm{N}}_{\mathrm{k}}}/{\mathrm{N}}_{\mathrm{k}}$;$\overline{{\mathrm{AUC}}_{\mathrm{classifiers}}}$ = ${\mathrm{AUC}}_{\mathrm{classifiers}\_{\mathrm{N}}_{\mathrm{k}}}/{\mathrm{N}}_{\mathrm{k}}$

19.      Record $\overline{{\mathrm{AUC}}_{\max }}$ and $\overline{{\mathrm{AUC}}_{\mathrm{classifiers}}}$ corresponding to each different ${\mathrm{r}}_{\mathrm{us}}$

20. end for

21. calculate (max($\overline{{\mathrm{AUC}}_{\max }}$ calculated by diff ${\mathrm{r}}_{\mathrm{us}}$))

22. ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$ = ${\mathrm{r}}_{\mathrm{us}}$ with max ($\overline{{\mathrm{AUC}}_{\max }}$)

23. ${\mathrm{AUC}}_{\mathrm{classifiers}\_\mathrm{best}}$ = $\overline{{\mathrm{AUC}}_{\mathrm{classifiers}}}$ with max($\overline{{\mathrm{AUC}}_{\max }}$)

24. for result in ${\mathrm{AUC}}_{\mathrm{classifiers}\_\mathrm{best}}$

25.     if |$\overline{{\mathrm{AUC}}_{\max }}$-result| < ${\mathrm{T}}_{\mathrm{dev}}$

26.          add the classfier (with its predict result) into ${\mathrm{C}}_{\mathrm{CV}}$

27. end for

28. defect prediction result = best prediction result by classifiers in ${\mathrm{C}}_{\mathrm{CV}}$

29. return defect prediction result and ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$

where $\left\{{\mathrm{X}}_{\mathrm{final},\mathrm{k}}|\mathrm{k}=1\dots {\mathrm{N}}_{\mathrm{k}}\right\}$ is the dataset obtained by processing PONR and StratifiedKFold grouped for ${\mathrm{X}}_{\mathrm{us}}$, which is the undersampled dataset corresponding to the particular undersampling ratio ${\mathrm{r}}_{\mathrm{us}}$ in ${\mathrm{X}}_{\mathrm{us}\_\mathrm{all}}$. Classifiers is the ML classifiers used in this method shown in

Table 4. ML algorithms.

Name	Description
LiR	Linear regression
RR	Ridge regression
LoR	Logistic regression
LDA	Linear discriminant analysis
QDA	Quadratic discriminant analysis
KR	Kernel ridge
SVC	C-Support vector classification
SGDC	Linear classifiers (SVM)
KNN	K-Nearest neighbors vote classifier
GNB	Gaussian naïve Bayes
DT	Decision tree classifier
RF	Random forest classifier
ET	Extra trees classifier
AB	AdaBoost classifier

. ${\mathrm{AUC}}_{\mathrm{best}}$ is the highest AUC of the result predicted in Classifiers, ${\mathrm{AUC}}_{\mathrm{classifiers}}$ is the AUC of each classifier in Classifiers, ${\mathrm{C}}_{\mathrm{CV}V}$ is classifiers filtered by CV scoring method, and ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$ is the optimal parameter obtained by the CV-based aggregation model.

Before training different ML classifiers to obtain different machine learning models using ${\mathrm{x}}_{\mathrm{train}}$, some classifiers need hyperparameter or threshold optimization to obtain the best performance. In this method, for RR and LAR, the RandomizedSearchCV algorithm is used to tune up. For KNN, the GridSearchCV algorithm is used to determine the best n-neighbors. For LiR, KR and other classifiers that need to optimize the threshold, set the initial threshold as the median of the prediction results, then use the dichotomy to iterate, and following this, select the threshold with the maximum score as the optimal threshold. The classifiers are trained to obtain ML models, then they are used to predict ${\mathrm{x}}_{\mathrm{validate}}$, and the results are scored using AUC. AUC indicates the probability that a positive sample is greater than a negative sample when a trained classifier is used to predict a pair of data. The formula for AUC is

$$\mathrm{AUC}=\frac{{\sum }_{\mathrm{i}\in \mathrm{positiceClass}}{\mathrm{rank}}_{\mathrm{i}}-\frac{\mathrm{M}\left(1+\mathrm{M}\right)}{2}}{\mathrm{M}\times \mathrm{N}}$$

(12)

Repeat the above operation until each group of data is regarded as ${\mathrm{x}}_{\mathrm{validate}}$ once. Calculate the average AUC $\overline{{\mathrm{AUC}}_{\max }}$ of the scores of different machine learning models in ${\mathrm{N}}_{\mathrm{k}}$ experiments and take the model algorithm with the largest $\overline{{\mathrm{AUC}}_{\max }}$ and the model algorithms in ${\mathrm{C}}_{\mathrm{CV}}$ within the difference $|\mathrm{AUC}-\overline{{\mathrm{AUC}}_{\max }}|⩽{\mathrm{T}}_{\mathrm{dev}}$ as the final selection algorithms.

Due to the different datasets, class imbalance, data distribution, and other characteristics of different software, in the process of constructing the predictor, the balance parameter settings of the undersampling process should be optimized. Here, a simple scheme is proposed in order to optimize ${\mathrm{r}}_{\mathrm{us}}$ in the CV aggregation process described above. For a dataset with a class unbalanced ratio r, experiment with different ${\mathrm{r}}_{\mathrm{us}}$ values with a step size between [1, r]. Take the ${\mathrm{r}}_{\mathrm{us}}$ with the highest $\overline{{\mathrm{AUC}}_{\max }}$ as ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$ shown in Algorithm 2. Then, take the parameter ${\mathrm{r}}_{\mathrm{us}\_\mathrm{best}}$ as the selected undersampling ratio to obtain ${\mathrm{X}}_{\mathrm{final}}$ during the actual prediction.

Train the selected aggregated ML model using ${\mathrm{X}}_{\mathrm{final}}$ to obtain the defect prediction model. Measure metrics of the test set of the target software, and then implement prediction. Multiple LM models in the aggregated model were selected in the previous step, and the prediction results may not be the same, so the prediction with the highest AUC should be selected as the final prediction result. However, because it costs much less to test a software defect than to repair it, the prediction results should include as many defective modules as possible, so that more defects can be found when testing. In practice, the final defect prediction result should combine the defect modules predicted by each model.

4. Experiment

We conducted three experiments to answer the three research questions raised in Section 4.1 separately:

• defect prediction in SDP datasets without noise (for RQ1);
• defect prediction in SDP datasets in different noise environments (for RQ2);
• validation of the proportion of labeled noisy samples in the dataset before and after the use of the data pre-processing method (for RQ3).

The datasets used in different experiments are described in Section 4.2, the selected ML algorithms for the aggregated model and experiment environment are shown in Section 4.3, and the SOTA methods compared to the proposed method in a different experiment is shown in Section 4.4.

4.1. Research Questions

The experiment sought to answer the following three research questions:

• RQ1: Is US-PONR effective in SDP datasets?
• RQ2: Can US-PONR perform better compared to the benchmark methods in unbalanced SDP datasets with noise?
• RQ3: Is US-PONR especially good at eliminating label noise samples?

RQ1 aims to validate the effectiveness of the proposed method in SDP datasets. RQ2 focuses on the effectiveness of US-PONR in noisy environments. And RQ3 further validates the effectiveness of the proposed method in noise reduction.

4.2. Datasets

4.2.1. PROMISE Public Dataset

To create the US-PONR method, we used the data of 24 versions of software from 11 different software projects in the PROMISE public dataset [57].

Table 5. Datasets.

Projects	Versions	Total Instances	Defective Instances	Unbalance Ratio r
ant	1.3, 1.4, 1.5, 1.6	947	184	4.147
camel	1.0, 1.2, 1.4, 1.6	2784	562	3.954
poi	2	314	37	7.486
synapse	1.0, 1.1, 1.2	635	162	2.920
log4j	1	138	34	2.971
jedit	3.2, 4.0, 4.1, 4.2, 4.3	1749	303	4.772
PDE	1	1497	209	6.163
JDT	1	997	206	3.840
velocity	1.6	229	78	1.936
xerces	1.2, 1.3	893	140	5.379
mylyn	1	1862	245	6.600

provides an overview of these projects and their software, which represent the origin dataset before any pre-processing. The selected datasets cover different scales of data and different degrees of class imbalance. The CK and process metrics used in the experiment were directly obtained from the dataset. The net metrics were obtained by converting the dataset into a network diagram through an ISEE platform developed by our laboratory and by integrating measurement elements proposed by Yang et al. [11] and by Zimmermann and Nagappan [13]. PROMISE datasets, comprising software donations from NASA and other sources, have become one of the main public datasets in the SDP field, and they have been modified by researchers to mitigate various problems [58]. As Baljinder et al. pointed out, the quality of the PROMISE dataset is now better than the noise-reduced NASA dataset [59]. Therefore, this article assumes that the PROMISE dataset used is clean and noise-free.

4.2.2. Noise Dataset Generation

In order to generate datasets with label noise, 18 researchers in our lab were selected to re-generate labels for the PROMISE dataset. After they were generated, labels that differed from the PROMISE labels were considered samples with label noise.

We separately replaced 10%, 20%, and 30% samples of the training set with noise samples and generated three training sets with different numbers of noise samples to test the performance of the model in different noise environments. Meanwhile, the validation and test sets were kept the same as the PROMISE dataset. The generated noisy datasets were used to answer RQ2 and RQ3.

4.3. Experiment Settings

As shown in Table 4 above, 14 machine learning algorithms were selected as candidates for model aggregation. Python’s scikit-learn was then used to process any algorithms that required hyperparameter optimization and feature selection.

The experiments were carried out using Ubuntu20.04 and Python 3.8 on an Intel E5-2620 CPU with 32 GB of memory and a NVidia GeForce RTX 2080 with 16 GB of RAM.

4.4. Comparison Methods

For RQ1, the SOTA oversampling method SMOTUNED [46] was used to verify the effectiveness of US-PONR in non-noisy SDP datasets.

For RQ2 and RQ3, to compare US-PONR with current mainstream methods of oversampling and noise removal, 12 methods were selected to join US-PONR in the noise experiment following undersampling, including SOMO [46], MWMOTE [48], SMOTE-IPF [51], SMOTE-RSB [53], SMOTE-FRST-2T [54], SMOTE [20], SMOTE-TomekLinks [42], Borderline-SMOTE [43], ADASYN [44], MSMOTE [50], Gaussian-SMOTE [47], and CCR [52].

Scikit-learn and the open-source library “smote_variants” [60] on GitHub were used to implement these methods.

5. Results

5.1. Answer to RQ1: Is US-PONR Effective in SDP Datasets?

In order to answer RQ1, three small experiments were conducted:

• Determining the optimization parameter.
• Comparing the results of US-PONR with the results of just US (undersampling) or PONR alone.
• Comparing US-PONR with the SOTA data pre-processing method.

The first experiment determined the optimal parameter of each dataset. These parameters were then used for the other two experiments. The second experiment sought to prove the necessity of using both US and PONR in the proposed method. The third experiment sought to validate the effectiveness of US-PONR. The dataset used in these experiments was the original PROMISE dataset without any introduced noise (shown in Table 5).

5.1.1. Determining the Optimization Parameter

To obtain the optimal parameter, different steps were taken to process undersampling on the selected datasets; step size was set at 1 between [1, r], where r is the degree of imbalance in the dataset. We trained the CV-based aggregation model under different conditions and then compared with the average AUC obtained by cross-validation; the undersampling ratio with the largest average AUC was selected as the optimal parameter.

When conducting this experiment, we discovered that the search interval was usually too large, so we recommend that one first use step = 1 to search for the interval where there may be optimal parameters and then use step = 0.2 to find the optimal parameters. Our test results indicate that the value interval should be set as follows: ant [3, 4], camel [3, 3.9], poi [5, 6], JDT [2.6, 3.4], synapse [2, 3], jedit [3.6, 4.4], velocity [1, 1.8], xerces [4.4, 5.2], log4j [1.6, 2.4], PDE [4.6, 5.4], and mylyn [4.6, 5.4].

The results for parameter optimization are shown in

Table 6. Parameter-undersampling_ratio optimization.

	US-PONR
	rus	AUC
ant	3.8	0.8218
camel	3.9	0.8612
JDT	3.4	0.9257
jedit	4.4	0.9381
log4j	2.2	0.8716
mylyn	5.4	0.9116
PDE	5	0.9292
poi	5.8	0.9416
synapse	2.4	0.7811
velocity	1.9	0.7842
xerces	4.8	0.9442

:

These results show that the optimal parameters of each dataset are as follows: ant (3.8), camel (3.9), JDT (3.2), jedit (4.2), log4j (2.2), mylyn (5.4), PDE (5), poi (5.8), synapse (2.4), velocity (1.8), and xerces (4.8).

5.1.2. Comparing US-PONR with US and PONR

After the best parameters were obtained for each dataset, the dataset pre-processed with US-PONR or only US or only PONR was trained with a CV-based aggregation model, individually, and the AUC for each of the three methods was compared as the output result.

Figure 2 shows how the AUC of US-PONR under the optimal parameters compared with the AUC of US and PONR, individually. US-PONR performed better than both US and PONR in most datasets but performed less well than PONR alone in the ant, log4j, and synapse datasets. The reason for the latter results may be that some important features of the dataset were deleted during undersampling, and the problem was then magnified during oversampling. Overall, the results show that US-PONR is a significant improvement over either US or PONR alone.

5.1.3. Comparing US-PONR with the SOTA Data Pre-Processing Method—SMOTUNED

Table 7. Comparison results of US-PONR and SMOTUNED in AUC and processing time.

	AUC		Time
	US-PONR	SMOTUNED	US-PONR	SMOTUNED
ant	0.8218	0.8896	11.12 s	378 s
camel	0.8612	0.9290	37.63 s	6364 s
JDT	0.9257	0.8631	9.96 s	585 s
jedit	0.9381	0.9445	22.77 s	1635 s
log4j	0.8716	0.8766	1.04 s	82 s
mylyn	0.9116	0.9003	23.25 s	1594 s
PDE	0.9292	0.9061	17.52 s	3029 s
poi	0.9416	0.9346	2.95 s	111 s
synapse	0.7811	0.8315	5.71 s	598 s
velocity	0.7826	0.8399	1.64 s	150 s
xerces	0.9442	0.9325	9.09 s	502 s

shows how US-PONR compared with SMOTUNED, the alternative SOTA data pre-processing method. The results show that the AUC of the two methods was good or bad in different datasets, but nonetheless SMOTUNED spent a great deal more time on each dataset than US-PONR. The reason for the latter result is that SMOTUNED carried out many iterations to obtain the best parameter value, and the larger the dataset, the more time it spent. US-PONR obtained the best parameter value by interval search, which greatly reduced its processing time.

5.2. Answer to RQ2: Can US-PONR Perform Better Compared to the Benchmark Methods in Un-Balanced SDP Datasets with Noise?

The purpose of this experiment was to explore how well US-PONR predicts defects in unbalanced datasets that contain different proportions of label noise. The proposed method was compared with other oversampling and filtering methods after undersampling.

In order to test datasets containing noise samples of different proportions, training datasets were generated with 10%, 20%, and 30% noise samples added to each set (see Section 4.2.2). Data pre-processing was then performed on these datasets to train the aggregate SDP model.

A total of 12 methods of oversampling and noise reduction were tested, alongside our proposed method, on the datasets under the three noise levels. Following the same experiment settings used to optimize dataset parameters (Section 5.1.1), the settings of each dataset were as follows (for 10%, 20%, 30% noise): ant (1.6, 1.6, 1.4), camel (1.4, 1.4, 1.4), JDT (2.8, 1.5, 1.5), jedit (2, 2, 2), log4j (1.2, 1.2, 1.2), mylyn (3.5, 2.7, 2), PDE (2, 2, 2.2), poi (1.8, 1.8, 1.2), synapse (2.2, 1.8, 1.2), velocity (1.6, 1.2, 1.2), and xerces (2, 2, 2).

Table 8. AUC of each method for different datasets at 10%, 20%, and 30% noise levels.

	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
ant
10% noise	0.8676	0.8624	0.8571	0.8630	0.8630	0.8716	0.7355	0.7481	0.6997	0.7201	0.7338	0.7628	0.8355
20% noise	0.7952	0.7012	0.7338	0.7287	0.7015	0.7080	0.6980	0.7290	0.6792	0.7133	0.7116	0.7457	0.7780
30% noise	0.7717	0.7359	0.7461	0.7185	0.7438	0.7314	0.7461	0.7344	0.7008	0.6929	0.7185	0.6945	0.7618
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
camel
10% noise	0.8522	0.8549	0.8508	0.8470	0.8528	0.8433	0.7653	0.7678	0.7443	0.7730	0.7353	0.7864	0.7523
20% noise	0.7985	0.7354	0.7423	0.7610	0.7459	0.7112	0.7251	0.7678	0.7232	0.7366	0.7398	0.7264	0.7021
30% noise	0.7774	0.7155	0.7366	0.7508	0.7208	0.6633	0.6990	0.7389	0.6849	0.7309	0.7105	0.7105	0.6809
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
JDT
10% noise	0.8960	0.7035	0.8271	0.7648	0.7091	0.7596	0.7989	0.8139	0.7406	0.8004	0.7775	0.8766	0.8859
20% noise	0.8166	0.6599	0.7896	0.7251	0.7402	0.7452	0.7993	0.7962	0.7220	0.7685	0.7540	0.8253	0.7100
30% noise	0.8020	0.6447	0.7643	0.7181	0.7091	0.7201	0.7738	0.7858	0.7207	0.7758	0.7410	0.8027	0.7028
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
jedit
10% noise	0.8865	0.8073	0.8829	0.8762	0.8364	0.8443	0.8720	0.8771	0.8437	0.8824	0.8308	0.9016	0.9061
20% noise	0.8882	0.8169	0.8856	0.8479	0.8273	0.8173	0.8558	0.8730	0.8525	0.8724	0.8155	0.8720	0.8816
30% noise	0.8804	0.8201	0.8622	0.8383	0.7960	0.8470	0.8350	0.8670	0.8086	0.8375	0.7995	0.8614	0.8589
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
log4j
10% noise	0.8684	0.7838	0.8354	0.7949	0.7792	0.7838	0.7532	0.8519	0.8052	0.7586	0.7805	0.7805	0.7887
20% noise	0.8108	0.8101	0.8205	0.7397	0.7632	0.7576	0.7342	0.8000	0.7442	0.7750	0.7317	0.7436	0.7619
30% noise	0.7945	0.7619	0.7805	0.7536	0.7654	0.7805	0.7686	0.7632	0.7541	0.7394	0.7541	0.7541	0.7448
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
mylyn
10% noise	0.8538	0.7403	0.7889	0.7314	0.6851	0.6731	0.8018	0.8067	0.7526	0.7632	0.7287	0.8688	0.8848
20% noise	0.7542	0.6035	0.6871	0.6870	0.5676	0.6823	0.7504	0.7795	0.7277	0.7651	0.6651	0.8293	0.8472
30% noise	0.7508	0.5664	0.7027	0.6454	0.5589	0.6546	0.6588	0.6254	0.6196	0.7123	0.6639	0.7699	0.8211
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
PDE
10% noise	0.7800	0.7001	0.8040	0.7598	0.7373	0.6843	0.8148	0.8099	0.7282	0.7693	0.7168	0.8900	0.8862
20% noise	0.7438	0.7211	0.7776	0.7251	0.7336	0.6602	0.7696	0.8044	0.7509	0.7891	0.7171	0.8529	0.8416
30% noise	0.7282	0.7202	0.7522	0.7052	0.7144	0.6699	0.7642	0.7681	0.7402	0.7773	0.6932	0.7798	0.8035
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
poi
10% noise	0.7259	0.7087	0.6812	0.6777	0.6557	0.7867	0.7717	0.7626	0.7134	0.6479	0.6667	0.7671	0.8056
20% noise	0.6612	0.6825	0.7107	0.6614	0.5912	0.6897	0.7813	0.7143	0.7273	0.7412	0.6013	0.7432	0.7285
30% noise	0.6947	0.6914	0.5882	0.6301	0.6739	0.6582	0.7647	0.6747	0.6452	0.6905	0.6897	0.6667	0.6410
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
synapse
10% noise	0.8037	0.5630	0.7304	0.7009	0.6281	0.7139	0.7295	0.7358	0.7176	0.7586	0.7107	0.7715	0.7988
20% noise	0.7866	0.6379	0.7361	0.6988	0.6490	0.6910	0.7330	0.6845	0.7077	0.7431	0.6833	0.7539	0.7818
30% noise	0.7510	0.6140	0.7190	0.7335	0.6199	0.7224	0.6846	0.6792	0.6585	0.6732	0.6815	0.6254	0.6649
	US-PONR	SOMO	MWMOTE	SMOTE_##3IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
velocity
10% noise	0.8496	0.7757	0.8333	0.7724	0.7628	0.7439	0.8171	0.7987	0.7642	0.7561	0.7561	0.8272	0.8067
20% noise	0.8190	0.7686	0.7716	0.7371	0.7353	0.7672	0.7457	0.7959	0.7371	0.7802	0.7802	0.7489	0.7974
30% noise	0.8163	0.7600	0.7908	0.7398	0.7245	0.7806	0.7398	0.7911	0.7500	0.7194	0.7653	0.7199	0.7449
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
xerces
10% noise	0.8616	0.7928	0.8544	0.8087	0.7309	0.7814	0.8477	0.8542	0.8108	0.8298	0.8033	0.8793	0.8734
20% noise	0.8535	0.7667	0.8242	0.7849	0.7330	0.7790	0.8316	0.8398	0.7993	0.8379	0.7824	0.8711	0.8700
30% noise	0.8448	0.6484	0.7942	0.7928	0.7225	0.7647	0.8321	0.8318	0.7880	0.8100	0.7818	0.8303	0.8441

shows the AUC for each method when applied to every dataset. The results indicate that no matter which dataset or method was used, the introduction of noise always reduced a method’s prediction performance. However, under most noise ratios of most datasets, the prediction performance of US-PONR was the best, while its decrease in performance as the noise ratio rose was minimal. When the US-PONR method was used on the POI dataset, the low-level noise score was lower than the high-level score. The reason for this anomaly is that the undersampling step size was set too large. In order to save time and money, the step size was set to 0.2. For the same dataset with different noise levels, the high noise level dataset gives the impression of having reached the optimal while the low noise level dataset does not.

Experiment results show that at the 30% noise level, our proposed method had a very significant advantage over all other methods for most datasets, especially for Gaussian SMOTE and CCR (which outperforms US-PONR at the low noise level in some datasets). This suggests that US-PONR is the best overall method in a noisy environment, except for the Mylyn and PDE datasets. The reason for the method’s poor performance on these datasets may be that they have heavy distributions (such as the “levy distribution”), which make it impossible to distinguish between good and bad samples in the feature space using PSM, leading to improper oversampling or noise reduction. The reason for US-PONR’s superior performance lies in the way it reduces noise using the PSM score, which can effectively capture noise samples.

5.3. Answer to RQ3: Is US-PONR Especially Good at Eliminating Label Noise Samples?

In order to verify the effectiveness of US-PONR in eliminating label noise, we designed an experiment to discover how many introduced noise samples were found and eliminated by each method in the 10%, 20%, and 30% noise environments.

We marked each introduced label noise sample and then checked the residual noise ratio in the datasets after using each method. The residual noise ratio is the number of the remaining marked noise samples divided by the total number of samples after using the data processing method. The residual noise ratio in the dataset is used to verify the effectiveness of the noise reduction method: the smaller the proportion of remaining marked noise samples in the datasets, the more effective the noise-reduction method. The setting of dataset parameters in this experiment was the same as that used to test RQ2.

Table 9. Residual noise sample ratio of each method for different datasets at 10%, 20%, and 30% noise levels.

	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
ant
10% noise	0.0177	0.0995	0.0675	0.0670	0.0867	0.0662	0.0644	0.0600	0.0643	0.0644	0.0647	0.0643	0.0672
20% noise	0.0435	0.1996	0.1479	0.1457	0.1679	0.1448	0.1417	0.1340	0.1415	0.1415	0.1408	0.1413	0.1530
30% noise	0.0926	0.2844	0.2416	0.2407	0.2639	0.2405	0.2355	0.2180	0.2355	0.2339	0.2347	0.2354	0.2193
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
camel
10% noise	0.0185	0.0996	0.0677	0.0650	0.0884	0.0685	0.0682	0.0597	0.0678	0.0682	0.0683	0.0675	0.0637
20% noise	0.0496	0.1987	0.1470	0.1439	0.1894	0.1510	0.1480	0.1318	0.1476	0.1480	0.1499	0.1468	0.1468
30% noise	0.1002	0.2998	0.2420	0.2390	0.2538	0.2512	0.2432	0.2222	0.2426	0.2406	0.2447	0.2418	0.2132
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
JDT
10% noise	0.0181	0.0939	0.0682	0.0678	0.0985	0.0671	0.0682	0.0614	0.0677	0.0683	0.0682	0.0678	0.0650
20% noise	0.0472	0.1890	0.1483	0.1475	0.1986	0.1475	0.1220	0.1317	0.1475	0.1478	0.1479	0.1481	0.1510
30% noise	0.0929	0.2851	0.2435	0.2419	0.2982	0.2416	0.2431	0.2249	0.2425	0.2422	0.2422	0.2435	0.2170
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
jedit
10% noise	0.0156	0.0998	0.0657	0.0653	0.0993	0.0665	0.0658	0.0580	0.0658	0.0658	0.0657	0.0659	0.0674
20% noise	0.0399	0.1997	0.1434	0.1434	0.1773	0.1445	0.1438	0.1312	0.1440	0.1441	0.1452	0.1433	0.1531
30% noise	0.0872	0.3000	0.2381	0.2369	0.2869	0.2396	0.2371	0.2164	0.2380	0.2375	0.2388	0.2378	0.2188
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
log4j
10% noise	0.0198	0.0873	0.0750	0.0725	0.0879	0.0689	0.0737	0.0654	0.0729	0.0747	0.0703	0.0680	0.0829
20% noise	0.0435	0.1996	0.1479	0.1457	0.1679	0.1448	0.1417	0.1340	0.1415	0.1415	0.1408	0.1413	0.1530
30% noise	0.0926	0.2844	0.2416	0.2407	0.2639	0.2405	0.2355	0.2180	0.2355	0.2339	0.2347	0.2354	0.2193
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
mylyn
10% noise	0.0119	0.0776	0.0634	0.0617	0.0995	0.0694	0.0630	0.0596	0.0633	0.0628	0.0631	0.0629	0.0612
20% noise	0.0343	0.1632	0.1386	0.1369	0.1997	0.1531	0.1388	0.1289	0.1385	0.1384	0.1389	0.1383	0.1277
30% noise	0.0782	0.2722	0.2317	0.2295	0.2988	0.2571	0.2326	0.2159	0.2321	0.2311	0.2329	0.2318	0.2205
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
PDE
10% noise	0.0109	0.0999	0.0642	0.0626	0.0993	0.0624	0.0635	0.0592	0.0634	0.0633	0.0636	0.0634	0.0612
20% noise	0.0331	0.1953	0.1403	0.1384	0.1989	0.1395	0.1398	0.1330	0.1395	0.1398	0.1398	0.1396	0.1387
30% noise	0.0753	0.2956	0.2339	0.2312	0.2989	0.2339	0.2334	0.2216	0.2332	0.2326	0.2330	0.2326	0.2208
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
poi
10% noise	0.0084	0.0963	0.0681	0.0572	0.0920	0.0639	0.0630	0.0551	0.0690	0.0632	0.0646	0.0595	0.0608
20% noise	0.0284	0.1927	0.1414	0.1338	0.1921	0.1457	0.1385	0.1219	0.1432	0.1424	0.1379	0.1374	0.1295
30% noise	0.0718	0.2898	0.2295	0.2167	0.2895	0.2492	0.2359	0.2143	0.2242	0.2306	0.2363	0.2329	0.2192
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
synapse
10% noise	0.0263	0.0988	0.0715	0.0716	0.0938	0.0704	0.0722	0.0556	0.0718	0.0714	0.0727	0.0721	0.0756
20% noise	0.0614	0.1954	0.1551	0.1526	0.1715	0.1539	0.1548	0.1375	0.1539	0.1547	0.1550	0.1540	0.1473
30% noise	0.1148	0.2930	0.2505	0.2497	0.2704	0.2506	0.2493	0.2222	0.2503	0.2507	0.2498	0.2502	0.2838
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
velocity
10% noise	0.0365	0.0964	0.0790	0.0791	0.0948	0.0784	0.0775	0.0662	0.0790	0.0791	0.0794	0.0775	0.0695
20% noise	0.0885	0.1891	0.1685	0.1614	0.1927	0.1693	0.1694	0.1465	0.1669	0.1719	0.1642	0.1642	0.1925
30% noise	0.1460	0.2933	0.2649	0.2595	0.2915	0.2743	0.2669	0.2470	0.2645	0.2665	0.2665	0.2627	0.3000
	US-PONR	SOMO	MWMOTE	SMOTE_ IPF	SMOTE_ RSB	SMOTE_ FRST_2T	SMOTE	SMOTE_ TomekLinks	Borderline_ SMOTE2	ADASYN	MSMOTE	Gaussian_ SMOTE	CCR
xerces
10% noise	0.0134	0.0818	0.0649	0.0641	0.0924	0.0718	0.0643	0.0435	0.0645	0.0640	0.0647	0.0648	0.0634
20% noise	0.0367	0.1670	0.1437	0.1409	0.1878	0.1641	0.1419	0.1062	0.1418	0.1420	0.1418	0.1417	0.1412
30% noise	0.0718	0.2553	0.2357	0.2340	0.2724	0.2654	0.2349	0.1903	0.2356	0.2354	0.2352	0.2355	0.2222

shows the proportion of residual marked label noise samples after using each method in 10%, 20%, and 30% noise environments. For example, in 10% noise level ant, the residual noise sample ratio recorded for US-PONR was 1.77%, which means that US-PONR identified and removed 8.23% of the noise samples. The table shows that in each dataset and noise level, the residual noise ratio was the lowest after using US-PONR (maximum: −22.82% in 30% noise Xerces; minimum: −15.4% in 30% noise level Velocity), which proves that the proposed method is effective in removing noise samples and improving the quality of datasets.

6. Threats to Validity

6.1. Datasets

In our SDP experiment, we tested US-PONR on 24 versions of software data from 11 projects in the PROMISE dataset, and our proposed method performed admirably. However, before being put into practice, US-PONR needs to be tested against actual commercial projects. Moving forward, we plan to apply this method to some commercial projects to verify its validity.

Although our team members manually labeled the samples to simulate as much as possible the process of generating label noise in real data (see Section 4.2.2), our test methodology inevitably brought some artificial features to the dataset. In the future, US-PONR’s effectiveness will also need to be verified in other noisy datasets.

Finally, the CK, process, and network metrics were chosen to be used in the experiment as feature vectors for representing software defects. There are other metrics that can be used to characterize software defects, such as the Hausdorff metric [61]. Validation of the method against these metrics also needs to be carried out in future work.

6.2. Model Hyperparameters

US-PONR and other methods for detecting software defects have difficulty controlling every variable and every model parameter, such as the n_neighbors k, the noise discrimination threshold t, or the weight of the number of samples generated. Therefore, in our experiment, we used the parameters recommended by their authors, and we performed no further tuning of them. Fine-tuning these parameters may improve their performance slightly.

6.3. Evaluation

In our study, AUC was used to evaluate the prediction performance of the models. Other indicators such as MCC were also calculated in the experiment, but due to the limited length of the article, these indicators are not fully addressed in the paper. Readers who are interested in detailed results should contact the authors by email.

7. Discussion and Conclusions

In the past two decades, software defect prediction (SDP) has developed rapidly as an important way to improve software quality and reliability. However, class imbalance and noise are often present in SDP datasets. With the continuous improvement of SDP algorithms based on machine learning, improving the quality of software defect data offers a further significant effect to improve SDP performance. Oversampling such as SMOTE is one of the most well-known methods to balance the datasets, but it can introduce overfitting noise to datasets. Thus, our article attempts to solve the data imbalance in the SDP dataset and reduce the noise in it.

In this paper, US-PONR is proposed as a dataset pre-processing method to improve SDP dataset quality. To alleviate imbalance in a dataset, US-PONR first uses undersampling to remove redundant samples due to version iterations, and then it uses oversampling to remove potential noise samples, relying on propensity score matching as a dimension of the feature vector. A total of 24 versions of software data from 11 projects were used in experiments to test the proposed method. In an SDP experiment conducted in different noise environments, US-PONR demonstrated its ability to improve the quality of SDP datasets. And in experiments on removing label noise samples, the new method demonstrated that it can effectively identify label noise samples and remove them.

However, there are still a few limitations related to the datasets will need to be addressed. For our data, the experiments only used the CK, process, and network metrics extracted from PROMISE datasets. In the future, we hope to test our method against commercial datasets and more metrics that represent software defects. And artificial noise may be introduced in the generation of noise datasets using for experiments, so the ability of the method to identify and remove label noise samples needs further validation in real software projects.