Forecasting Spare Parts Demand of Military Aircraft: Comparisons of Data Mining Techniques and Managerial Features from the Case of South Korea

Abstract

In a weapon system, the accurate forecasting of the spare parts demand can help avoid the excess inventory, leading to the efficient use of budget. It can also help develop the combat readiness of the weapon system by improving weapon system utilization. Moreover, as performance-based logistics (PBL) projects have recently emerged, the accurate demand forecasting of spare parts has become an important issue for the PBL contractors as well. However, for the demand forecasting of spare parts, the time series methods, typically used in the military sector, have low prediction accuracies and the PBL contractors are mostly based on the judgment of practitioners. Meanwhile, most of the previous studies in the military sector have not considered the managerial characteristics of spare parts (e.g., reparability and the irregularity of maintenance). No previous work has considered any such features, which can indicate the reliability of spare parts (e.g., mean time between failures (MTBF)), although they can affect the spare parts demand. Therefore, to develop a more accurate forecasting of the spare parts demand of military aircraft, we designed and examined a systematic approach that uses data mining techniques. To fill up the research gaps of related works, our approach also considered the managerial characteristics of spare parts and included the new features that represent the reliability of spare parts. Consequently, given the case of South Korea and the full feature set, we found random forest gave better results than the other data mining techniques and the conventional time series methods. Using the best technique Random Forest, we identified the contribution of each managerial feature set to improving the prediction accuracy, and we found the reliability and operation environment are valuable feature sets in a significant way, so they should be collected, managed more carefully, and included for better prediction of spare parts demand of military aircraft.

1. Introduction

1.1. Background and Purpose

A weapon system is an integrated and computerized system for the control and operation of particular weapons (e.g., fighter aircraft and missiles). For the weapon system, total life cycle system management (TLCSM) has been employed to systematically manage the entire life cycle process from the initial planning of the weapon system to research, development, acquisition, production, maintenance, and disposal. Because the cost of operating maintenance generally accounts for 60–70% of the cost in the total life cycle, securing spare parts economically for the operation maintenance after the acquisition of the weapon system is essential for efficiently operating the weapon system.

To assure the supplies of spare parts economically, it is necessary to forecast the demand of the spare parts with high accuracy. The low accuracy may increase budget losses due to: lowered utilization rates by maintenance delays, inventory overruns by inventory depletion, and excess stocks. Moreover, failure to provide the correct number of repairs in a timely manner can have a negative impact on weapon system readiness. Thus, the accurate demand forecasting of spare parts in the military sector is important not only for reducing the cost of operating and maintaining the weapon system’s total life cycle, but also for developing the combat readiness of the weapon system.

To deal with the demand forecasting of spare parts, the Republic of Korea (ROK) military has typically used the time series techniques, which are statistical methods based on the past demands. However, their accuracies have been reported to be about 70% by item and about 30–40% by quantity, showing a limit to the improvement of forecasting accuracies [1]. Hence, to resolve such difficulties, the ROK military has started to operate the performance-based logistics (PBL) system, which is a follow-up logistics support project [2,3]. In detail, the contractor that operates the PBL system, namely the PBL contractor, is responsible for the maintenance and dissemination of a certain part of the weapon system as an employer instead of the military sector. During the contract period, performance scores for each task are synthesized annually to calculate the grade according to the performance goal. The incentives according to the calculated grade are paid to the PBL contractor. On the other hand, the PBL contractor should pay a penalty if the performance goal is not met.

Therefore, the PBL contractor should try to achieve the performance goal in order to generate profits for the business. To achieve the performance goal of operating the PBL system, the PBL contractor must hold the inventory of each spare part. However, this inventory cost is not recognized as a business cost. If the PBL contractor’s inventory is not used for military maintenance, it will be treated as a loss cost incurred by the PBL contractor. So, as the PBL project expands, subsequent PBL contractors should also make continuous efforts to improve the accuracy in forecasting the demand of spare parts.

Consequently, forecasting the demand of spare parts accurately has become an important issue for the PBL contractor’s carrying out a lossless business, as well as for the military’s combat readiness and efficient budget utilization. In these regards, there are challenging issues to be resolved for more accurate forecasting the demand of spare parts: First, the time series techniques that the military sector has typically used as a method to forecast the demand of spare parts are not accurate; second, the contractors of the PBL system have predicted the amount of inventory for spare parts on the basis of their experiences as the PBL project managers, resulting in low accuracies.

Fortunately, in case of the ROK military, equipment maintenance and management information system, namely Defense Logistics Integrated Information System (DELIIS), has recently been developed for the supply and management of spare parts and tools, related to the main equipment of weapon systems [4]. It has been used by various maintenance departments, ranging from organizing units to the ministry of defense, in the ROK military, and the integration over land, sea, and air force systems is ongoing. In addition to the historical records that describe how spare parts were managed, it has accumulated other various data of the weapon systems such as the operation time, the number of sorties, the procurement period of spare parts, and the supply period. The accumulated data can provide an opportunity in improving the demand forecasting of spare parts.

However, the accumulated data of the ROK military’s DELIIS has remained unexplored because they are not accessible by public sectors for security reasons. In such circumstances, this study aims to report the results of our explorations, which were possible with permission and cooperation to using the secured data of DELIIS for more accurate forecasting the demand of spare parts, particularly for the ROK military aircraft.

In other words, by using the accumulated data of the ROK military’s DELIIS, we identified feature variables related to managing the spare parts of military aircraft, and proposed a research framework to design a systematic approach to forecast the spare parts demand of military aircraft by using those feature variables and commonly used data mining techniques. Moreover, we evaluated the prediction performance of different data mining techniques, and compared the best technique among them to the conventional time series techniques, which have been previously used in the ROK military. Lastly, we identified which managerial feature set is more useful for more accurate demand forecasting of the military aircraft spare parts.

1.2. Reviews on Related Works

Demand forecasting has been studied in various fields, but their methods can be generally grouped into qualitative and quantitative [5,6]. Moreover, the quantitative methods can be divided into three: time-series, feature-based, and hybrid [7]: Time series methods have been widely applied to demand forecasting, while they often use historical data only, which makes it difficult to perform demand forecasting [8,9]; feature-based methods use explanatory variables and estimate demand as a linear/nonlinear or so function of the explanatory variables [10]; hybrid approaches mix one or more types of techniques (e.g., time series and neural network) [11].

Data mining, among many qualitative methods of demand forecasting, is a casual method that helps decision support analytically by extracting features, relationships, patterns, and rules through exploration, analysis, and modeling of large-scale data [12,13]. Data mining has emerged as an alternative tool for modeling and forecasting due to its ability to capture the non-linearity in the data. While the shortcoming of data mining is a large amount of training data [14], with the advent of big data era, data mining has recently been widely used for demand forecasting in the various fields, where data can be collected easily, such as energy [10,15], tourism [16,17], transportation [18,19,20], water management [21,22], remanufacturing [23], bike sharing [24,25], retail pharmacies [26], hospitals [27,28], logistics [14], and spare parts management [14,29,30,31], showing its usefulness.

In terms of the above-mentioned taxonomy, this paper can be classified as demand forecasting for spare parts management, which applies the casual method of the quantitative method (i.e., data mining).

Related to spare parts management, prior works can be largely grouped into spare parts classification and demand forecasting [7]. Particularly, spare parts demand forecasting is essential to spare parts inventory control, and it has been notoriously difficult because the demand of spare parts is typically irregular and lumpy. As a result, forecasting the spare parts demand has become even more critical [8,32,33].

Table 1. Previous works on spare parts demand forecasting in various industries (searched for our initial investigation).

Previous Work	Industry	Used Method
		Time Series	Feature-Based	Hybrid
Romeijnders, Teunter and van Jaarsveld [8]	Aviation	√
Zhu, Jaarsveld and Dekker [32]	Maintenance/Repair		√
Sharma, Kulkarni and Yadav [33]	Military			√
Jónás, Tóth and Dombi [34]	Electronics			√
Vargas and Cortés [35]	Automobile	√	√	√
Rosienkiewicz, Chlebus and Detyna [11]	Mining			√
Li, Liu, Shen and Cheng [36]	Maintenance/Repair		√
van Jaarsveld, Dollevoet and Dekker [37]	Maintenance/Repair		√
Kim, Dekker and Heij [38]	Consumer goods	√
Wang, Pan, Wang and Wei [39]	Aviation	√	√
Sun, Hao, Su and Ren [40]	Aviation	√
Guo, Diao, Zhao, Wang and Sun [41]	Aviation	√
Scala, Rajgopal and Needy [42]	Energies		√

shows the previous works, searched initially by us to investigate spare parts demand forecasting in various industries. Though the applications of spare parts demand forecasting have been made in different industries such as electronics [34], automobile [35], mining [11], maintenance/repair [36,37], consumer goods [38], aviation [39,40,41], energies [42], etc., those can be grouped into three according to their demand forecasting methods as shown in Table 1.

From Table 1, we could confirm that the prior works on spare parts demand forecasting have applied different methods to solve their specific problems in their industries. This indicates that it is unclear which method can be best applicable for each problem of an industry before performing experiments and evaluations. It is also consistent with the conclusion of literature review by Bacchetti and Saccani [7]. Hence, to achieve the purpose of this study, we need to narrow down to the military sector and investigate thoroughly prior works, related to spare parts demand forecasting for weapon systems [43,44]. The military sector also has suffered from spare parts supply problems, caused by inaccurate forecasts of spare parts demand. This originates from that spare parts demand in the military sector is often non-normal, characterized by being intermittent, slow-moving, and erratic. Especially, another problem is that there are obstacles to collect data for analysis owing to confidentiality, lack of data, inaccessibility, and so on [33]. However, as the establishment of the DELIIS made it possible to organize information related to the operation of the ROK military’s weapon systems, researches in South Korea have recently applied data mining to forecasting the demand of spare parts for the weapon systems. This has led to the increasing attention to spare parts demand forecasting in the military sector, particularly in South Korea.

Table 2. Previous data mining-based studies on demand forecasting of spare parts in the military sector.

Previous Work	Data	Managerial Feature Sets				Data Mining Techniques
		Reliability (F1)	Information on Spare Parts (F2)	Operation Environment (F3)	Consumption Performance (F4)
Kim and Lee [46]	2010~2014 data of the Republic of Korea (ROK) Air Force		Repair rate, mortality rate, lead time (in days), price of spare parts	Aircraft operating time (in hours), number of aircraft sorties	The number of a spare part, consumed in the past year	Decision Tree, Bayesian Network, Neural Network, Linear Regression, Core Vector Regression
Yoon and Kim [4]	2009~2016 data of the Republic of Korea (ROK) Navy		Repair rate; number of items, selected in the authorized stockage list (ASL)	Operating days of navy vessels	The number of a spare part, consumed in the past year; ratio of deliveries to requests	Classification and Regression Tree (CART), Random Forest, Neural Network, Linear Regression
Boukhtouta and Jentsch [43]	84-month data (year unknown) of the Canadian Armed Forces (CAF)				The number of a spare part, consumed on the past month	Support Vector Machine (SVM)
Oh [45]	2010~2016 data of the Republic of Korea (ROK) Navy				The number of a spare part, consumed in the past year	Long Short-Term Memory, Neural Network, Recurrent Neural Network, Gated Recurrent Unit
Jeong, et al. [47]	2010~2017 data of the Republic of Korea (ROK) Air Force		Item price	Number of aircraft operated, number of flights, aircraft operating time (in hours)	The number of a spare part, consumed in the past year	Gradient Boosting, Random Forest, Support Vector Regression
Kim and Ma [48]	2012–2016 data of the Republic of Korea (ROK) Air Force				The number of a spare part, consumed in the past year; the number of a spare part, consumed per an aircraft operating time (in hours) in the past year; the number of a spare part, consumed per an aircraft sortie in the past year	Linear Regression, Random Forest, Neural Network
Kim [49]	2010~2016 data of the Republic of Korea (ROK) Army		Item price, lead time (in days)	Tank operating time (in hours), number of the tanks	The number of a spare part, consumed in the past year	Logistics Regression, Support Vector Regression, Neural Network
Lee and Kim [50]	2010~2014 data of the Republic of Korea (ROK) Air Force		Item price, lead time (in days), repair rate, mortality rate	Aircraft operating time (in hours), number of sorties	The number of a spare part, consumed in the past year	Decision Tree, Bayesian Network, Core Vector Regression
Kim and Lee [51]	2010~2014 data of the Republic of Korea (ROK) Ministry of National Defense (MND)		Item price, lead time (in days)	Aircraft operating time (in hours), aircraft operating distance	The number of a spare part, consumed in the past year; the number of procurements per a spare part by the past year	Decision Tree, Bayesian, Support Vector Machine
Pawar and Tiple [52]	Vietnam War dataset		Item price, lead time (in days), acquisition year, maintenance year, name of a spare part, National Stock Number (NSN) of a spare part, quantity per the assembly unit of a spare parts		The number of a spare part, consumed in the past year	Support Vector Machine, Random Forest, Logistic Regression, Decision Tree, Neural Network, XGBoost
This study	2013~2018 data of the Republic of Korea (ROK) military and the Korea Aerospace Industries (KAI)	Design reliability, operational reliability, calibration reliability	Item price, lead time (in days), spare parts system	Aircraft operating time (in hours), number of aircraft sorties, number of aircraft operated, aircraft utilization rate	The number of a spare part, consumed in the past year	Random Forest, Support Vector Regression, Linear Regression, Neural Network

shows the recent works that we investigated regarding the demand forecasting of spare parts using data mining in the military sector. They are classified mainly by two perspectives, that is, features by their managerial implications and data mining techniques. It also shows how our study can be classified according to the taxonomy. Considering managerial implications, we grouped features into the four groups of features, while we excluded climate-related feature variables that are uncontrollable and unmanageable (e.g., temperature, humidity, precipitation, and wind speed). In other words, to investigate managerial implications in our experimental results, the feature variables applied to the models in this study were classified into four categories, reliability (denoted by F₁), information on spare parts (denoted by F₂), operation environment (denoted by F₃), and consumption performance (denoted by F₄).

From Table 2, our findings can be summarized as follows: First, most of the previous studies in Table 2 used the typical machine learning methods such as linear regression, decision tree and neural network to forecasting the demand of spare parts; second, while they have recently extended to deep learning [45], deep learning has rarely been adopted because it requires a vast data set and a very long time for training models, which are not possible for the cases in the military sector; third, machine learning methods showed better prediction results when compared to the conventional time series methods. Based on these findings, our study selected typical machine learning methods for data mining techniques, and compared them with the conventional time series methods to confirm their superiority.

1.3. Research Gaps and Questions

By referring to Table 2, we found research gaps in the previous studies for data mining-based demand forecasting of spare parts in the military sector, and they can be summarized as follows:

First, most of the previous studies have not considered the managerial characteristics of spare parts such as reparability and the irregularity of maintenance. However, practically in maintaining a weapon system, the spare parts can be classified into repairable and non-repairable. In addition, they can be put into scheduled maintenance, made regularly, or unscheduled maintenance, performed irregularly. Unlike the previous works, to reflect the need in the field, we classified spare parts into repairable or unrepairable, and scheduled or unscheduled. Then, we selected the unrepairable spare parts, managed in an unscheduled manner, as targets for our study. Though a few studies considered the intermittent occurrences of spare parts [47,48], their selecting the spare parts was not based on the managerial characteristics of spare parts but on the failure distribution of spare parts. On the other hand, we considered the characteristics of spare parts, and targeted at the unrepairable spare parts with intermittent maintenance.

Second, there is no previous work that considered the mean time between failures (MTBF) in hours as a feature, although MTBF indicates the reliability of a spare part, which can affect the spare part’s demand. Hence, MTBF and the other indicators, derivable from MTBF, need to be considered as features. In this regard, our study introduced three new features to represent the reliability of a spare part as below, and used them for forecasting the demand of a spare part:

• Design reliability is MTBF, the mean time (in hours) between failures of a spare part, which is estimated at the stage of system development. It represents the failure frequency of the spare part.
• Operation reliability represents the failure frequency of a spare part in the operation of the weapon system. Considering the actual operation time (in hours), it is obtained by getting the cumulative number of consumption for a spare part during the operation period, and dividing it by the total operating hours.
• Calibration reliability is to compensate for the weaknesses of both design and operation reliabilities. It is a weighted sum of the design reliability, which considers the entire operation period, and the operational reliability, which reflects only the results operated so far, with the operating time (in hours) as their common denominator.

Eventually, considering the findings and research gaps identified by our reviewing the previous studies in Table 2, we constructed the research questions as below:

• RQ1. Which data mining technique will give a better prediction result? How is it compared to the conventional time series methods?
• RQ2. Which feature set will give a better prediction result? Is there any managerial implication regarding the ranking among the feature sets?

1.4. Organization of This Paper

The rest of this paper is composed of three sections. Section 2 describes the research framework, which we proposed and used to design and examine a systematic approach for the spare parts demand forecasting. Section 3 demonstrates the results of applying the research framework to the case of South Korea, comparing different techniques in terms of prediction accuracy and exploring the usefulness of each managerial feature set on improving the accuracy of spare parts demand forecasting. In the end, Section 4 concludes the paper, summarizing the entire study and describing implications with limitations for future works.

2. Materials and Methods

To design a systematic approach to forecast the spare parts demand of military aircraft, we proposed a research framework, as shown in Figure 1, and we answered our research questions by applying it to the case of South Korea. The following subsections explain the details of the proposed research framework.

2.1. Data Acquisition

In this study, we selected the Korea Aerospace Industries (KAI), an aircraft manufacturer of the ROK, and its KUH-1 Surion, a Korean medium maneuver helicopter, as the case of the ROK military aircraft (See Figure 2). Over four years from 2006, Surion had been developed primarily by KAI, Agency for Defense Development (ADD) and Korea Aerospace Research Institute (KARI) jointly with Eurocopter, to replace the aging small attack helicopter 500 MD and the small maneuvering helicopter UH-1H for the ROK Army. It was first deployed in December 2012, and was planned to produce approximately 220 units for the army, replacing 500 MD and UH-1H sequentially [53].

Regarding the Surion, the 2013–2018 data used for this study were collected from the DELIIS, which manages information related to maintenance and supply of the weapon systems in South Korea. In addition, the spare parts consumption data, which had been caused by the maintenance of the defect in the warranty period, were collected by acquiring management data of the manufacturer, KAI. Because we confirmed that the inputs of spare parts consumed during the warranty period were missing in the DELIIS, we added KAI’s data to analyze all the spare parts consumption that occurred during the operation of the military aircraft. In addition, if a certain spare part was retrofitted for any reason, the spare part’s name might be changed. So, for that case, we considered the consumption of the spare part name changed the same as the consumption of the previous spare part name.

Moreover, to consider only the irregularly occurring and consumable spare parts, as mentioned in the Section 1.3, only the spare parts, unscheduled and unrepairable, were extracted from the collected data by using the maintenance characteristics and usage characteristics. In the case of scheduled spare parts, they were excluded from our study because demand forecasting on them is easy to do as the exchange is carried out according to a set time plan. In addition, repairable spare parts have the characteristics of being recycled or scrapped depending on the condition of spare parts at the maintenance. Such repairable spare parts were excluded from our analysis because they are to be recycled in principle and their disposal rate is very low, about 3% according to our interviews on the PBL practitioners of KAI.

In the end, the collected data related to consumption performance could be arranged by 236 spare parts (unscheduled and unrepairable) and six years (2013–2018). In other words, we processed the collected data to be a matrix M with m = 1536 rows (=256 instances $\times$ six years) and has (n + 1) columns (=n feature variables + a target variable).

2.2. Data Representation

To forecast the demand of a spare part, we need to represent the consumption of a spare part by its related feature variables. Hence, by pooling the matrix of our collected data, M, over 236 spare parts and using the previous two years, (t − 1) and (t − 2), to generate additional features for year t, we constructed the data of 944 instances over the four years, ranged from 2015 to 2018.

In detail, the target variable, which we aimed to forecast as the demand of a spare part in year t (t = 2015, …, 2018), can be defined as

$$d_t=\mathrm{the}\mathrm{number}\mathrm{of}\mathrm{a}\mathrm{spare}\mathrm{part},\mathrm{consumed}\mathrm{in}\mathrm{year}t.$$

Moreover,

Table 3. Definition of managerial features, used for forecasting the spare parts demand of military aircraft in year t, d_t (t = 2015, …, 2018).

Feature Set		Feature	Definition	Type	Mean	S.D.
F1	Reliability	f1,1	Design reliability, defined as the mean time (in hours) between failures of a spare part and estimated when its weapon system was developed, i.e., mean time between failures (MTBF)	Numerical	3,471,347.963	36,731,312.509
		f1,2	Operational reliability, measured by the consumption rate of a spare part after aircraft placement	Numerical	16,468.541	15,081.829
		f1,3	Calibration reliability, measured by the calibration value between f1,1 and f1,2	Numerical	57,790.487	55,011.342
F2	Information about a spare part	f2,1	The acquisition price of a spare part	Numerical	742,957.3385	2,265,494.688
		f2,2, f2,3, …, f2,20	The binary features to represent the spare part system type (categorical). The types were 1, 2, …, 19, classified by the shape and function of spare parts, and they were mapped by the one hot encoding method to 19 binary digit features, e.g., (f2,2, f2,3, …, f2,20) = (1, 0, …, 0) for the type 1.	Categorical (encoded as binary digit features)	-	-
		f2,21	Lead time, which is the period time (in days) from the purchase request to the completed record for a spare part	Numerical	8.792	3.789
F3	Operating environment	f3,1	The number of aircraft, operated with a spare part, in year (t − 1)	Numerical	66.000	21.023
		f3,2	The number of sorties of aircraft, related to a spare part, in year (t − 1)	Numerical	5243.500	1145.620
		f3,3	The operating time (in hours) of an aircraft, related to a spare part, in year (t − 1)	Numerical	8289.460	1342.853
		f3,4	The aircraft utilization rate in year (t − 1)	Numerical	77.750	7.762
F4	Consumption performance	f4,1	The number of consumption of a spare part in year (t − 1), i.e., d(t − 1)	Numerical	2.375	13.769
		f4,2	The number of consumption of a spare part in year (t − 2), i.e., d(t − 2)	Numerical	3.833	24.178

lists and explains features, which were obtained from the collected data, grouped into the four managerial feature sets. First, the consumption performance of a spare part in the previous two years is used as two feature variables to represent the consumption trend in the past. In addition, feature variables that correspond to the reliability, information about spare parts, and operating environment were added.

Particularly in the operating environment feature set, the operating time and the number of flights data are best to use if they are by spare parts. However, we could obtain such data only at the aircraft level from the DELIIS and the KAI. Nevertheless, because the operation of the aircraft means that all the aircraft’s spare parts are operated, we used the aircraft operating time and the number of flights data to represent the operating time and the number of flights for a spare part in this study.

2.3. Data Mining-Based Prediction

For our study, the target variable (i.e., the future demand of a spare part) is numerical, and therefore data mining techniques, which allow the continuous target variable and commonly were used in the previous studies, were selected for our forecasting the demands of spare parts. Those data mining techniques are random forest, support vector regression, linear regression, and neural network (i.e., multilayer perceptron). For each of the data mining techniques, we performed a 10-fold cross validation as an experiment, and repeated the same experiments 30 times. For each of the experimental repetitions, we used a different random seed, but the random seed was kept identical for the same iteration of different data mining techniques, by referring to the previous studies [13,54]. To implement the four data mining techniques, we programmed JAVA codes based on the data mining toolkit WEKA (Waikato Environment for Knowledge Analysis) version 3.7.0 with default settings, because it is the well-known open-source toolkit for solving data mining problems.

2.4. Evaluation with Comparisons

In this component, we evaluated the performance of the four data mining techniques, based on the prediction results. To do so, mean absolute error (MAE) and root mean squared error (RMSE) were used because they have been used to evaluate the performance in the previous studies, whose target variables have continuous values [55]. To explain, MAE and RMSE are defined as follows: First, MAE is measured by summing the differences between the predicted values and the actual values, and averaging the absolute of the sum; second, RMSE is obtained by summing the squares of the differences between the predicted values and the actual values, and rooting the average of the sum. Thus, the smaller value of either MAE or RMSE indicates the less prediction error, and implies better performance.

Moreover, the best technique was selected among the four data mining techniques, and it was compared with the conventional time series forecasting techniques as the benchmarking methods (i.e., simple moving average, weighted moving average, and exponential smoothing), which have been adopted typically by the previous works of Table 2.

To compare each feature set, F_i, with the full feature set that contains the four managerial feature sets (i.e., F = F₁ + F₂ + F₃ + F₄), first we repeated the following three steps for each feature set: selecting a feature set to investigate, removing the feature set from the full feature set, and generating the new feature sets for experiments (i.e., F_i-removed feature sets, F_i^(-) (e.g., F₁^(-) = F₂ + F₃ + F₄)). Second, using the best technique selected, we repeated a 10-fold cross validation experiment 30 times. Third, we performed pairwise t tests based on experiments to investigate whether removing the feature set raises the increase of errors in a statistical way. In other words, we investigated if a feature set could improve the prediction performance surely.

In addition, based on these results, the performance rankings of the four managerial feature sets were evaluated, enabling us to consider which feature set needs to be managed more carefully and included by priority to obtain better prediction performance.

3. Results

3.1. Evaluation Results of Data Mining Techniques

For each of the four data mining techniques, the average value of a performance metric was obtained by repeating the same experiments 30 times, which have different random seeds. They are shown in

Table 4. Evaluation results of the repeated experiments with four data mining techniques.

Performance Metric	Random Forest		Support Vector Regression		Linear Regression		Neural Network
	Mean	S.D.	Mean	S.D.	Mean	S.D.	Mean	S.D.
MAE	4.406	0.193	4.627	0.097	6.445	0.204	6.996	1.483
RMSE	21.936	1.370	24.458	0.643	30.268	3.527	33.096	10.215

Note: The best evaluation result is highlighted as bold.

. Among the four data mining techniques for the experiments, random forest gave the best results, which show a smaller value than the other data mining techniques in terms of both MAE and RSME. The possible reason for this can be that random forest uses the ensemble technique to improve accuracy by combining prediction results of several decision trees and it reduces overfitting probability by using a subset of features [56].

3.2. Comparisons between Our Best Data Mining Technique and Traditional Time Series Methods

To compare our best prediction technique (i.e., random forest), with the conventional time series methods as benchmarks, the same experiments for demand forecasting were performed using the simple moving average method, weighted moving average method, and exponential smoothing method, which are being used commonly among the ROK forces.

In detail, the simple moving average method estimated d_t by averaging the spare parts demand of the past two years, d_{(t − 1)} and d_{(t − 2)} (i.e., ${\widehat{d}}_t=\frac{1}{2}\times \left(d_{\left(t-1\right)}+d_{\left(t-2\right)}\right)$). In case of the exponential smoothing method, d_t was estimated by reflecting 70% of the previous year’s performance, d_{(t − 1)}, and 30% of the predicted value for d_{(t − 1)} (i.e., ${\widehat{d}}_t=0.7\times d_{\left(t-1\right)}+0.3\times {\widehat{d}}_{\left(t-1\right)}$). The weighted moving average method predicted d_t by giving different weights to the past two years (i.e., ${\widehat{d}}_t=0.6\times d_{\left(t-1\right)}+0.4\times d_{\left(t-2\right)}$).

Consequently, our best technique random forest showed better results than all three conventional time series methods as shown in

Table 5. Evaluation results of conventional time series methods, compared with the best technique.

Performance Metric	Random Forest	Simple Moving Average	Weighted Moving Average	Exponential Smoothing
	Mean	Mean	Mean	Mean
MAE	4.406	4.875	4.923	4.866
RMSE	21.936	24.484	24.668	24.581

Note: The best evaluation result is highlighted as bold.

. Here, the evaluation results for each conventional time series method were obtained by averaging its performance metrics over 2015–2017.

3.3. Comparisons on Different Managerial Feature Sets

Table 6. Comparison results on different managerial feature sets.

			Full Feature Set	Fi-Removed Feature Set, Fi(-)
			F	F1(-)	F2(-)	F3(-)	F4(-)
Performance Metric	MSE	Mean	4.406	5.766	4.343	5.158	4.325
		S.D.	0.193	0.198	0.196	0.188	0.150
	RMSE	Mean	21.936	24.489	22.084	22.775	21.983
		S.D.	1.370	1.295	1.392	1.281	1.359
Hypothesis test	H1: Fi improved prediction performance in terms of MSE	t	-	26.913	−1.247	15.306	−1.801
		p-value	-	0.000 ***	0.216	0.000 ***	0.0771 *
	H2: Fi improved prediction performance in terms of RMSE	t	-	7.419	0.415	2.450	0.133
		p-value	-	0.000 ***	0.680	0.017 **	0.895
Rank of Fi by contribution	In terms of MSE		-	1	3	2	3
	In terms of RMSE		-	1	3	2	3

Note: Significance levels are * p < 0.1, ** p < 0.05, and *** p < 0.01.

shows the experimental results, which we obtained by repeating 30 times of a 10-fold cross validation for each of F_i-removed feature sets, F_i^(-), after selecting and using random forest as the best technique. Particularly, the pairwise t tests based on experimental results showed whether F_i^(-) influenced the change of errors. In detail, when F₁ and F₃ were removed respectively, MSE and RMSE increased in a statistically significant way (<0.05). This indicates that F₁ and F₃ respectively contributed to the better prediction and so we can accept the hypotheses, H1 for F₁ and H2 for F₃. Based on the pairwise t tests, the performance rankings of the four managerial feature sets were evaluated as F₁ > F₃ > F₂ = F₄ in terms of both MSE and RMSE, representing that the reliability feature set F₁ ranked highest. Thus, the rankings can be considered and used when we should focus on some managerial features due to limited resources for better prediction performance.

4. Conclusions

The demand forecasting of spare parts is crucial not only for nations to strengthen their military combat readiness and use budget efficiently, but also for businesses like PBL contractors to carry out follow-up supports without loss. Hence, to improve the prediction accuracy and consider managerial aspects for the prediction, we came up with using data mining techniques for predicting the spare parts demand of military aircraft. We also used managerial features, which can be grouped into four feature sets: reliability of spare parts, information of spare parts, operating environment of military aircraft, and the consumption of spare parts. Moreover, according to steps in our suggested research framework, we investigated which data mining technique performs best, and which managerial feature set is more contributable to improving prediction performance.

Consequently, among the four data mining techniques, random forest showed the best prediction performance, and it also outperformed the conventional time series methods, which have been used commonly in the military sector. These results gave an answer to RQ1, and it is consistent with the previous works in Table 2, which showed that data mining techniques are superior to the conventional time series methods.

In addition, we confirmed that features related to reliability, introduced and used by this study for the first time, and operation environment turned out significant to the demand forecasting of spare parts. This filled up the research gaps regarding the features, and resolved RQ2. On the other hand, the other types of features turned out less effective in forecasting the demand of spare parts. Our proposed research framework can be used to improve the prediction performance by identifying, focusing, and developing highly relevant features, as well as removing features that may hinder from improving the prediction performance.

Eventually, our study can help for nations to improve the combat readiness of the military weapon system and use the budget effectively by reducing the cost of operating and maintaining the military weapon system’s total life cycle. In addition, when considering the current situation that the responsibility for spare parts management has been transferred from the military sector to the business side by the PBL projects, we expect this study will have a significant effect among the businesses in the relevant industries by avoiding loss costs and thereby increasing their profits.

Further research can be made to overcome the limitations of this study:

First, Surion aircraft, targeted by our study, are still in delivery to the ROK military, but, after the delivery is completed, we will face different data set. For example, the number of spare parts, consumed after the delivery, will inevitably increase as the number of aircraft, flight time, and the number of flights operated increase. In a similar vein, the ROK army is in the process of developing the capabilities for maintaining Surion aircraft. It also means that the types of spare parts that will be consumed may change in the future. Therefore, we need to continually update the data set, and repeat applying our research framework to the same case. In this regard, we also plan to seek a different weapon system that has been delivered completely, and apply our research framework to the weapon system.

Second, without a doubt, data reliability is the most important part to improve the accuracy of forecasting the demand of spare parts. For this reason, our study also tried to make up for the missing input data of DELIIS by using the manufacturer KAI’s data, generated during the operation period. Likewise, the continuous attention and management and relevant policy should be made to assure data reliability. Accordingly, further research can be built upon the more reliable data.