Cost Estimating Using a New Learning Curve Theory for Non-Constant Production Rates

Abstract

Traditional learning curve theory assumes a constant learning rate regardless of the number of units produced. However, a collection of theoretical and empirical evidence indicates that learning rates decrease as more units are produced in some cases. These diminishing learning rates cause traditional learning curves to underestimate required resources, potentially resulting in cost overruns. A diminishing learning rate model, namely Boone’s learning curve, was recently developed to model this phenomenon. This research confirms that Boone’s learning curve systematically reduced error in modeling observed learning curves using production data from 169 Department of Defense end-items. However, high amounts of variability in error reduction precluded concluding the degree to which Boone’s learning curve reduced error on average. This research further justifies the necessity of a diminishing learning rate forecasting model and assesses a potential solution to model diminishing learning rates.

1. Introduction

The U.S. Government Accountability Office (GAO) critiqued the cost and schedule performance of the Department of Defense (DoD)’s $1.7 trillion portfolio of 86 major weapons systems in their 2018 “Weapons System Annual Assessment.” The GAO cited realistic cost estimates as a reason for the relatively low cost growth of the portfolio in comparison to earlier portfolios [1]. Congress and its oversight committees maintain a watchful eye on the DoD’s complex and expensive weapons system portfolio. Inefficient programs are scrutinized and may be terminated if inefficiencies persist. Funding of inefficient programs will also lead to the underfunding of other programs. In the public sector, these terminated and underfunded programs may result in capability gaps that negatively impact our nation’s defense. In the private sector, the inefficient use of resources often spells failure for a company.

A key to the efficient use of resources is accurately estimating the resources required to produce an end-item. Learning curves are a popular method of forecasting required resources as they predict end-item costs using the item’s sequential unit number in the production line. Learning curves are especially useful when estimating the required resources for complex products. The most popular learning curve models used in the government sector are over 80 years old and may be outdated in today’s technology-rich production environment. Additionally, researchers have demonstrated both theoretically and empirically that the effects of learning slow or cease over time [2,3,4].

A new model, named Boone’s learning curve, has been recently proposed to account for diminishing rates of learning as more units are produced [5]. The purpose of this research is to survey the need for alternative learning curve models and further examine how Boone’s learning curve performs in comparison to the traditional learning curve theories in predicting required resources. This research uses a large number of diverse production items to compare Boone’s model to the traditional theories of Wright and Crawford. While many different learning curve models exist (i.e., DeJong, Stanford B, Sigmoid, etc.), some of these others may not be as accurate in cases where the learning rate decreases over time. The next section is a review of the learning curve literature relevant to diminishing learning rates, followed by a description of our methodology and analysis to compare Boone’s learning curve to traditional models. We conclude the paper discussing managerial implications and limitations followed by recommendations for the way forward.

2. Literature Review and Background

The two learning curve models cited by the GAO Cost Estimating and Assessment Guide (2009) are Wright’s cumulative average learning curve theory developed in 1936 and Crawford’s unit learning curve theory developed in 1947. Although both learning curve theories use the same general equation, the theories have contrasting variable definitions. Wright’s learning curve is shown in Equation (1):

$$\overline{Y}=A{x}^b$$

(1)

where $\overline{Y}$ is the cumulative average cost of the first x units, A is the theoretical cost to produce the first unit, x is the cumulative number of units produced, and b is the natural logarithm of the learning curve slope (LCS) divided by the natural logarithm of two. Note, the LCS is the complement of the percent decrease in cost as the number of units produced doubles. For example, with a learning curve slope of 80% and a first unit cost of 100 labor hours, the average cost of the first two units would be 80 labor hours, or 60 labor hours for the second unit. Regardless of the number of units produced, there is a constant decrease in labor costs with each doubling of units due to the constant learning rate.

Several years following the creation of Wright’s cumulative average learning curve theory, J.R. Crawford formulated the unit learning curve theory. Crawford’s theory deviates from Wright’s by assuming that the individual unit cost (as opposed the cumulative average unit cost) decreases by a constant percentage as the number of units produced doubles. Crawford’s model is shown in Equation (2):

$$Y=A{x}^b$$

(2)

where Y is the individual cost of unit x, A is the theoretical cost of the first unit, x is the unit number of the unit cost being forecasted, and b is the natural logarithm of the LCS divided by the natural logarithm of two. For example, with a learning curve slope of 80% and a first unit cost of 100 labor hours, the cost of the second unit would be 80 labor hours. Note, Crawford’s unit theory is the similar to Wright’s in function form; but the difference arises in the variable interpretation lead to a different forecast.

Figure 1 below shows a comparison between Wright’s and Crawford’s theories using the two numerical examples provided. Cumulative average theory and unit theory will produce different predicted costs provided the same set of data despite all predicted costs being normalized to unit costs. Figure 1 demonstrates this point where unit theory was used to generate data using a first unit cost of 100 and a learning curve slope of 90%. The original unit theory data was converted to cumulative averages in order to estimate cumulative average theory learning curve parameters.

Cumulative average theory learning curve parameters. Cumulative average theory estimated a learning curve slope of 93% and a first unit cost of 101.24. These Cumulative Average Theory parameters were then used to predict cumulative average costs. These predicted costs were then converted to unit costs. This conversion allows for the cumulative average predictions to be directly compared to the original Unit Theory generated data. As shown in Figure 1, the cumulative average learning curve predictions first overestimate, then underestimate, and ultimately overestimate the generated unit theory data for all remaining units. Together, Wright’s and Crawford’s theories form the basis of the traditional learning curve theory.

One assumption of these traditional learning curve theories is that they only apply to processes that may benefit from learning. Typically, these costs are only a subset of total program costs; hence appropriate costs must be considered when applying learning curve theory to yield viable parameter estimates. In a complex program, costs can be viewed in a variety of ways to include recurring and non-recurring costs, direct and indirect costs, and costs for various activities and combinations of end-items that can be stated in units of hours or dollars. Learning curve analysis focuses solely on recurring costs in estimating parameters because these costs are incurred repeatedly for each unit produced [6]. Researchers have also focused solely on direct labor costs due to the theoretical underpinnings of learning occurring at the laborer level [2,3]. Additionally, researchers have historically studied end-items that include only the manufactured or assembled hardware and software elements of the end-item [2,3]. Lastly, labor hours in lieu of labor dollars are generally used in analysis so that data can be compared across fiscal years without the need to adjust for inflation. Therefore, the literature indicates using direct, recurring, labor costs in units of labor hours. These costs should be considered only for the certain elements that include the manufacturing or assembly of hardware and software of an end-item.

An implicit assumption in the traditional learning curve theories is that knowledge obtained through learning does not depreciate. However, empirical evidence demonstrates that knowledge depreciates in organizations [7,8]. Argote [7] showed that knowledge depreciation occurs at both the individual and the organizational levels. Many variations of the traditional models make use of the concept of performance decay (commonly called forgetting) to model non-constant rates of learning. Forgetting and its relationship to learning can take many forms and is essential to consider in contemporary learning curve analysis.

Forgetting is the concept that an individual or organization will experience a decline in performance over time resulting in non-constant rates of learning. Badiru [4] theorizes that forgetting and resulting performance decay is a result of factors “including lack of training, reduced retention of skills, lapse in performance, extended breaks in practice, and natural forgetting” (p. 287). According to Badiru [4], these factors may be caused by internal processes or external factors. Badiru [4] lists three cases in which forgetting arises. First, forgetting may occur continuously as a worker or organization progresses down the learning curve due in part to natural forgetting [4]. The impact of forgetting may not wholly eclipse the impact of learning but will hamper the learning rate while performance continues to increase at a slower rate. Second, forgetting may occur at distinct and bounded intervals, such as during a scheduled production break [4] or towards the end of production as workers are transferred to other duties. Finally, forgetting may intermittently occur at random times and for stochastic intervals such as during times of employee turnover [4]. Others have expanded on the causes of forgetting and have drawn similar conclusions to Badiru [4,9,10,11]. This decline in performance decays the learning rate and causes longer manufacturing times and higher costs than would be forecasted using traditional learning curve theory.

The concept of forgetting and its impact on non-constant rates of learning has proven relevant in contemporary learning curve research. Several forgetting models have been developed to include the learn-forget curve model (LFCM) [11], the recency model (RCM) [12], the power integration and diffusion (PID) model [13], and the Depletion-Power-Integration-Latency (DPIL) model [13] among others [10]. However, these forgetting models focus solely on the phenomenon of forgetting due to interruptions of the production process [9,10,14]. Jaber [9] states that “there has been no model developed for industrial settings that considers forgetting as a result of factors other than production breaks” (pp. 30–31) and mentions this as a potential area of future research. Although forgetting models have emerged after Jaber’s [9] article, a review of the popular forgetting models cited confirms Jaber’s statement.

A related concept to the forgetting phenomenon is the plateauing phenomenon. According to Jaber [9] (2006), plateauing occurs when the learning process ceases and manufacturing enters a production steady state. This ceasing of learning results in a flattening or partial flattening of the learning curve corresponding to rates of learning at or near zero. There remains debate as to when plateauing occurs in the production process or if learning ever ceases completely [3,9,15,16,17]. Jaber [9] provides several explanations to describe the plateauing phenomenon that include concepts related to forgetting. Baloff [18,19] recognized that plateauing is more likely to occur when capital is used in the production process as opposed to labor. According to some researchers, plateauing can be explained by either having to process the efficiencies learned before making additional improvements along the learning curve or to forgetting altogether [20]. According to other researchers, plateauing can be caused by labor ceasing to learn or management’s unwillingness to invest in capital to foster induced learning [21]. Related to this underinvestment to foster induced learning, management’s doubt as to whether learning efficiencies related to learning can occur is cited as another hindrance to constant rates of learning [22]. Li and Rajagopalan [23] investigated these explanations and concluded that no empirical evidence supports or contradicts them while ascribing plateauing to depreciation in knowledge or forgetting. Jaber [9] concludes that “there is no tangible consensus among researchers as to what causes learning curves to plateau” and alludes that this is a topic for future research (pp. 30–39).

Despite the controversy in the research surrounding forgetting and plateauing effects, empirical studies have shown learning curves to exhibit diminishing rates of learning. For instance, the plateauing phenomenon at the tail end of production was investigated by Harold Asher in a 1956 RAND study. The U.S. Air Force contracted RAND after the service noticed traditional learning curves were underestimating labor costs at the tail end of production [3]. Asher intended to study if the logarithmically transformed traditional learning curves were approximately linear. This linearity would indicate constant rates of learning throughout the production cycle. The alternative hypothesis for these learning curves was a convexity of the logarithmically-transformed traditional learning curves that would indicate diminishing rates of learning as the number of units increased [3]. An example of a learning curve with a diminishing learning rate is shown in Figure 2 in logarithmic scale. The first unit cost is 100 with an initial learning curve slope of 80% decaying at a rate of 0.25% with each additional unit. For example, the second unit’s learning curve slope is 80.25%.

Asher investigated this hypothesis of convex logarithmically transformed learning curves by analyzing the learning curves of the various shops within a manufacturing department producing aircraft. Asher used airframe cost data with the appropriate amount of detail to perform a learning curve analysis on the lower level job shops within the manufacturing department. He divided the eleven major kinds of aircraft manufacturing operations into four shop groups each with a set of direct labor cost data [3]. If non-constant rates of learning were present, the shop group curves would differ in their rates of learning and may themselves be convex in logarithmic scale. This would indicate their aggregate learning curve would also be convex in logarithmic scale.

Asher’s results showed that the learning curves of the manufacturing shop group had different learning slopes and were convex in logarithmic scale [3]. Asher claims the convexity within the manufacturing shop group learning curves is due to the disparate operations within the job shops and stated that each had their own unique learning curve [3]. He asserts that a linear approximation is reasonable for a relatively small quantity of airframes produced but becomes increasingly unwarranted for larger quantities. This is due in part because larger quantities of produced end-items are likely to experience diminishing rates of learning. Moreover, highly aggregated learning curves are also likely to experience diminishing rates of learning. Because the aggregated manufacturing cost curve is usually the lowest level of detail on which learning curve analysis is performed, the manufacturing cost curve will have diminishing rates of learning as cumulative output increases. These results further justify a learning curve model with diminishing rates of learning.

Wright’s and Crawford’s learning curve theories provided the basis of the traditional approach that learning occurs at a constant rate as the number of units produced increases. Since this initial discovery, several log-linear learning curve models were founded in attempts to more accurately model data from manufacturing processes. These contemporary models diverge from constant rates of learning by including adjustments in various forms. The six most popular models (including the traditional model) are shown in Figure 3 in logarithmic scale and include log-log graphing lines to more clearly illustrate the differences between models. These illustrated models include the traditional log-linear model or Wright/Crawford curves, the plateau model [19], the Stanford-B model [24], the De Jong model [25], the S-curve model [21], and Knecht’s upturn model [26].

Recent studies have investigated whether the Stanford-B, De Jong, and S-Curve models more accurately predict program costs in comparison to the traditional theories. Moore [16] and Honious [17] studied how prior experience in the manufacturing of an end-item along with the proportion of touch labor in the manufacturing process affected the accuracy of the Stanford-B, De Jong, and S-curve models in comparison to the traditional models. The authors concluded that these models improved upon the traditional curves for only a narrow range of parameter values. Their research provided insight that the traditional learning curve models become less accurate at the tail-end of production when the proportion of human labor is high in the manufacturing process. Moreover, Honious [17] explicitly references a plateauing effect at the end of production. These findings provide further justification for investigating non-constant rates of learning.

The Stanford-B, De Jong, and S-Curve univariate models illustrated in Figure 3 alter the resulting learning curve slope based on alterations to the theoretical first unit cost parameter A. However, the learning curve slopes of these models are not directly a function of the number of cumulative units produced. The plateau model and Knecht’s upturn model also illustrated in Figure 3 each produce a learning curve whose slope is directly affected by the number of cumulative units produced. The plateau model uses a step function to reduce the learning rate to 0% (i.e., the learning curve slope is 100%) past a certain number of cumulative units produced. In contrast, Knecht’s Upturn Model amends the learning curve exponent term b by multiplying b by Euler’s number e raised to the term of a constant multiplied by the number of cumulative units produced. Mathematically, this is expressed $\overline{Y}=A{x}^{b·e^{xc}}$, where $\overline{Y}$is the cumulative average unit cost, A is the theoretical first unit cost, x is the number of cumulative units produced, b is the natural logarithm of the learning curve slope divided by the natural logarithm of 2, and c is a constant. The forgetting models stated within the manuscript also amend the learning curve slope based indirectly on the number of cumulative units but only apply when interruptions to the production process occur.

In response to these researchers’ findings, Boone [5] developed a learning curve model with a learning rate that diminishes as more units are produced. Conversely, the traditional learning curve theories diminish the rate of cost reductions as the number of units produced doubles. However, the existing literature provides evidence that the cost reductions with each doubling of units may not be constant as the number of units produced increases. Therefore, Boone [5] sought to attenuate the cost reductions that occur with each doubling of units produced by decreasing the learning rate as the number of units increases.

Boone [5] devised a model that decreases the learning curve exponent b as the number of units produced x increases. He first considered a model without an additional parameter to reduce the learning curve exponent b directly by the unit number. However, he decided to temper the effect each additional unit has on the parameter b by adding an additional parameter c. The resulting learning curve is shown in Equation (3):

$$\overline{Y}=A{x}^{\frac{b}{1+\frac{x}{c}}}$$

(3)

where $\overline{Y}$ is the cumulative average cost of the first x units, A is the theoretical cost to produce the first unit, x is the cumulative number of units produced, b is the natural logarithm of the learning curve slope (LCS) divided by the natural logarithm of two, and c is a positive decay value. For example, a learning curve slope of 80%, first unit cost of 100 labor hours, and decay value of 100, Boone’s model yields a cumulative average cost at the second unit of 80.35 labor hours—or 60.70 labor hours for the second unit. What began as an 80% learning curve model has decayed to an 80.35% learning curve for the second unit. In comparison to Wright’s learning curve using the same parameters, the effect of learning has decreased slightly in the production of unit two. The inclusion of the decay value increases the learning curve slope, and hence decreases the learning rate as more units are produced. Note, Boone’s model can also be modified to incorporate Crawford’s unit theory–refer to Equation (3) for the necessary modifications.

Boone’s learning curve diverges from the constant learning assumptions in both Wright’s and Crawford’s learning curve models by incorporating the unit number in the denominator of the exponent—thus decreasing the effect of b as the number of units produced increases. Furthermore, the decay value moderates this diminishing effect, so the amount of learning decreases more slowly. In general, Boone’s model is flatter near the end of production and steeper in the early stages compared to the traditional theories. Note, as the decay value approaches zero (holding other factors constant), the exponent term approaches zero representing a learning curve slope approaching 100%. As the decay value approaches infinity, the parameter b remains constant, and Boone’s learning curve simplifies to the traditional learning curve [5].

Boone [5] tested his learning curve using unit theory to provide a consistent comparison to Crawford’s learning curve. Based on the scope of his research and lack of comparison using cumulative average theory, a more robust examination and analysis of Boone’s learning curve should be accomplished.

3. Methodology

One goal of this research is to examine the accuracy of Boone’s learning curve in comparison to the popular Wright and Crawford learning curve theories. In order to perform this analysis, production cost and quantity data from a diverse set of DoD systems was collected from government Functional Cost-Hour Reports, Progress Curve Reports, and the Air Force Life Cycle Management Center Cost Research Library. The dataset consisted of recurring costs (either in dollars or labor hours) by production lot for 169 unique end-items. Our data included end-items from a variety of systems (i.e., bomber, cargo, and fighter aircraft, missiles, and munitions), contractors, and time periods (1957–2018). Additionally, only production runs with at least four lots were included. The dataset for the Cumulative Average Theory analysis only includes 140 of the 169 end-items. This theory relies on continuous data because each lot’s cumulative average cost and cumulative quantity is a function of all previous lots’ costs and quantities. In order to compare Boone’s model to the traditional theories, each model will be fitted to data: (1) Boone’s and Wright’s models using cumulative average theory, and (2) Boone’s and Crawford’s models using unit theory. Then, the predicted values for each model will be compared to the actual costs using root mean squared error (RMSE) and mean absolute percentage error (MAPE).

Labor costs were collected from the work breakdown structure (WBS) for the specific item being manufactured (e.g., aircraft frame) or from the documentation provided by the government. Our data included three broad functional cost categories: labor, material, and other. These costs are included in both forms of recurring and non-recurring costs. There are also four functional labor categories delineated that include manufacturing, tooling, engineering, and quality control labor. These four labor category costs, when summed with the material costs and other costs, comprise the total cost for each WBS element for recurring and non-recurring costs.

The definition for the manufacturing labor cost category most clearly aligns with the extant literature to be the focus as the pertinent labor cost category for learning curve research. According to the WBS elements, the manufacturing labor category “includes the effort and costs expended in the fabrication, assembly, integration, and functional testing of a product or end item. It involves all the processes necessary to convert raw materials into finished items [28].” This manufacturing labor category aligns with the categories examined by Wright, which he called “assembly operations [2],” along with those cost categories Crawford studied, which he called “airframe-manufacturing processes [3].” Therefore, the manufacturing labor cost category as defined by the government is associated with the types of labor costs studied by traditional learning curve theorists and succeeding research.

The learning curve parameters for each model (i.e., Equations (1)–(3)) will be estimated by minimizing the sum of squares error (SSE) using Excel’s generalized reduced gradient (GRG) nonlinear solver and evolutionary solver. The SSE is calculated by squaring the vertical difference of the observed data and predicted data for each lot and summing these squared differences across all lots.

With lot data, cumulative theory models can be estimated directly. Conversely, when utilizing unit learning curve theory, Crawford’s and Boone’s models are estimated using an iterative process based on lot midpoints, adapted from Hu and Smith [29]. The algebraic lot midpoint is defined as “the theoretical unit whose cost is equal to the average unit cost for that lot on the learning curve” [6]. The lot midpoint supplants using sequential unit numbers when using lot cost data.

Lot midpoints and model parameters are calculated iteratively due to the lack of a closed-form solution for the lot midpoint. First, an initial lot midpoint (for each lot) is determined using a parameter-free approximation formula [6]—see Equation (4):

$$\mathrm{Lot} \mathrm{Midpoint}\left(\mathrm{LMP}\right)=\frac{F+L+2\sqrt{FL}}{4}$$

(4)

where F is the first unit number in a lot and L is the last unit number in a lot. These lot midpoint estimates are then used to estimate the learning curve parameters for Crawford’s model (Equation (2)) using the GRG non-linear optimization algorithm. Next, using the estimated parameter b, a new set of lot midpoints are determined using a simple and popular formula—Asher’s Approximation [6]; see Equation (5):

$$\mathrm{Lot} \mathrm{Midpoint}\approx {\left[\frac{{\left(L+\frac{1}{2}\right)}^{b+1}-{\left(F-\frac{1}{2}\right)}^{b+1}}{\left(L-F+1\right)\left(b+1\right)}\right]}^{\left(\frac{1}{b}\right)}$$

(5)

where F is the first unit number in a lot, L is the last unit number in a lot, and b is the estimated value from Equation (2). Learning curve parameters will then be re-estimated using these more precise lot midpoint estimates. The iterative process is repeated until changes between successive values of the estimated lot midpoints and b are sufficiently small [29] (see Appendix A for a summary of this process). In order to use an iterative process for Boone’s model, Asher’s Approximation from Equation (5) was adapted to incorporate Boone’s decaying learning curve slope. This adaptation allows the lot costs of Boone’s learning curve to decrease as more units are produced which affects the lot midpoint estimates; the formula is shown in Equation (6):

$${\mathrm{Lot} \mathrm{Midpoint}}_{\mathrm{i}}\approx {\left[\frac{{\left(L+\frac{1}{2}\right)}^{b^{\prime }+1}-{\left(F-\frac{1}{2}\right)}^{b^{\prime }+1}}{\left(L-F+1\right)\left(b^{\prime }+1\right)}\right]}^{\left(\frac{1}{b^{\prime }}\right)}$$

(6)

where F is the first unit number in a lot, L is the last unit number in a lot, $b^{\prime }=\frac{b}{1+\left(\frac{LM{P}_{i-1}}{c}\right)},$ and i is the iteration number.

This iterative process of calculating the lot mid-point then solving a non-linear least squares problem requires the execution of a series of non-linear optimization algorithms. Boone’s model requires the GRG algorithm which found solutions in a longer but still reasonable amount of time. While more burdensome than the traditional models due to the longer run time and the requirement to provide bounds for the parameters. For Boone’s model, the bounds for A and b have a fairly straightforward basis by which to define the bounds. In practice, the A parameter is often supported by a point estimate of the cost of the first theoretical unit. Thus, a bound can be built around this value with tools such as a confidence interval. The b parameter is defined by the learning curve slope which for all practical purposes will be in the (0, 1) interval—most likely on the higher end. As for the c parameter, the basis for the bound is more of a challenge. From a model implementation standpoint, the bound can be arbitrarily large if a long solve time is not limiting. Practically, the bound should be reasonably set; this aspect of the model is an avenue of future research which is discussed in the conclusion. This algorithm does allow the analyst to define stopping conditions such as convergence threshold, maximum number of iterations, or maximum amount of time. Additionally, there is an option called multi-start which uses multiple initial solutions to help locate a global solution verse possibly only finding a local solution. These options allow the user to mitigate the extra burden if necessary. Overall, the computing burden to calculate these models was on the order of minutes per weapon system.

The final estimated parameters for Boone’s model and the traditional learning curves were used to create predicted learning curves. These predicted curves were then compared to observed data. Total model error was calculated by comparing the difference between observations and predicted values to understand how accurately the models explained variability in the data. Two measures were used to determine the overall model error. The first error measure was Root Mean Square Error (RMSE) that is calculated by taking the square root of the total SSE divided by the number of lots. RMSE is not robust to outliers—i.e., the effects of outliers may unduly influence this measure. RMSE is often interpreted as the average amount of error of the model as stated in the model’s original units.

The second measure was mean absolute percentage error (MAPE). MAPE is calculated by subtracting the predicted value from the observed value, dividing this difference by the observed value, taking the absolute value, and multiplying by 100%. These absolute percent errors are then summed over all observations and divided by the total number of observations. MAPE provides a unit-less measure of accuracy and is interpreted as the average percent of model inaccuracy. Unlike RMSE, MAPE is robust to outliers.

After calculating these measures of overall model error, a series of paired difference t-tests are conducted to determine if reductions in error from Boone’s learning curve are statistically significant. In order to conduct the first paired difference t-test, Boone’s learning curve RMSE using cumulative average theory will be subtracted from Wright’s learning curve RMSE, and the difference will be divided by Wright’s learning curve RMSE. This calculation will yield a percentage difference rather than raw difference to compare end-items of varying differences in magnitude equitably. The null hypothesis posits that Boone’s learning curve results in an equal amount (or more) of error in predicting observed values compared to Wright’s learning curve. The alternative hypothesis is that the percentage difference is greater than zero. Support for the alternative hypothesis signifies that Boone’s learning curve results in less error predicting observed values than Wright’s learning curve. This methodology will be repeated five times to examine each learning curve theory using the two error measures and the different units of production costs—see

Table 1. Paired Difference Hypothesis Tests Conducted.

Learning Curve Theory	Error Measure	Units of Measure
Cumulative Average Theory	Root Mean Squared Error Percentage Difference	Total Dollars(K)
		Labor Hours
	Mean Absolute Percent Error Percentage Difference	Total Dollars(K)&Labor Hours Combined
Unit Theory	Root Mean Squared Error Percentage Difference	Total Dollars(K)
	Mean Absolute Percent Error Percentage Difference	Total Dollars(K)&Labor Hours Combined

.

An assumption to utilize the paired difference t-test is that the data are approximately normally distributed. For hypothesis tests with large sample sizes, the central limit theorem can be invoked. Alternatively, a Shapiro–Wilk test will be used to evaluate the normality assumption for small samples. If the Shapiro–Wilk test does not support the normality assumption, the non-parametric Wilcoxon Rank Sum test will be used. A 0.05 level of significance will be used for all statistical tests.

4. Analysis & Results

The detailed results for Wright’s and Boone’s learning curves using cumulative average theory are provided in Appendix B

Table A1. Error Comparison using Cumulative Average Theory.

Program	Number of Lots	Number of Units	Component Estimated	Units	Traditional RMSE	Boone RMSE	RMSE Percentage Difference	Traditional MAPE	Boone MAPE	MAPE Percentage Difference
Program 1	6	483	PME–Air Vehicle	Dollars	557.9	111.7	80.0%	3.6%	0.7%	80.9%
Program 1	6	483	PME–Air Vehicle	Hours	15.5	0.3	98.0%	27.2%	0.5%	98.2%
Program 1	6	483	Airframe	Dollars	411.2	114.1	72.3%	2.8%	0.7%	74.7%
Program 1	6	483	Airframe	Hours	21.7	1.5	93.0%	31.0%	1.7%	94.6%
Program 2	5	638	PME–Air Vehicle	Dollars	129.8	6.5	95.0%	2.6%	0.1%	95.6%
Program 3	5	500	PME–Air Vehicle	Dollars	1630.3	291.1	82.1%	20.8%	3.9%	81.5%
Program 4	19	205	PME–Air Vehicle	Dollars	581.7	581.8	0.0%	3.1%	3.1%	0.0%
Program 4	19	205	Airframe	Dollars	546.0	546.4	−0.1%	3.2%	3.2%	−0.1%
Program 5	7	459	PME–Air Vehicle	Dollars	400.8	44.7	88.8%	2.7%	0.3%	88.2%
Program 5	7	459	Electronic Warfare (1)	Dollars	4.8	3.2	32.3%	7.2%	4.8%	33.7%
Program 6	6	98	PME–Air Vehicle	Dollars	99.3	32.2	67.6%	1.1%	0.3%	69.4%
Program 6	6	98	Electronic Warfare (1)	Dollars	12.7	1.7	86.8%	3.6%	0.6%	82.4%
Program 6	6	98	Electronic Warfare (2)	Dollars	15.0	13.3	11.4%	2.3%	2.0%	12.9%
Program 6	6	98	Electronic Warfare (3)	Dollars	1.8	1.1	40.3%	1.3%	0.8%	39.6%
Program 7	7	110	PME–Air Vehicle	Dollars	145.0	98.3	32.2%	1.0%	0.7%	32.6%
Program 7	7	110	Electronic Warfare (1)	Dollars	8.4	3.6	57.2%	2.7%	1.0%	61.3%
Program 7	7	110	Electronic Warfare (2)	Dollars	140.3	107.2	23.6%	1.2%	0.8%	27.5%
Program 7	7	110	Electronic Warfare (3)	Dollars	0.9	0.9	0.0%	0.5%	0.5%	−0.1%
Program 7	7	110	Electronic Warfare (4)	Dollars	140.7	111.3	20.9%	1.3%	1.0%	24.2%
Program 7	7	110	Electronic Warfare (5)	Dollars	21.3	21.0	1.1%	2.2%	2.1%	5.2%
Program 8	8	3529	PME–Air Vehicle	Dollars	27.7	23.6	14.8%	1.4%	1.3%	7.8%
Program 8	8	3529	PME–Air Vehicle	Hours	0.1	0.1	−27.5%	1.1%	1.3%	−27.9%
Program 9	9	3798	PME–Air Vehicle	Dollars	166.5	170.7	−2.5%	8.4%	8.8%	−3.7%
Program 10	10	3803	PME–Air Vehicle	Dollars	8.0	4.8	39.6%	2.5%	1.2%	51.7%
Program 10	10	3803	PME–Air Vehicle	Hours	24.4	14.0	42.7%	4.3%	2.0%	54.0%
Program 11	6	180	PME–Air Vehicle	Dollars	514.0	508.4	1.1%	0.9%	0.8%	4.2%
Program 12	10	20	PME–Air Vehicle	Dollars	699.2	694.1	0.7%	5.8%	5.7%	1.0%
Program 12	10	20	PME–Air Vehicle	Hours	1042.5	906.5	13.1%	9.5%	8.4%	11.8%
Program 12	7	11	Mission Computer (1)	Dollars	44.3	44.3	0.0%	2.5%	2.5%	0.0%
Program 13	5	100	PME–Air Vehicle	Dollars	53,386.7	21,143.7	60.4%	12.8%	4.8%	62.1%
Program 13	5	100	Airframe	Dollars	6569.7	6578.0	−0.1%	3.7%	3.7%	0.0%
Program 14	5	275	PME–Air Vehicle	Dollars	3114.0	145.5	95.3%	3.8%	0.2%	95.5%
Program 15	10	77	PME–Air Vehicle	Dollars	44,386.0	44,390.2	0.0%	9.5%	9.5%	0.0%
Program 15	12	83	PME–Air Vehicle	Hours	79,242.0	79,247.5	0.0%	6.5%	6.5%	0.0%
Program 15	11	83	Airframe	Dollars	39,624.4	39,628.0	0.0%	10.6%	10.6%	0.0%
Program 15	10	68	Mission Computer (1)	Dollars	1959.3	1959.4	0.0%	17.0%	17.0%	0.0%
Program 16	9	76	PME–Air Vehicle	Dollars	436.3	144.4	66.9%	2.6%	1.0%	62.9%
Program 17	5	50	PME–Air Vehicle	Dollars	13,023.6	13,029.8	0.0%	2.8%	2.8%	−0.1%
Program 18	9	31	PME–Air Vehicle	Dollars	2942.5	2941.9	0.0%	1.0%	0.9%	0.0%
Program 19	6	98	PME–Air Vehicle	Dollars	313.3	313.4	0.0%	0.5%	0.5%	−0.1%
Program 20	11	84	PME–Air Vehicle	Dollars	1568.7	1121.9	28.5%	1.7%	1.5%	7.8%
Program 20	7	59	Electronic Warfare (1)	Dollars	452.8	143.0	68.4%	4.6%	1.3%	71.5%
Program 20	11	84	Electronic Warfare (2)	Dollars	98.7	76.5	22.5%	3.4%	3.6%	−6.3%
Program 20	7	59	Electronic Warfare (5)	Dollars	562.5	517.4	8.0%	1.8%	1.8%	1.7%
Program 21	6	326	PME–Air Vehicle	Dollars	5267.1	2408.8	54.3%	8.0%	4.2%	47.4%
Program 21	7	344	Airframe	Dollars	4819.5	2544.3	47.2%	9.1%	5.4%	40.4%
Program 21	7	344	Avionics	Dollars	763.2	429.9	43.7%	6.6%	3.9%	40.8%
Program 21	14	453	PME–Air Vehicle	Hours	3493.6	3495.9	−0.1%	4.8%	4.8%	0.1%
Program 21	14	453	Airframe	Hours	4338.4	4339.7	0.0%	6.2%	6.2%	0.1%
Program 22	8	538	PME–Air Vehicle	Hours	856.7	857.7	−0.1%	2.5%	2.6%	−0.1%
Program 22	8	538	Airframe	Hours	5608.5	5609.7	0.0%	15.8%	15.9%	−0.1%
Program 23	5	469	PME–Air Vehicle	Dollars	637.5	339.3	46.8%	5.4%	2.9%	47.3%
Program 24	10	59	PME–Air Vehicle	Dollars	3032.5	3033.0	0.0%	2.2%	2.2%	0.0%
Program 25	9	348	PME–Air Vehicle	Dollars	117.8	118.1	−0.2%	0.9%	0.9%	−0.2%
Program 26	5	109	PME–Air Vehicle	Dollars	3247.4	1676.8	48.4%	11.0%	6.0%	45.7%
Program 26	5	109	PME–Air Vehicle	Hours	607.1	453.5	25.3%	5.7%	4.2%	25.9%
Program 27	18	631	PME–Air Vehicle	Dollars	1669.6	913.3	45.3%	3.6%	1.9%	46.2%
Program 28	6	425	PME–Air Vehicle	Dollars	320.0	322.0	−0.6%	0.9%	0.9%	−0.6%
Program 28	7	522	PME–Air Vehicle	Hours	1776.1	1785.6	−0.5%	1.8%	1.8%	−0.1%
Program 28	7	522	Airframe	Hours	1389.9	1393.9	−0.3%	1.2%	1.2%	−0.2%
Program 29	9	358	PME–Air Vehicle	Hours	610.6	611.1	−0.1%	0.9%	0.9%	0.4%
Program 29	9	358	Airframe	Hours	4804.8	2124.2	55.8%	7.3%	2.9%	60.1%
Program 30	5	204	PME–Air Vehicle	Dollars	513.5	212.7	58.6%	1.2%	0.5%	56.1%
Program 31	5	605	PME–Air Vehicle	Dollars	1482.6	629.1	57.6%	6.1%	2.9%	53.1%
Program 32	5	870	PME–Air Vehicle	Dollars	61.3	61.6	−0.5%	0.4%	0.4%	−0.3%
Program 33	10	178	PME–Air Vehicle	Dollars	7093.5	7101.1	−0.1%	3.5%	3.5%	−0.1%
Program 33	10	178	PME–Air Vehicle	Hours	8131.1	8144.1	−0.2%	2.9%	2.9%	−0.1%
Program 33	10	178	Airframe	Dollars	1906.9	1910.8	−0.2%	1.7%	1.7%	−0.2%
Program 33	10	712	Body	Dollars	232.2	234.9	−1.2%	1.5%	1.6%	−1.3%
Program 33	10	178	Alighting Gear	Dollars	76.6	76.6	0.0%	7.9%	7.9%	0.0%
Program 33	10	178	Auxiliary Power Plant	Dollars	90.7	90.7	−0.1%	3.9%	3.9%	−0.1%
Program 33	10	178	Electronic Warfare (1)	Dollars	775.5	776.1	−0.1%	6.5%	6.5%	−0.1%
Program 33	10	178	Electronic Warfare (2)	Dollars	360.1	273.4	24.1%	58.3%	46.0%	21.2%
Program 33	10	178	Electronic Warfare (3)	Dollars	62.5	62.4	0.2%	5.7%	5.7%	0.1%
Program 33	10	178	Empennage	Dollars	352.2	352.3	0.0%	5.1%	5.1%	−0.1%
Program 33	10	178	Hydraulic	Dollars	22.7	22.7	−0.1%	2.2%	2.2%	−0.1%
Program 33	10	178	Wing	Dollars	296.5	296.9	−0.1%	2.3%	2.3%	−0.1%
Program 34	6	67	PME–Air Vehicle	Dollars	11,059.1	11,061.2	0.0%	6.6%	6.6%	0.0%
Program 34	6	67	PME–Air Vehicle	Hours	9058.6	9061.7	0.0%	4.4%	4.4%	0.0%
Program 34	6	67	Airframe	Dollars	2798.1	2004.6	28.4%	2.8%	1.7%	37.9%
Program 34	6	201	Body	Dollars	1924.5	828.9	56.9%	19.0%	8.7%	54.0%
Program 34	6	67	Alighting Gear	Dollars	316.5	166.9	47.3%	17.2%	8.3%	51.9%
Program 34	6	67	Electrical	Dollars	50.7	50.7	−0.1%	1.9%	1.9%	−0.1%
Program 34	6	67	Electronic Warfare (1)	Dollars	428.3	428.4	0.0%	5.3%	5.3%	0.0%
Program 34	5	49	Empennage	Dollars	202.2	202.2	0.0%	4.1%	4.1%	0.0%
Program 34	6	67	EO/IR	Dollars	45.6	36.6	19.7%	1.2%	1.1%	13.1%
Program 34	6	67	EOTS	Dollars	347.6	347.7	0.0%	6.5%	6.5%	0.0%
Program 34	6	67	Hydraulic	Dollars	122.3	101.5	17.0%	8.4%	6.2%	26.8%
Program 34	6	67	Mission Computer (1)	Dollars	484.8	484.9	0.0%	0.9%	0.9%	−0.2%
Program 34	6	67	Surface Controls	Dollars	196.0	196.0	0.0%	4.9%	4.9%	0.0%
Program 34	6	67	Wing	Dollars	998.4	998.6	0.0%	3.3%	3.3%	−0.1%
Program 35	5	41	PME–Air Vehicle	Dollars	3578.6	3579.8	0.0%	1.5%	1.5%	0.0%
Program 35	5	41	PME–Air Vehicle	Hours	2003.7	2004.7	0.0%	1.1%	1.1%	0.0%
Program 35	5	50	Airframe	Dollars	609.3	610.4	−0.2%	0.6%	0.6%	−0.3%
Program 35	5	150	Body	Dollars	235.8	156.5	33.6%	1.9%	1.4%	28.0%
Program 35	5	50	Alighting Gear	Dollars	13.2	13.2	−0.1%	0.5%	0.5%	0.0%
Program 35	5	50	Electronic Warfare (1)	Dollars	259.6	259.7	0.0%	3.2%	3.2%	0.0%
Program 35	5	50	EO/IR	Dollars	121.6	121.7	0.0%	1.3%	1.3%	−0.1%
Program 35	5	50	EOTS	Dollars	177.9	177.9	0.0%	2.8%	2.8%	−0.1%
Program 35	5	50	Hydraulic	Dollars	58.2	58.2	0.0%	3.1%	3.1%	0.0%
Program 35	5	50	Radar	Dollars	256.8	256.9	0.0%	3.2%	3.2%	0.0%
Program 35	5	50	Surface Controls	Dollars	121.5	121.5	0.0%	2.6%	2.6%	0.0%
Program 35	5	50	Wing	Dollars	1213.5	1213.6	0.0%	3.8%	3.8%	0.0%
Program 36	13	1285	PME–Air Vehicle	Dollars	28.8	29.4	−2.1%	0.6%	0.6%	−2.2%
Program 37	6	432	PME–Air Vehicle	Dollars	791.3	793.8	−0.3%	3.4%	3.4%	−0.4%
Program 38	6	52	PME–Air Vehicle	Dollars	253.6	154.9	38.9%	1.2%	0.7%	41.6%
Program 38	6	44	PME–Air Vehicle	Hours	831.5	614.2	26.1%	1.3%	0.8%	42.8%
Program 39	19	1023	PME–Air Vehicle	Dollars	19.3	19.3	−0.2%	0.7%	0.7%	−0.2%
Program 40	5	1725	PME–Air Vehicle	Dollars	19.2	0.6	96.7%	2.0%	0.1%	97.0%
Program 41	10	16	PME–Air Vehicle	Dollars	14,787.6	14,787.8	0.0%	5.2%	5.2%	0.0%
Program 41	10	16	Data Link (1)	Dollars	138.8	138.8	0.0%	3.7%	3.7%	0.0%
Program 42	11	203	PME–Air Vehicle	Dollars	1000.0	1000.1	0.0%	7.0%	7.0%	0.0%
Program 42	11	899	Electronic Warfare (1)	Dollars	67.5	67.7	−0.2%	13.9%	13.9%	−0.5%
Program 43	11	203	PME–Air Vehicle	Dollars	1121.7	1121.9	0.0%	5.5%	5.5%	0.0%
Program 43	13	251	PME–Air Vehicle	Hours	1944.2	1762.2	9.4%	3.4%	3.2%	6.1%
Program 44	5	136	PME–Air Vehicle	Dollars	57.1	16.3	71.4%	1.1%	0.3%	71.4%
Program 45	9	155	PME–Air Vehicle	Dollars	149.6	149.7	−0.1%	0.3%	0.3%	−0.1%
Program 46	6	68	PME–Air Vehicle	Dollars	3435.9	3436.0	0.0%	1.7%	1.7%	0.1%
Program 46	6	68	PME–Air Vehicle	Hours	2286.4	2286.6	0.0%	2.6%	2.6%	0.0%
Program 46	6	68	Airframe	Dollars	539.1	527.6	2.1%	2.3%	2.1%	10.9%
Program 46	6	68	Data Link (1)	Dollars	44.0	44.0	0.0%	3.0%	3.0%	0.0%
Program 46	6	68	Electronic Warfare (1)	Dollars	221.8	221.9	0.0%	5.4%	5.4%	0.0%
Program 46	6	68	Electronic Warfare (2)	Dollars	220.0	220.0	0.0%	6.5%	6.5%	0.0%
Program 46	6	68	Electronic Warfare (3)	Dollars	17.7	8.8	50.4%	2.2%	1.0%	54.6%
Program 46	6	68	Electronic Warfare (4)	Dollars	530.0	530.0	0.0%	5.2%	5.2%	0.0%
Program 46	6	68	EO/IR	Dollars	120.7	120.8	0.0%	15.7%	15.7%	0.0%
Program 46	6	68	Mission Computer (1)	Dollars	477.9	478.0	0.0%	4.3%	4.3%	0.0%
Program 47	9	36	PME–Air Vehicle	Dollars	1039.4	1039.4	0.0%	2.5%	2.5%	0.0%
Program 47	9	36	PME–Air Vehicle	Hours	8278.7	8278.6	0.0%	15.5%	15.5%	0.0%
Program 47	9	36	Data Link (1)	Dollars	170.2	170.2	0.0%	17.7%	17.7%	0.0%
Program 48	5	179	PME–Air Vehicle	Dollars	1858.3	391.3	78.9%	3.1%	0.6%	79.4%
Program 49	6	180	PME–Air Vehicle	Dollars	435.3	99.8	77.1%	4.4%	1.0%	76.5%
Program 50	5	488	PME–Air Vehicle	Dollars	349.3	350.7	−0.4%	3.3%	3.4%	−0.8%
Program 51	6	663	PME–Air Vehicle	Dollars	5.6	3.6	36.6%	0.6%	0.4%	24.8%
Program 52	5	380	PME–Air Vehicle	Dollars	456.9	454.6	0.5%	9.0%	8.9%	0.3%
Program 53	6	749	PME–Air Vehicle	Dollars	37.2	36.6	1.7%	0.5%	0.5%	4.3%
Program 54	8	194	PME–Air Vehicle	Dollars	28.8	28.8	−0.1%	0.6%	0.6%	−0.1%
Program 55	9	677	PME–Air Vehicle	Dollars	74.8	74.8	0.0%	1.6%	1.6%	0.0%
Program 56	5	590	PME–Air Vehicle	Dollars	6.6	6.6	0.5%	0.2%	0.2%	6.3%
Program 57	5	579	PME–Air Vehicle	Dollars	22.8	22.8	−0.1%	0.8%	0.8%	0.0%

and

Table A2. Error Comparison using Unit Theory.

Program	Number of Lots	Number of Units	Component Estimated	Units	Traditional RMSE	Boone RMSE	RMSE Percentage Difference	Traditional MAPE	Boone MAPE	MAPE Percentage Difference
Program 1	7	503	Airframe	Hours	4.6	3.5	23.4%	7.1%	5.0%	28.7%
Program 1	6	483	PME–Air Vehicle	Hours	5.4	1.5	72.5%	11.3%	2.9%	74.0%
Program 1	7	503	PME–Air Vehicle	Dollars	2260.6	517.0	77.1%	12.9%	3.2%	75.2%
Program 1	7	503	Airframe	Dollars	2383.2	857.9	64.0%	14.6%	4.9%	66.4%
Program 2	5	638	PME–Air Vehicle	Dollars	315.4	195.3	38.1%	5.8%	4.3%	26.3%
Program 3	5	500	PME–Air Vehicle	Dollars	2984.5	1120.2	62.5%	49.4%	17.6%	64.4%
Program 4	7	357	Airframe	Dollars	2662.2	2664.3	−0.1%	13.1%	13.2%	−0.1%
Program 4	9	424	PME–Air Vehicle	Dollars	9323.3	4999.8	46.4%	37.9%	14.1%	62.8%
Program 5	19	205	Airframe	Dollars	2446.1	2445.8	0.0%	12.6%	12.6%	−0.3%
Program 5	19	205	PME–Air Vehicle	Dollars	3228.6	3228.9	0.0%	12.4%	12.4%	0.0%
Program 6	7	459	Electronic Warfare (1)	Dollars	20.9	20.9	0.0%	30.8%	30.8%	0.0%
Program 6	7	459	PME–Air Vehicle	Dollars	1439.9	738.1	48.7%	11.3%	5.9%	47.2%
Program 7	5	321	PME–Air Vehicle	Dollars	37.9	33.3	12.2%	3.8%	3.8%	1.1%
Program 8	6	98	Electronic Warfare (3)	Dollars	5.2	4.9	6.1%	4.8%	4.8%	1.4%
Program 8	6	98	Electronic Warfare (2)	Dollars	84.2	70.3	16.5%	11.1%	10.6%	4.7%
Program 8	6	98	PME–Air Vehicle	Dollars	375.2	339.5	9.5%	4.2%	3.7%	13.4%
Program 8	6	98	Electronic Warfare (1)	Dollars	27.5	18.7	31.9%	10.2%	5.9%	42.5%
Program 9	7	110	Electronic Warfare (5)	Dollars	102.9	99.2	3.5%	9.7%	10.4%	−6.6%
Program 9	7	110	Electronic Warfare (3)	Dollars	6.4	6.4	0.0%	4.7%	4.7%	0.0%
Program 9	7	110	Electronic Warfare (4)	Dollars	653.6	653.6	0.0%	6.2%	6.2%	0.0%
Program 9	7	110	Electronic Warfare (2)	Dollars	709.4	709.4	0.0%	6.1%	6.1%	0.0%
Program 9	7	110	PME–Air Vehicle	Dollars	668.5	668.5	0.0%	5.1%	5.1%	0.0%
Program 9	7	110	Electronic Warfare (1)	Dollars	31.6	29.1	8.0%	8.7%	8.0%	8.3%
Program 10	9	1586	PME–Air Vehicle	Dollars	115.5	115.6	−0.2%	12.5%	12.5%	−0.2%
Program 10	10	1796	PME–Air Vehicle	Hours	150.8	150.9	0.0%	12.5%	12.5%	−0.1%
Program 11	8	3529	PME–Air Vehicle	Hours	0.9	0.7	21.2%	27.5%	44.9%	−63.4%
Program 11	8	3529	PME–Air Vehicle	Dollars	97.1	97.5	−0.4%	10.1%	10.4%	−2.1%
Program 12	16	7891	PME–Air Vehicle	Hours	520.1	525.6	−1.1%	86.2%	86.2%	0.0%
Program 12	21	10035	PME–Air Vehicle	Dollars	243.8	239.2	1.9%	30.1%	28.8%	4.2%
Program 13	6	3385	EO	Dollars	12.1	9.4	22.5%	10.7%	9.6%	10.0%
Program 13	10	3803	PME–Air Vehicle	Dollars	33.6	24.8	26.1%	10.3%	7.5%	27.1%
Program 13	10	3803	PME–Air Vehicle	Hours	130.1	100.5	22.7%	21.5%	17.1%	20.7%
Program 14	6	180	PME–Air Vehicle	Dollars	2249.4	1008.9	55.2%	6.4%	2.3%	64.2%
Program 15	10	20	PME–Air Vehicle	Hours	3430.3	3430.4	0.0%	41.5%	41.5%	0.0%
Program 15	10	20	PME–Air Vehicle	Dollars	3013.9	3013.9	0.0%	17.4%	17.4%	0.0%
Program 15	7	11	Mission Computer (1)	Dollars	213.9	213.9	0.0%	11.6%	11.5%	0.6%
Program 16	5	100	Airframe	Dollars	10,807.3	7455.4	31.0%	7.0%	4.1%	41.8%
Program 16	5	100	PME–Air Vehicle	Dollars	137,225.9	81,884.9	40.3%	51.7%	26.9%	48.0%
Program 17	5	275	PME–Air Vehicle	Dollars	8837.5	1396.3	84.2%	17.6%	3.3%	81.6%
Program 18	12	83	PME–Air Vehicle	Hours	266,012.8	266,015.3	0.0%	39.3%	39.3%	0.0%
Program 18	11	83	Airframe	Dollars	89,956.0	89,961.1	0.0%	39.1%	39.1%	0.0%
Program 18	10	68	Mission Computer (1)	Dollars	4143.0	4143.2	0.0%	68.2%	68.2%	0.0%
Program 18	11	83	PME–Air Vehicle	Dollars	82,138.6	82,143.3	0.0%	23.2%	23.2%	0.0%
Program 19	5	45	Airframe	Dollars	501.2	501.2	0.0%	53.9%	53.9%	0.0%
Program 19	5	45	PME–Air Vehicle	Dollars	649.0	649.0	0.0%	17.6%	17.6%	0.0%
Program 19	5	45	Mission Computer (1)	Dollars	61.7	59.7	3.2%	9.8%	9.7%	1.2%
Program 20	9	76	PME–Air Vehicle	Dollars	1108.7	522.5	52.9%	7.2%	3.6%	49.9%
Program 21	5	50	PME–Air Vehicle	Dollars	24,625.3	6362.0	74.2%	7.4%	2.3%	69.5%
Program 22	9	31	PME–Air Vehicle	Dollars	16,636.3	16,636.4	0.0%	6.6%	6.6%	0.0%
Program 23	5	14	PME–Air Vehicle	Dollars	14,475.8	14,476.0	0.0%	8.7%	8.7%	0.0%
Program 24	6	98	PME–Air Vehicle	Dollars	2259.9	2260.1	0.0%	3.3%	3.3%	0.0%
Program 25	7	59	Electronic Warfare (5)	Dollars	2808.4	2805.2	0.1%	14.8%	15.4%	−4.0%
Program 25	11	84	PME–Air Vehicle	Dollars	5083.2	4228.8	16.8%	8.7%	9.2%	−5.2%
Program 25	11	84	Electronic Warfare (2)	Dollars	248.9	248.6	0.1%	13.9%	14.3%	−2.9%
Program 25	7	59	Electronic Warfare (1)	Dollars	1259.1	653.3	48.1%	16.1%	7.1%	55.6%
Program 26	7	344	Airframe	Dollars	11,474.7	8294.9	27.7%	22.7%	21.5%	5.3%
Program 26	7	344	Avionics	Dollars	2218.8	2102.8	5.2%	29.5%	26.9%	8.8%
Program 26	7	344	PME–Air Vehicle	Dollars	12,898.4	8742.1	32.2%	20.7%	16.9%	18.4%
Program 27	14	453	PME–Air Vehicle	Hours	54,142.9	53,766.4	0.7%	59.9%	63.1%	−5.4%
Program 27	14	453	Airframe	Hours	70,415.0	69,426.8	1.4%	58.8%	59.1%	−0.5%
Program 28	8	538	PME–Air Vehicle	Hours	3828.8	3829.8	0.0%	9.8%	9.9%	0.0%
Program 28	8	538	Airframe	Hours	3865.3	3866.2	0.0%	7.6%	7.6%	0.0%
Program 29	8	529	Hydraulic	Dollars	156.9	156.4	0.3%	22.3%	22.9%	−2.8%
Program 29	12	477	Airframe	Dollars	6490.2	5974.2	7.9%	14.2%	14.4%	−1.8%
Program 29	12	477	Wing	Dollars	712.3	712.7	−0.1%	27.8%	27.8%	−0.1%
Program 29	11	433	Electronic Warfare (1)	Dollars	57.5	57.5	0.0%	13.5%	13.5%	−0.1%
Program 29	8	309	Electrical	Dollars	230.6	230.7	−0.1%	8.2%	8.2%	0.0%
Program 29	12	1045	Body	Dollars	1922.2	1826.7	5.0%	26.0%	25.9%	0.7%
Program 29	5	177	Empennage	Dollars	32.3	22.0	31.8%	6.1%	4.6%	24.5%
Program 29	12	477	PME–Air Vehicle	Dollars	8218.5	5525.3	32.8%	15.0%	10.2%	32.0%
Program 29	8	309	Alighting Gear	Dollars	205.7	42.2	79.5%	11.6%	2.0%	83.1%
Program 30	5	469	PME–Air Vehicle	Dollars	1283.8	891.8	30.5%	13.5%	8.3%	38.3%
Program 31	10	59	PME–Air Vehicle	Dollars	11,978.9	11,979.3	0.0%	8.6%	8.6%	0.0%
Program 32	9	348	PME–Air Vehicle	Dollars	430.6	430.8	0.0%	3.5%	3.5%	−0.1%
Program 33	5	109	PME–Air Vehicle	Hours	993.9	994.0	0.0%	9.5%	9.5%	0.0%
Program 33	5	109	PME–Air Vehicle	Dollars	6824.7	6824.8	0.0%	28.2%	28.2%	0.0%
Program 34	18	631	PME–Air Vehicle	Dollars	6926.7	2799.9	59.6%	17.0%	6.6%	61.0%
Program 35	6	425	PME–Air Vehicle	Dollars	1135.8	1137.5	−0.2%	3.5%	3.5%	−0.2%
Program 35	7	522	PME–Air Vehicle	Hours	4615.3	4458.5	3.4%	6.3%	6.1%	3.1%
Program 35	7	522	Airframe	Hours	6757.0	6280.7	7.0%	5.7%	5.4%	4.8%
Program 36	9	358	PME–Air Vehicle	Hours	5118.7	5120.1	0.0%	6.8%	6.8%	0.0%
Program 36	9	358	Airframe	Hours	12,155.2	11,257.1	7.4%	15.5%	14.3%	7.6%
Program 37	5	204	PME–Air Vehicle	Dollars	1468.7	921.0	37.3%	2.9%	1.9%	36.4%
Program 38	5	605	PME–Air Vehicle	Dollars	2641.9	1527.7	42.2%	14.9%	8.1%	46.0%
Program 39	5	870	PME–Air Vehicle	Dollars	310.9	311.5	−0.2%	2.3%	2.3%	−0.2%
Program 40	10	178	Electronic Warfare (3)	Dollars	751.2	551.9	26.5%	69.7%	74.7%	−7.1%
Program 40	10	712	Body	Dollars	617.6	577.6	6.5%	4.8%	5.1%	−7.6%
Program 40	10	178	Airframe	Dollars	4251.9	4226.4	0.6%	4.8%	4.9%	−1.0%
Program 40	10	178	Electronic Warfare (2)	Dollars	721.7	721.7	0.0%	393.4%	393.4%	0.0%
Program 40	10	178	Electronic Warfare (1)	Dollars	1642.3	1643.0	0.0%	20.7%	20.7%	0.0%
Program 40	10	178	PME–Air Vehicle	Hours	13,454.5	13,466.8	−0.1%	6.0%	6.0%	−0.1%
Program 40	10	178	Auxiliary Power Plant	Dollars	385.1	385.1	0.0%	24.9%	24.9%	0.0%
Program 40	10	178	PME–Air Vehicle	Dollars	12,231.7	12,236.6	0.0%	7.9%	7.9%	0.0%
Program 40	10	178	Alighting Gear	Dollars	233.6	233.6	0.0%	30.1%	30.1%	0.0%
Program 40	10	178	Wing	Dollars	607.4	607.6	0.0%	6.2%	6.2%	0.0%
Program 40	10	178	Empennage	Dollars	702.1	702.1	0.0%	17.4%	17.4%	0.0%
Program 40	10	178	Hydraulic	Dollars	72.2	70.2	2.8%	9.0%	8.8%	2.2%
Program 41	6	67	PME–Air Vehicle	Hours	12,741.5	12,743.8	0.0%	9.5%	9.5%	0.0%
Program 41	5	49	Empennage	Dollars	242.2	242.2	0.0%	5.8%	5.9%	0.0%
Program 41	6	67	PME–Air Vehicle	Dollars	16,643.9	16,645.6	0.0%	10.7%	10.7%	0.0%
Program 41	6	67	Surface Controls	Dollars	281.7	281.7	0.0%	7.7%	7.7%	0.0%
Program 41	6	67	EOTS	Dollars	442.3	442.4	0.0%	9.5%	9.5%	0.0%
Program 41	6	67	Wing	Dollars	1927.0	1927.3	0.0%	7.4%	7.4%	0.0%
Program 41	6	67	Electrical	Dollars	57.2	57.2	0.0%	2.1%	2.1%	0.0%
Program 41	6	67	Electronic Warfare (1)	Dollars	547.3	547.3	0.0%	8.1%	8.1%	0.0%
Program 41	6	67	Hydraulic	Dollars	281.5	274.6	2.4%	19.4%	19.0%	2.0%
Program 41	6	67	Mission Computer (1)	Dollars	1698.1	1542.4	9.2%	4.6%	3.7%	19.5%
Program 41	6	67	Airframe	Dollars	6877.8	5547.4	19.3%	8.7%	6.4%	26.8%
Program 41	6	67	Alighting Gear	Dollars	582.3	521.1	10.5%	28.3%	25.0%	11.6%
Program 41	6	67	EO/IR	Dollars	233.0	89.4	61.6%	9.3%	3.1%	66.8%
Program 41	6	201	Body	Dollars	3431.8	2343.2	31.7%	42.6%	29.9%	29.7%
Program 42	5	41	PME–Air Vehicle	Dollars	8498.6	8499.6	0.0%	6.2%	6.2%	0.0%
Program 42	5	41	PME–Air Vehicle	Hours	15,696.5	15,696.9	0.0%	10.7%	10.7%	0.0%
Program 42	5	50	EOTS	Dollars	593.3	593.3	0.0%	11.6%	11.6%	0.0%
Program 42	5	50	EO/IR	Dollars	578.4	578.4	0.0%	7.5%	7.5%	0.0%
Program 42	5	50	Hydraulic	Dollars	297.0	297.0	0.0%	15.4%	15.4%	0.0%
Program 42	5	50	Surface Controls	Dollars	424.9	424.9	0.0%	11.0%	11.0%	0.0%
Program 42	5	50	Radar	Dollars	733.8	733.8	0.0%	10.9%	10.9%	0.0%
Program 42	5	50	Airframe	Dollars	5222.7	5222.8	0.0%	5.9%	5.9%	0.0%
Program 42	5	50	Electronic Warfare (1)	Dollars	746.5	746.5	0.0%	10.7%	10.7%	0.0%
Program 42	5	50	Wing	Dollars	3726.6	3726.7	0.0%	16.5%	16.5%	0.0%
Program 42	5	50	Alighting Gear	Dollars	78.6	77.4	1.5%	3.6%	3.5%	2.3%
Program 42	5	150	Body	Dollars	1588.5	892.1	43.8%	12.6%	8.7%	30.8%
Program 43	13	1285	PME–Air Vehicle	Dollars	88.1	88.8	−0.8%	1.9%	1.9%	−1.0%
Program 44	6	432	PME–Air Vehicle	Dollars	1621.0	1623.3	−0.1%	10.0%	10.0%	−0.2%
Program 45	9	63	PME–Air Vehicle	Dollars	2152.3	1557.1	27.7%	9.5%	6.4%	33.2%
Program 46	6	44	PME–Air Vehicle	Hours	7736.9	7255.3	6.2%	17.6%	16.7%	4.8%
Program 46	10	113	PME–Air Vehicle	Dollars	797.9	627.0	21.4%	3.8%	2.9%	22.7%
Program 47	19	1023	PME–Air Vehicle	Dollars	115.2	115.2	0.0%	4.3%	4.2%	0.2%
Program 48	5	1725	PME–Air Vehicle	Dollars	59.8	3.1	94.9%	6.8%	0.3%	95.4%
Program 49	10	16	Data Link (1)	Dollars	470.3	470.3	0.0%	20.4%	20.4%	0.0%
Program 49	10	16	PME–Air Vehicle	Dollars	41,008.9	41,009.2	0.0%	14.1%	14.1%	0.0%
Program 50	7	577	PME–Air Vehicle	Dollars	1674.7	1224.7	26.9%	5.5%	4.6%	15.7%
Program 51	12	244	PME–Air Vehicle	Hours	625.6	612.8	2.0%	191.4%	191.8%	−0.2%
Program 52	11	899	Electronic Warfare (1)	Dollars	90.1	90.2	−0.1%	29.2%	29.3%	−0.1%
Program 52	11	203	PME–Air Vehicle	Dollars	2995.1	2992.0	0.1%	24.9%	23.6%	5.2%
Program 53	13	251	PME–Air Vehicle	Hours	4585.2	4585.2	0.0%	6.7%	6.7%	0.0%
Program 53	11	203	PME–Air Vehicle	Dollars	2459.9	2460.0	0.0%	9.6%	9.6%	0.0%
Program 54	11	184	PME–Air Vehicle	Hours	7010.4	7010.7	0.0%	18.0%	18.0%	0.0%
Program 54	9	134	PME–Air Vehicle	Dollars	1907.3	970.0	49.1%	11.8%	6.5%	44.9%
Program 55	5	136	PME–Air Vehicle	Dollars	321.6	277.7	13.7%	5.5%	4.7%	14.8%
Program 56	9	155	PME–Air Vehicle	Dollars	1356.5	1356.6	0.0%	3.9%	3.9%	0.0%
Program 57	6	68	EO/IR	Dollars	326.0	326.0	0.0%	1261.8%	1261.8%	0.0%
Program 57	6	68	PME–Air Vehicle	Dollars	8574.7	8470.9	1.2%	4.3%	4.3%	−0.5%
Program 57	6	68	Electronic Warfare (1)	Dollars	998.8	998.9	0.0%	58.9%	58.9%	0.0%
Program 57	6	68	Electronic Warfare (2)	Dollars	750.2	750.2	0.0%	31.3%	31.3%	0.0%
Program 57	6	68	Data Link (1)	Dollars	94.8	94.8	0.0%	7.2%	7.2%	0.0%
Program 57	6	68	Electronic Warfare (4)	Dollars	1156.3	1156.3	0.0%	12.2%	12.2%	0.0%
Program 57	6	68	Mission Computer (1)	Dollars	1030.6	1030.6	0.0%	13.0%	13.0%	0.0%
Program 57	6	68	PME–Air Vehicle	Hours	6435.9	6435.0	0.0%	12.3%	12.3%	0.3%
Program 57	6	68	Airframe	Dollars	1443.2	1285.1	11.0%	6.7%	5.4%	18.5%
Program 57	6	68	Electronic Warfare (3)	Dollars	53.4	21.8	59.1%	7.2%	3.0%	58.5%
Program 58	9	36	PME–Air Vehicle	Hours	60,347.2	60,347.3	0.0%	78.2%	78.2%	0.0%
Program 58	9	36	Data Link (1)	Dollars	227.8	227.8	0.0%	29.3%	29.3%	0.0%
Program 58	9	36	PME–Air Vehicle	Dollars	4570.2	4570.2	0.0%	10.9%	10.9%	0.0%
Program 58	5	18	EO/IR	Dollars	3488.4	3469.8	0.5%	28.8%	28.7%	0.3%
Program 59	5	179	PME–Air Vehicle	Dollars	4583.3	1334.5	70.9%	8.1%	2.8%	65.4%
Program 60	6	180	PME–Air Vehicle	Dollars	1010.5	333.9	67.0%	12.4%	4.6%	63.1%
Program 61	5	488	PME–Air Vehicle	Dollars	502.3	486.5	3.1%	9.2%	7.7%	16.3%
Program 62	6	78	PME–Air Vehicle	Hours	6027.1	5952.3	1.2%	33.8%	34.3%	−1.6%
Program 62	6	97	Airframe	Hours	2648.5	2649.0	0.0%	20.5%	20.5%	0.0%
Program 62	9	110	PME–Air Vehicle	Dollars	13,027.5	13,028.9	0.0%	24.0%	24.0%	0.0%
Program 63	6	663	PME–Air Vehicle	Dollars	23.2	21.1	9.2%	2.9%	2.6%	11.6%
Program 64	5	380	PME–Air Vehicle	Dollars	1520.9	1521.2	0.0%	57.4%	57.4%	0.0%
Program 65	6	749	PME–Air Vehicle	Dollars	116.6	115.9	0.6%	1.7%	1.8%	−5.1%
Program 66	8	194	PME–Air Vehicle	Dollars	128.3	119.3	7.0%	2.6%	2.4%	8.6%
Program 67	9	677	PME–Air Vehicle	Dollars	273.5	273.5	0.0%	5.1%	5.1%	0.0%
Program 68	5	590	PME–Air Vehicle	Dollars	87.1	87.2	0.0%	2.8%	2.8%	0.0%
Program 69	5	579	PME–Air Vehicle	Dollars	305.7	305.8	0.0%	9.5%	9.5%	0.0%

. A total of 118 end-items in units of total dollars and 22 components in units of labor hours were analyzed. Each entry lists the program number, number of production lots, number of items produced, type of end-item, and units of the production costs. Additionally, each entry lists both error measures and the respective percent difference between the models. Positive (negative) differences indicate Boone’s model has less (more) error than Wright’s.

Boone’s curve performs better for two reasons. First, Boone’s model can explain costs to at least the same degree of accuracy as the traditional learning curve theories due to the extra parameter. Second, increased accuracy could also be explained by Boone’s functional form. Despite these theoretical explanations, Boone’s model had more error than Wright’s for some observations; these negative percentage differences occur because an upper bound was placed on Boone’s decay value. An upper bound of 5000 was used for the decay value (same as Boone’s original paper). The practical effect of this particular bound can be observed by the number of end-items where the traditional models significantly outperformed Boone’s (i.e., a MAPE difference larger than 0.5%): 7 out of 140 for cumulative average theory and 15 out 169 for unit theory. Thus, the majority of the results were not affected by this artificial limitation which was chosen by trial and error. In practice, the bound could be set arbitrarily large so that it is not binding. Boone’s learning curve. This upper bound was necessary s since the GRG algorithm requires bounds on the estimated parameters.

Some percentage error differences are approximately (but not exactly) zero. Observations with percentage error differences of approximately zero were defined as those within the bounds $(-\mathrm{0.25\%},\mathrm{0.25\%})$. These bounds were used by the researchers to distinguish between observations with approximately zero and non-zero percentage error differences in order to inform the descriptive statistics.

Boone’s model had less error for 41% of observations, was approximately equal to Wright’s for 50% of observations, and had more error for 9% of observations. While Boone’s model is an improvement on Wright’s for some observations, many times the models fit the data equally well (i.e., an approximate zero difference).

The results of the paired difference t-tests for cumulative average theory are shown in

Table 2. Cumulative Average Theory Descriptive and Inferential Statistics.

Hypothesis Test: H0: µ ≤ 0    HA: µ > 0
Learning Curve Theory	Error Measure	Units of Measure	Sample Mean $(\overline{\mathit{x}})$	Sample Standard Deviation (s)	Number of Observations	Test Statistic	p-Value	Result
Cumulative Average Theory	Root Mean Squared Error Percentage Difference	Total Dollars(K)	19.3%	28.90%	118	7.23	<0.001	Reject H0
		Labor Hours	15.20%	31.20%	22	18.5	0.28	Fail to reject H0
	Mean Absolute Percent	Total Dollars(K)&Labor Hours Combined	18.60%	29.50%	140	7.45	<0.001	Reject H0

and a sample graph is shown in Figure 4. No outliers, as defined by a value which fell more than three interquartile ranges from the upper 90% and lower 10% quantiles, were present in any of the tests.

The results of these hypothesis tests were mixed. For the RMSE percentage difference (measured in total dollars) and MAPE percentage difference, the paired difference t-tests led to rejection of the null hypothesis—indicating the increase in accuracy is statistically significant. Conversely, RMSE percentage difference (measured in hours) failed to reject the null hypothesis. Due to the small sample size, large sample theory could not be used, and the data failed a Shapiro–Wilk test (p-value = 0.721). Therefore, a Wilcoxon rank signed test was used. This indicates that Boone’s improvement in accuracy over Wright’s is not statistically significant when costs are measured in labor hours. However, small sample sizes can cause paired difference tests to have low power that may cause hypothesis tests to incorrectly fail to reject the null hypothesis [30].

Now considering unit theory, the results from Crawford’s and Boone’s learning curve models are presented in Appendix B. A total of 141 end-items (measured in total dollars) and 28 end-items (measured in labor hours) were analyzed.

Similar to cumulative average theory, observations with percent error differences of approximately zero were defined as those within the bounds $(-\mathrm{0.25\%},\mathrm{0.25\%})$. Boone’s model had less error for 43% of observations across all percent difference error measures in comparison to crawford’s learning curve. Boone’s learning curve error was approximately equal for 52% of observations, and had more error for 5% of observations.

The results of the paired difference testing for unit theory are provided in

Table 3. Unit Theory Descriptive and Inferential Statistics.

Hypothesis Test: H0: µ ≤ 0    HA: µ > 0
Learning Curve Theory	Error Measure	Units of Measure	Sample Mean $(\overline{\mathit{x}})$	Sample Standard Deviation (s)	Number of Observations	Test Statistic	p-Value	Result
Unit Theory	Root Mean Squared Error Percentage Difference	Total Dollars(K)	13.80%	22.70%	141	7.23	<0.001	Reject H0
		Labor Hours	6.00%	14.80%	28	74.00	0.046	Reject H0
	Mean Absolute Percent Error Percentage Difference	Total Dollars(K)&Labor Hours Combined	11.30%	23.10%	169	6.36	<0.001	Reject H0

and a sample graph is shown in Figure 5. Again, no outliers were present in any of the paired difference t-tests.

The results of these paired difference tests indicate the improvement with Boone’s model is statistically significant. Again, the RMSE percent difference (for labor hours) used a Wilcoxon rank sum test (due to the failure of the Shapiro–Wilk test with a p-value less than 0.001).

5. Conclusions

A large, diverse dataset of DoD production programs was used to test if Boone’s learning curve more accurately explained error in comparison to traditional learning curve theories. The direct recurring cost data from bomber, cargo, and fighter aircraft along with missiles and munitions programs in units of total dollars and labor hours were analyzed using Cumulative Average and Unit Learning Curve theories. Various components of these programs were analyzed from wings and data link systems to the airframes and air vehicles. Boone’s learning curve was tested against both cumulative average and unit learning curve theories using two different measures of model error that resulted in six paired difference tests. This methodology resulted in 998 total observations across all measures and ensured the generalizability of Boone’s learning curve was tested.

Boone’s learning curve improved upon the traditional learning curve estimates for approximately 42% of the sampled program components while approximately equaling the traditional learning curve error for approximately 51% of program components. Boone’s learning curve resulted in a range of mean percentage difference reductions of 6% to 18.6% across all measures. The standard deviations of these improvements were high with coefficients of variation ranging from 150% to 247% across all measures. Absent additional analysis, these high amounts of variability make it challenging to conclude the degree to which Boone’s learning curve will improve the accuracy of explaining program component costs in comparison to the traditional estimation methods. Specifically, more research is needed to understand the shape of the learning curve and how it behaves related to production circumstances. It remains unclear which programs are more accurately modelled using Boone’s learning curve and to what degree Boone’s learning curve will more accurately model program component costs.

The paired difference tests between Boone’s learning curve and the traditional theories indicate that Boone’s learning curve reduces error to a significant degree across a wide range of measures. Five of the six paired difference tests resulted in rejecting the null hypothesis that Boone’s learning curve had an equal amount or more error than the traditional theories at a significance level of 0.05.

Due to data availability, program lot data was used instead of unitary data. Although Boone’s learning curve should perform just as well using either type of data, this research cannot conclusively state that Boone’s learning curve will more accurately explain programs in unitary data. Also, the majority of data utilized were end-item components in units of total dollars. The total dollar cost includes all cost categories rather than solely labor costs. These data are not ideal when applying learning curve theory and may bias learning curves to display diminishing rates of learning. Despite these potential issues, total dollar cost data are regularly utilized by cost estimators in the field due to data availability. Therefore, the practical applications of this analysis remain valid despite the limitations of using imperfect total dollar cost data in learning curve analysis.

Boone’s learning curve was tested on programs whose lot costs were already known and whose parameters can be directly estimated. In other words, Boone’s learning curve was tested against the traditional theories on how well it explained rather than predicted program costs. In order to utilize Boone’s learning curve to predict costs, a decay value would be selected a priori. Similar to the learning curve slope, an analyst could use the decay value from similar programs to provide a range values to make predictions. Additionally, future research should investigate if Boone’s Decay Value can be predicted using various attributes of a program. Tests could be performed on how well Boone’s learning curve predicts costs for a program using analogous programs in comparison to the traditional theories. Lastly, additional labor hour data should be collected and analyzed in order to dispel the potential bias of learning curves displaying diminishing rates of learning when analyzed in units of total dollars.