An Integrated Key Performance Measurement for Manufacturing Operations Management

: This paper proposes a comprehensive production performance measurement framework and illustrates the method to evaluate the performance and guide practitioners to make further improvement. The development comprises four steps. (1) Performance indicators derived from business excellence models are enumerated to provide the performance model: 74 indicators, which can be classiﬁed in terms of their characteristics, are identiﬁed in six criteria. (2) A multiple criteria decision-making approach based on the analytic hierarchical and network processes, which determine the weights of the criteria and indicators, is applied. In addition, this study introduced additional formulas to derive the ﬁnal performance values. (3) A performance measurement framework that integrates the measurement and result analysis processes is implemented. (4) The proposed framework is veriﬁed through a case study. The results of the case study show that the proposed framework identiﬁes the gaps and discrepancies among the management levels, enabling the determination of means for continuous improvement.


Background
The ability to measure operational performance is an important factor for competing enterprises in the global market. Performance measurement helps in the evaluation of the long-term effects of outputs for improving competitiveness and decision-making power. A company's competitiveness and profits are reduced by a consistent continuation of subpar performance, as this eventually leads to a failure to meet customer need. In this overall perspective, using performance measurement to understand the company's circumstances is critical to gaining the success and sustaining competitiveness in the era of constantly changing markets [1].
To measure this performance, well-extracted performance indicators are important sources for the effective performance measurement. It is generally believed that inspecting all the processes of the company environment yields well-extracted indicators that can increase the chances for success [2]. Evaluation of well-extracted indicators provides advantages to a company in the form of defects corrections. Therefore, gathering indicators should be obtained regarding financial or non-financial measures. Before the 1990s, financial measurements such as firm revenue, market share, and return on

Research Statement
The general objective of this study is the development of a framework for performance measurement using an MCDM method (hybrid AHP/ANP approach) shown in Figure 1. This hybrid approach determines the weights of the criteria and performance indicators according to the relationship between each element. To organize the list of performance indicators, we investigate business excellence models and categorize them into six criteria. Furthermore, we develop a performance measurement framework that includes an assessment stage and a continuous improvement stage. The four contributions of the study are as follows: • We list non-financial performance indicators based on the three business excellence models: Baldridge, EFQM and Deming. One of the core contributions of this work is the categorization of production related indicators into six criteria.

•
We use of an MCDM approach to define the weights that state the importance of each criterion and indicator according to pairwise comparisons. We develop an additional formulation to derive the performance value based on the results of the MCDM method.
• We specify the process of inducing continuous improvement, which includes a method to analyze the assessment results showing the performance measurement gaps among the three management levels, i.e., strategic, tactical and operational level. • We present the results of a case study that show the developed framework; additionally, we use the case study output to verify the proposed framework process.

Designing the Structure of the Indicators
Several studies have suggested many criteria for use as the indicators, e.g., cost, delivery dependence, flexibility, quality, utilization, time and efficiency. Table 1 illustrates the selection of criteria and methods for measuring their production performance. There are some limitations of the related model in that it just introduces the list of criteria sorts the preference criteria, without deriving differences in criteria preference among employees Conventional performance measurement studies are based on criteria strongly related with production performance, and they characterize information solely on production management aspects. However, production performance is complex in that worker productivity, production line environment status and efforts to improve strongly influence the results of performance. The main differentiators for this study are 'Overall Waste', 'Human Resource' and 'Continuous Improvement'. These criteria enable the measurement to cover current production management aspects such as lean manufacturing and Toyota Production System. The criteria comprise six components that cover manufacturing performance, and Table 2 explains the criteria characteristics and presents the number of indicators used in this study (Appendix A provides additional details). Table 2. Characteristics of accepted criteria and included indicators.

Criteria Characteristics Indicators
Production This covers all of the manufacturing processes that take place on the shop floor. It considers the production system's effectiveness and measures the manufacturing process efficiency.

9
Quality This is a measure of the reliability of a company's control system, and it shows the trust level from the raw material to the finished product.

18
Flexibility Companies try to match the variable nature of customer's needs such that the product mix and production volumes need to be varied. For this reason, this criterion measures the manufacturing system's flexibility performance.

5
Overall Waste This is excerpted from the Toyota Production System, which was mainly presented by Ohno Taiichi. It measures the expenses of the laboring force, time and other resources due to the inefficiency of the process or glut of inventory.

10
Human Resource This is an imperative resource on the shop floor, even though automation facilities decrease its importance. It comprises individual performance measurement indicators as well as organizational performance measurement indicators.

19
Continuous Improvement This is an ongoing effort to improve existing manufacturing processes and resources. Companies should encourage continuous improvement to promote manufacturing efficiency. It measures the organization wide approach to improvement activities. 13

Fundamentals of the AHP/ANP Method
AHP decomposes the unstructured problem into a hierarchical structure, such as a decision tree. Decision elements at each hierarchical level are compared through the pair-wise method and assigned relative scales. The detailed process of AHP comprises three steps [29]: (1) The problem is decomposed into a hierarchical structure. The top level can be decomposed into some criteria and its lower level elements which cannot decompose furthermore. (2) Pair-wise comparison is performed. The pair-wise comparison indicates the perception of which element has more weight than the other. (3) The maximum eigenvalues and eigenvectors for the relative weights of the elements are derived. Then, the consistency property of the matrix must be checked to verify reliability of the decision making.
The Consistency Index (CI) and Consistency Ratio (CR), which illustrate the reliability, are defined as follows: where λ max is the maximum eigenvalue, n is the number of elements and the random index (RI) is the average consistency index of randomly generated pair-wise comparison indicators (Table 3). Using RI, the CR can be calculated, and this value shows whether the evaluations are sufficiently consistent. In general, if CR > 0.1, the decision-making structure is reliably consistent. If CR < 0.1, then it is inconsistent, and the evaluation procedure must be repeated until the CR satisfies the threshold.
The ANP method is an extension of the AHP that generalizes the AHP by replacing the hierarchy with a network that is composed of different clusters (groups of elements). The ANP has unique characteristics in that it assumes a mutual interaction relationship that exist between the clusters and contains feedback loops among the sub-elements [23]. Between the clusters and their elements different relationships (inner or outer dependence, feedback) exist as shown in Figure 2. The ANP still involves the hierarchical representation of relationships, but it does not require as strict a hierarchical structure as the AHP [30]. The ANP approach is capable of handling interdependence among elements by obtaining the composite weights through the development of a 'super matrix' which is similar to Markov chains [31].

Applying the Indicators with the AHP and ANP Method
The AHP has been criticized because the decision problems are structured in a hierarchical manner. Some decision-making problems cannot be structured hierarchically because they involve the interaction and dependence of higher-level elements on lower-level elements [29]. The ANP also has some limits, e.g., it cannot perform comparisons between elements at the same hierarchical level, but it can be used for network structured decision-making problem [32]. For this reason, a hybridization of AHP and ANP was developed to overcome the complementary shortcomings. [30] used the AHP to evaluate various factors for different product mixes and then used the ANP to analyze more complex interrelationships among the decision levels and attributes. [33] use the AHP and ANP to determine the weight of each criterion when generating the performance model for a decision-making problem regarding the location of a facility.
In this study, the AHP and ANP are utilized to determine the weights of the criteria and indicators and the structure of the performance measurement is a categorization of 74 indicators into six criteria. A few of the 74 indicators have mutual interactions because some circumstances cannot be reflected by a single indicator. In addition, assuming relationships between the criteria and indicators with the hierarchical structure is problematic in that all indicators do not have a mutual relationship and their relationships vary depending on varying circumstances. For this reason, when forming the structure of the performance measurement, we should simultaneously include their mutual interactions and independent relationships. Figure 3 presents the structure of the performance measurement. For simplification, we assume that the indicators have a hierarchical structure in accordance with the criteria. For this reason, the AHP should be utilized to assess indicators. The absence of the mutual interactions of the indicators are corrected by applying the ANP.

A Multi-Stage Framework
The conceptual framework involves five stages and has three main sections: performance measuring using MCDM, the analysis of gaps among different levels, and the alignment process with corporate and manufacturing strategy. Figure 4 shows the overall process of the framework and the following subsections explain the purpose and functions of each stage.

Stage 1: Extracting Indicators Based on Production Types
The starting point of the framework begins with the selection of the production type: Make to Stock (MTS), Make to Order (MTO) and Engineer to Order (ETO). This selection identifies the context of the manufacturing circumstance and presents the basis of the related indicators. Once the production type is identified, it is reflected in the indicators list in Stage 2. Each product type has weighted indicators that adequately reflect the production circumstance as shown in Table 4. Defined weighted indicators play an important role as additional performance factors when Stage 5 is progressing. The output of this stage presents each management level with a list of indicators for the subsequent stages. The main challenge is identifying the indicators that reflect the company's circumstance. Additionally, the indicators must be extracted in a balanced approach

Stage 2: Selecting Management Levels
In general, there are three hierarchical levels in manufacturing components: strategic level, tactical level and operational level. These levels are distinguished by their duties, position, job tenure, etc. The strategic level consists of the board of directors, president, vice-president and other members of the chief executives. They are responsible for controlling and overseeing the entire organization's goals. The tactical level consists of the department managers related to production. They plan dispatch production, arrange production scheduling in detail, and supervise the shop floor. Lastly, the operational level consists of production workers on machines. These workers perform the production activities in manufacturing facilities including the equipment, inventory and storage areas. At this stage, the stakeholders who execute the assessment are selected.

Stage 3: Verification of Extracted Indicators
Throughout this stage, each management level checks the indicators in the list, and the outputs of this stage will be used in the performance assessment. This stage also monitors whether the previous stages are being satisfactorily performed such that the desired output can be attained. To meet the objectives, this stage is complemented with three different indicator lists, because each management level might choose different indicators based on their needs. If inadequate selections are identified, the process returns to the Stage 1. This process should be repeated until dissatisfaction has converged to a small value. In general, it is appropriate to perform the repeated process one to three times because relative improvement of the reliability value decreases as the number of iterations increases.

Stage 4: Assessment
This stage includes performing questionnaires based on the extracted indicators to make a grade, and the AHP/ANP analysis is applied to deduce the final results. If the basic value is drawn through the AHP/ANP analysis, the priority value among criteria, the priority value among indicators and the measured value of the overall production performance are derived with additional formulae. During this stage, the level of performance is calculated and the level of inconsistency among the management levels regarding each indicator and criterion can be measured. If the level of inconsistency is proven to be too big, Stage 5 is executed without the preceding stages. Figure 5 shows a flow chart of the assessment process. Step 1 includes the survey, the AHP/ANP analysis, and the calculation of an index through an additional formula. The priority extracted indicators are estimated through pair-wise comparisons made by interested parties in the strategic, tactical, and operational levels using indicators acquired from the previous stage. In addition, the scores of the indicators and the criteria are rated on a point scale where 1 is poor, 3 is normal and 5 is excellent. Consequently, the mean estimated by each interested party is allotted as a Score. The Criteria Weight (CW), which is a rating of the importance of each criterion is determined through the ANP using pair-wise comparison. The Indicator Weight (IW), which is a value of the importance of each indicator, is determined through the AHP. Using CW and IW, the final outputs of performance measurement, criteria priority and final performance value, can be derived. The notation for the measurements of indices in this study are listed as follows: i Interested party index (i = 1, 2, 3), j Criteria index (j = 1, 2, . . . , 6), k Indicator index (k = 1, 2, ..., m), m Number of extracted indicators, α Weight by production environment, TWC ij Score of the jth criterion from the ith interested party (criteria priority), IW ijk Weight of the kth indicator of the jth criterion from the ith interested party (AHP), Score ijk Score of the kth indicator allotment of the jth criterion from the ith interested party, OP i Overall point of the ith interested party and CW ij Weight of the jth criterion from the ith interested party (ANP).
The TWC ij and OP i can be calculated by using the following equations, and these results are to be analyzed to induce continuous improvement.
The criteria priority (TWC ij ) determined through Equation (2) indicates which criterion has more importance than other criteria. The value α is used to allot greater importance to the priority value of a given criterion. This value can be estimates as α = IW + 1, because IW has an importance weight score that is always under 1. Equation (3) is used to acquire the final outcome of the measurement. Through this value, the performance level of the current production scheme can be identified at each management level.
Step 2 is the evaluation of the inconsistency via analysis of the values derived from Step 1. The notations for the inconsistency and their definitions are listed as follows: l ik Lower value of the score of the kth indicator of the jth criterion given by each management level, u jk Upper value of the score of the kth indicator of the jth criterion given by each management level, m jk Difference between highest value (u jk ) and lowest value (l jk ), l jk Minimum value of the lower values existing in the common section and m jk Maximum value of the difference value (m jk ) existing in the common section.
Based on these notations, we can identify two issues: inconsistency and discrepancy for inducing continuous improvement. First, the inconsistency, also called the gap of indicators among management levels, can be analyzed. One of the most significant characteristics of this framework is the ability to select indicators to consider through consultation with interested parties. Different selections of indicators mean that the part to which importance is attached in considering a manufacturing strategy differs depending on each management level. The difference in the selection, expressed as a gap, becomes the factor that may be the reason for an inefficiency of production. Here, unless the selected indicator was extracted by specific management level, the value is 0 or l ik = 0. If the minimum value of all indicators is zero, then l jk = 0. A gap is judged to be the most serious inconsistency.
Second, large discrepancies in each management level are judged. A large discrepancy suggests that there are huge strategy gaps among the management levels resulting in lower productivity.
The upper values (u jk ) and the lower values (l jk ) of specific indicators for a specific criterion given in each management level are measured. In general, if the difference between them (m jk ) is more than 0.2, the criterion must be analyzed. In order to assign an order of appropriate measurement, priority should be decided. First, the m jk values should be sorted in a descending order then the indicators with the maximum value, that is, m jk should be sent to next stage.

Inducing Continuous Improvement
Second, large discrepancies in each management level are judged. A large discrepancy suggests that there are huge strategy gaps among the management levels resulting in lower productivity. The upper values (u jk ) and the lower values (l jk ) of specific indicators for a specific criterion given in each management levels are measured. In general, if the difference between them (m jk ) is more than 0.2, the criterion must be analyzed. In order to assign an order of appropriate measurement, priority should be decided. First, the m jk values should be sorted in a descending order than the indicators with the maximum value, that is, m jk should be sent to the next stage.
The output of the analysis of inconsistency in Step 2 of the previous stage may be analyzed to drive continuous improvement. An example of a noticeable table of Gap and Discrepancy is shown in Table 5. The 'o' symbol means that the specific management level chose the indicator, whereas the '×' mark means that it was not be selected. First, if a gap occurs, it is considered that there is a serious disagreement among management levels regarding the corporate and manufacturing strategy. In this case, the interested parties among the management levels discuss their opinions on the indicators for which gaps had occurred. In addition, they discuss the importance of the specific indicators so that there will not be any gaps in the future, and they agree on whether to even continue extracting indicators in the future.
Second, if a discrepancy occurs, it is necessary to reconcile the difference between the approaches of a strategy among management levels. After indicators with a large m jk value are established, it is important for the interested parties to determine the current status and understand the degree of resource commitment regarding the related process. Through this, the difference in perspectives on the measurement of the performance factor among management levels is analyzed. It is necessary for each management level to resolve this by presenting opinions regarding the indicators.

Selecting Production Type and Extracting Indicators
In order to verify the utilization of this framework, we have conducted a case study of a steel manufacturer. This company is the primary vendor of POSCO, which is the largest steel company in the Republic of Korea. The company consists of 120 workers, 70 of which were working on production lines. Six strategic managers, five tactical managers and five operational workers were invited to assess the performance. The company utilizes an MTO production strategy owing to its collaboration with a large steel company. As mentioned in Section 3.4, the indicators (34 strategic, 37 tactical and 21 operational) were presented in a survey through consultation with each management level.

Step 1: Questionnaires and AHP/ANP Application
We performed a pair-wise comparison survey for the criteria and indicators to apply the AHP/ANP analysis. Because the score allotment for criteria assumes the mutual interaction relationship, it was analyzed by using the ANP. Table 6 presents the score allotment. Table 6. Priority weights of the criteria after analytic network process (ANP). The calculated consistency of each management levels was in the range of 0.1 to 0.2, afforming the reliability of the data. First, it was found through the data that there is a similar priority of criteria between the strategic and tactical levels, but that of the operational level was somewhat different. From this fact, we could identify that there was a difference for priority of criteria between the office level and the plant level. Then, the AHP was applied when allotting a score to an indicator. To acquire a result, each stakeholder provided a pair-wise comparison for all indicators and allotted a score (1,3,5) for each indictor. The allotted scores show the averages of indicator scores and the IW means the results of AHP. Table 7 contains the scores of indicators in the criterion of quality. The consistency also shows reliability and the '×' mark in Table 7 means that the specific management level did not choose the indicator, specified as a gap in the analysis of inconsistency. This means that gap analysis is required to fix the problem.

Criteria
Next, the outcome scores of the final production performance were calculated using Equation (2) and they are shown in Table 8. If a value history is available, this value allows for analysis of the current status of production performance via comparison to previous values. Moreover, it becomes possible to compare individual management levels by using this final outcome score value. Using Equation (3), the overall production performance score can be derived: Strategic is 3.375, Tactical is 3.596, and Operational is 2.878. Because the operational level score was lower than scores of other management levels, the indicators allotted to the lower scores must be checked, and their causes must be identified.

Step 2: Gap and Discrepancy Analysis for Inducing Continuous Improvement
In this case study, it was found that a gap had occurred in eight of the indicators: four in 'Quality', two in 'Overall waste', one in 'Human resources' and one in 'Continuous improvement'. The results of Gap analysis were shown in Table 9.
From discussions with management levels regarding the outcome of the gap, the indicator that seemed to most significantly contribute to the gap analysis was the 'Frequency of production innovation activity'. The strategic and tactical levels selected this indicator, but the operational level did not because of the resistance to innovation the work site. The strategic and tactical levels wanted to heighten productivity through innovation activities, but the operational level was not interested in innovation because there were fewer advantages. For this, a long-term plan for innovation was established.
Next, from the analysis of the discrepancy indicator extracted with IW, it was determined that there were differences in the priority of waste by inventory, standardization rate, Cpk, etc. Figure 6 shows the details of the discrepancy.

Continuous improvement
Frequency of production innovation activity × Figure 6. A graph of discrepancy.

Discussion of Application Study
From the discrepancy analysis, it was found that there were great differences in the indicators in the production and quality criteria. This means that different priorities existed during work site production depending on each management level, so the problem would be found through consultation. The following text describes consultation feedback. The indicators that contained the most significant results were those of the 'Waste by stock' and 'Standardization rate'. The value of 'Waste by stock' was high in the tactical level yet low in the strategic level. Investigating this indicator, it was determined that stock management was not performed well at the site and many expenses were paid according to the stock. On the site, stocks and products were stored in outdoor yards (in addition to indoor storage), and the managers in the tactical level felt a need for the expansion of the plant; in contrast, the stakeholders at the strategic level had yet to even consider expansion. Next, the value of 'standardization rate' indicator was high for the operational level yet low for the tactical level. Investigating this indicator, it was determined that the site was interested in increasing efficiency through standardization, but the tactical level had a lower priority for the standardization. Through this observation, the tactical level could understand the desire for standardization at the site and develop the standardization methods for the future.
As a result of the case study, it was found that there the significance of indicators varied. It seems that this was primarily caused by varying interest in the priority of the work to be completed at each level. In order to overcome this limitation, it is necessary to identify the problems through consultation among the management levels for the indicators with high discrepancies. Among the 34 indicators, the 10 indicators that were prioritized at all management levels are as follows; input of materials; Waste by accident; Delivery observation rate; Understanding process rate; Manufacturing cycle time; Waste by stock; Pass rate of finished goods; C pk ; Innovation of production process; Straight to process rate.

Conclusions
In this study, we investigated business excellence models for extracting related indicators. Based on the extracted indicators, we built a performance measurement structure for applying a hybrid AHP/ANP. A five-stage framework of performance measurement was proposed. It includes estimating the weight between hierarchical consistent indices through applying the hybrid AHP/ANP in the assessment stage and extracting implications to be applied to the site in the diagnostic stage. A case study was conducted to verify the proposed framework. This research will be very helpful to managers who want to evaluate their performance.
As the market environment has complex factors, the evaluation of performance is widely accepted as an MCDM problem. Especially in the manufacturing industry, which has complete distinctive management levels, lessening the gap and discrepancy among management levels is very important to get the company's competitiveness. Our proposed framework affords a steppingstone for identifying those issues. Our proposed model can be used by small-and medium-size companies to measure the performance completely, because building structure of indicators and doing pair-wise comparison to each management level requires time and is cost consuming. Moreover, defining the relations among indicators is ambiguous as company size is bigger. Our proposed model fits the manufacturing industry, models (Baldridge, EFQM, Deming, etc.) from which we extracted the criterion are specialized in manufacturing. Moreover, a characteristic of our research, dividing the management level into three, is easily applied into manufacturing industry.
This study has four aspects to be improved in future research. First, if more performance indicators are selected, the assessment process will be rather time consuming. Thus, there is a challenge to neglect unimportant indicators before assessment. Developing the knowledge management system is helpful in this issue. Second, more case studies could verify and strengthen the proposed framework. Further case studies are recommended to analyze the framework's usefulness and difference between companies or industries. Third, as many states of the art technologies, such as the Internet of Things (IoT), are being applied into the real sites, new indicators should be developed to reflect their performance. Feedback from professionals would definitely enrich the list of indicators. Finally, as environmental factors become important in manufacturing, it is necessary to establish environmental criteria.

Conflicts of Interest:
The authors declare no conflict of interest.