Parsimonious AHP-DEA Integrated Approach for Efficiency Evaluation of Production Processes

Abstract

This document proposes an innovative composite indicator to measure and control the performance of production processes. The aim is to provide a tool for controlling the efficiency of the processes, assessed in relation to the number and the impact of occurring “errors”, which can take into account the opinion of experts in the specific domain. This allows for the definition of a more realistic and effective decision support system. Our composite indicator is based on an integrated approach based on Data Envelopment Analysis (DEA), and a new multi-criteria method such as Parsimonious Analytical Hierarchy Process (PAHP). The results obtained on a real test case, based on the automotive production domain, show that the composite indicator built with PAHP-DEA allows us to have clear evidence of the efficiency level of each process and the overall impact of errors on all the processes under evaluation. From a methodological point of view, we have for the first time combined the new thrifty AHP with the DEA. From an application point of view, this work introduces a new tool capable of evaluating the efficiency of production processes in an extremely competitive sector, exploiting the knowledge of the experts in the domain of errors, internal processes and the dynamics that occur.

1. Introduction

The production processes represent the fulcrum for achieving the efficiency objectives set by the companies (Vesperi et al. 2021). The automotive sector represents a particularly complex and dynamic application field (Canonico et al. 2021) and the companies operating in this sector have a particular attention to the analysis of internal production processes in terms of technical efficiency (Fattoruso et al. 2022). The modifications of plants and working methods, the optimization of procedures in order to reduce / eliminate waste and losses, process flexibility and customer satisfaction with the final product represent some of the cornerstones of the success in the automotive sector (Schonberger 2010). Downstream of a production activity, the analysis of the efficiency of the processes is determined in relation to the number of errors, or discrepancies on the products, that occur in them: (a) if the output is satisfactory, the process must be standardized and maintained; (b) if the output is not satisfactory, it will be necessary to analyze the causes that generated the errors and implement countermeasures within the process to prevent them from recurring (Petrillo et al. 2019).

It therefore becomes necessary for companies to adopt approaches that aim at efficiency in terms of improving production processes through the logic of optimization, monitoring and integration of systems and work methods; one of the approaches that meets these needs is Business Process Management (BPM) (Ammirato et al. 2019a).

A BPM approach can be useful to ameliorate production processes since it follows the process life-cycle by defining feasibility analysis necessary to avoid wasting time and resources, supporting managers in controlling that it is done in the best possible way and verifying how well the results are aligned with the stated prevailing objectives, favoring corrective actions. A BPM approach in this sense means responding to the need to be quick in the implementation of new solutions so as not to affect daily operations. This goes through constant monitoring and refinement of processes to strengthen their capacity and repeatability. Therefore, it is necessary to lay the foundations for the definition of an efficiency process and maintaining and improving the process itself over time. Many authors point out that the effects and innovation of a BPM approach within business processes are highly amplifying through the use of Internet-based (IT-based) technologies, particularly from Internet of Things (IoT) technologies (see, e.g., Van der Aalst 2013; Gubbi et al. 2013; Ozil 2015). Innovation due to the fact that IoT-type technologies transform workplaces into cyber-physical spaces (so-called “smart environments”, see Laput et al. 2017) thanks to the simultaneous introduction of smart objects, cloud computing, big data and artificial intelligence (Monostori 2014). Adoption of IoT technologies requires the definition of effective innovation management, more complex than a simple automation process (Forrester 2015). Indeed, IoT makes it possible to integrate people and automated systems into the process through structured workflows to achieve better performance (Del Giudice 2016).

The analysis of efficiency in production processes therefore passes through a careful analysis of the data relating to the errors that occur in them (Hafizi et al. 2019). The IoT devices collect large amounts of data that are processed, transmitted, managed and tracked. Without the ability to make this data usable and to create knowledge, there is no innovation (Eftekhari and Akhavan 2013). In order to allow IoT technologies to be integrated and managed in the best possible way in companies to improve the efficiency of processes, a method is required that allows for identifying the way in which data must be selected and processed (Ammirato et al. 2019b).

The research questions that guided this work are as follows:

• Is it possible to introduce an efficiency indicator in companies in the automotive sector that helps decision makers to pursue efficiency objectives? If yes, is it possible to integrate this indicator adopting IoT technologies through a BPM approach?
• Is it possible to define an indicator that provides the involvement of decision makers? If yes, does the involvement of the decision maker in the construction of efficiency indicators provide reliable results?

Our goal is to propose a methodological approach that provides a new indicator for controlling the efficiency of processes and that takes into account the opinion of the domain experts. In this sense, the goal is to have a more realistic and effective decision support for the company.

We report our theoretical background in Section 2, methodological approach in Section 3, while in Section 4 we illustrate a case study and discuss the main results. In Section 5 we report some concluding remarks.

2. Theoretical background

From the literature emerges that an effective tool for monitoring and measuring performance and its efficiency are the Composite Indicators (CIs) (Esty et al. 2006). These indexes are defined through the weighting and aggregation of sub-indicators (Saisana et al. 2005), representing critical aspects in the application domain under consideration. The use of CIs allows the overall information of the sub-indicators to be represented—even when they are controversial—in a comprehensive way. Among the most used methodologies for the construction of composite indicators are the Multi-Criteria Decision Aiding (MCDA) (cfr. e.g., Munda 2005; Zhou et al. 2006; Hajkowicz 2006) and Data Envelopment Analysis (DEA) (cfr e.g., Cherchye et al. 2007; Ulucan and Atıcı 2010).

A common feature of MCDA approaches is the determination of weights with the involvement of Decision Makers (DMs). This aspect is very often criticized as the objectivity of weights is discussed (Wang 2015). Instead, the non-parametric DEA approach defines weights endogenous for all Decision-Making Units (DMUs) not providing any prior information (Cherchye et al. 2008). In this sense, the problem of the objectivity of weights is solved (de Almeida and Dias 2012).

In the construction of a composite index, the DEA framework provides two main approaches: definition of common weights or differentiated weights for all DMUs. The identification of common weights requires that all DMUs have the same set of weights (see e.g., Emerson et al. 2012): the logic is that the weights should be fair and consistent for each DMU (Hatefi and Torabi 2010) in order to have an objective comparison. The determination of the weights in a differentiated way for the DMU involves the use of a system of preferences that allows maximizing individual performances (Zhou et al. 2007, 2010).

Several authors integrate the use of DEA with MCDA methods (cf., e.g., Olanrewaju et al. 2013; Shakouri et al. 2014; Gouveia et al. 2021; Wang and Dang 2021; Rivero Gutiérrez et al. 2022; Antonio et al. 2022) by exploiting their similarities in formulation (Stewart 1996). Hatefi and Torabi (2010), e.g., propose the construction of a CI through an MCDA-DEA approach in which the entities are evaluated through a series of common weights. The authors propose a comparative study between the models present in the literature. Wang (2015) also defines a CI by defining weights with MCDA approaches. The author also proposes the study of the indicator in the evolution over time by analyzing the underlying driving factors.

Many authors, in particular, foresee the integration of DEA with Analitic Hierarchy Process (AHP) highlighting its advantages. Azadeh et al. (2008) present a method for the performance improvement and optimization of railway systems through simulations involving an integrated DEA-AHP approach. The authors highlight that the integration between DEA and multicriteria approaches is particularly useful when both quantitative and qualitative variables are present. Lin et al. (2011), again, in their work in which they integrate DEA-AHP for the performance evaluation of Chinese local governments, highlight the usefulness of the simultaneous use of DEA with MCDA methodologies when the problem is characterized by several criteria and one wants to evaluate and classify several alternatives. For further work that propose DEA-AHP integration underlining its advantages, one can consult Kuo et al. (2010), who used the DEA and Fuzzy AHP integrated approach for supplier selection by presenting a case study on an auto lighting system company, and Wang et al. (2022), who use DEA and Grey AHP for the analysis of adequacy policies and support mechanisms for sustainable solar energy. It appears evident that the use of MCDA with DEA approaches are particularly useful for systems in which there are qualitative and quantitative evaluation measures, such as in production systems in our case.

From the analysis of the literature, it can be seen that among the multicriteria methods most used in the integration with DEA approaches is the AHP method. The AHP method is a widely used method in multi-criterion decision contexts (Ishizaka et al. 2011). We recall that the AHP is based on the construction of pairwise comparison matrices (PCMs) of alternatives and criteria in order to obtain the priority of the elements being evaluated (Cavallo and D’Apuzzo 2009). However, it should be noted that the literature on MCDA highlights the limitations of AHP for addressing complex decision-making problems. Among the most relevant problems are the number of alternatives which must not be greater than 7 (Ishizaka and Labib 2009) and rank inversion (Belton and Gear 1983), problems in which the addition or deletion of one or more alternatives can modify the final rank (to deepen the debate in the literature, see Maleki and Zahir 2013 and Krejčí and Stoklasa 2018).

Taking into account the advantages of integrated approaches of AHP and DEA and evaluating the limits that the AHP presents in complex problems, our work proposes for the first time the use of a new version of the AHP, the Parsimonious AHP (PAHP), with DEA. PAHP, proposed in the literature in 2018 (Abastante et al. 2018), has all the advantages of the classic AHP and solves its problems thanks to the introduction of reference points that allow the decision maker to:

-

analyze problems with a very large number of alternatives;

-

considerably reduce the number of pairwise comparisons;

-

make the decision-maker more aware in defining their preferences;

-

solve rank reversal problems.

To learn more, see (Abastante et al. 2019).

Therefore, the aim of this paper is to define an innovative approach, based on the integration of PAHP and DEA methodologies, for the efficiency evaluation, by means of the measurement of a proper CI. The proposed framework is also compared in terms of decision support provided, for evaluating the efficiency of production processes. In line with the principles of IoT technologies adoption by means of a BPM approach, the proposed framework can serve as a base for an automated tool able to effectively monitor and control production processes in manufacturing.

The mathematical model for the calculation of CI requires the availability of data relating to a certain set of errors, which in this context act as sub-indicators of performance, which occurred in a given time interval within a certain homogeneous set of processes. Therefore, the aim is to determine for each process, seen in this context as a Decision-Making Unit, a performance measure constructed in terms of a CI, i.e., a complex indicator obtained as a weighted average of various sub-indicators. In this type of analysis, the crucial aspect consists of determining the weights to be associated with the individual sub-indicators in evaluating the performance of each process. In order to ensure objectivity in the comparative assessment process, common weights will be adopted in determining the efficiency of each DMU, and the results will be compared with those obtained with classical Data Envelopment Analysis approaches.

As already stated, the approach we are proposing can be considered as a subjective approach: the weights are defined through an evaluation procedure of the sub-indicators according to a series of criteria suggested and validated by the decision-makers. Classical objective approaches, on the contrary, determine the weights to be associated with each sub-indicator (error) in a completely automatic way, by solving one or more of the optimization models.

We propose a classification of the performance of DMUs, with very recent approaches of multi-criterion analysis, analyzing their potential for improvement by carrying out both static and dynamic analyzes.

The innovative elements of this work are the definition of a composite indicator through the integration of a very recent multicriteria method such as the PAHP with the DEA. Furthermore, our approach provides for the active involvement of DMs as the major holders of knowledge on company managerial issues, essential for being able to interpret and fix errors that occur in the company. In this way, we try to give a healthy subjectivity that can give more coherent solutions in this context than the classic DEA approaches. Moreover, the mathematical models and methods that we propose can be easily implemented and act as the kernel of a decision support system, capable of interfacing with current information tools and with the IoT technologies in use in the production plant. This enables an effective innovation management by combining people and automated systems to achieve better performance.

3. Materials and Methods

The method we propose aims at evaluating the performance of a set of (production) processes $M$, in terms of occurrence and impact of a set of errors $E.$ According to the DEA framework, processes can be considered the DMUs under evaluation, assumed to have one dummy input with unitary value (see, Hatefi and Torabi 2010), while errors are the (undesirable) outputs.

The overall efficiency of process s is measured by means of a $CI$ obtained starting from the weighted sum of the frequency of each error $i$occurring in $s(e_{is}$):

$$C{I}_s=1-\left[{\displaystyle \sum }_{i\in E}{w}_i{e}_{is}\right]{}_{NORM}$$

(1)

In order to compare the $CIs$ obtained for several processes, even very different, we have normalized the value of the weighted sum by considering the ratio with the maximum value measured within the process set, so to have for each process a value in the $\left[0.1\right]$ range.

For the evaluation of the relevance of each error, we consider a set of criteria G, where g_j(e_i) define the evaluation of the error ${\mathit{e}}_{\mathit{i}}$ with respect to criterion g_j. We denote with e_is the occurrence of error $\mathit{i}$ on process$\mathit{s}$. The determination of the weights for each error ${\mathit{w}}_{\mathit{i}}$, common for all the processes, represents a weighted sum of the criteria priority p_j and the errors local priority lp_i:

$${\mathit{w}}_{\mathit{i}}={\displaystyle \sum }_{\mathit{j}=\mathit{1}}^{\mathit{j}}\mathit{l}{\mathit{p}}_{\mathit{i}}·{\mathit{p}}_{\mathit{j}}$$

(2)

where:

${\mathit{p}}_{\mathit{j}}$ is determinated as follow:

$$\mathit{A}·\mathit{v}={\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}·\mathit{v}$$

(3)

In formula (3) $\mathit{A}$ is the Pairwise Comparison Matrix (PCM) (see e.g., Cavallo and D’Apuzzo 2009; Cavallo and Brunelli 2018) built through a judgment of comparison (e_ij) between ordered pairs of errors (${\mathit{e}}_{\mathit{i}}$,${\mathit{e}}_{\mathit{j}}$) by the DM, using the Saaty Scale (Saaty 1977). $\mathit{A}$ is a matrix: (a) ${\mathit{A}}_{\mathit{n}\mathit{x}\mathit{n}}$ positive; (b) reciprocal if ${\mathit{e}}_{\mathit{i}\mathit{j}}=\frac{\mathbf{1}}{{\mathit{e}}_{\mathit{j}\mathit{i}}}$ $\forall \mathit{i}, \mathit{j}$ and with ${\mathit{e}}_{\mathit{i}\mathit{i}}=\mathbf{1}$ $\forall \mathit{i}$; (c) made up of finite elements, in fact for each criterion ${\mathit{g}}_{\mathit{j}}$, ${\mathit{e}}_{\mathit{i}\mathit{j}}\ne$ ∞ $\forall \mathit{i}, \mathit{j}$ (Greco et al. 2016).

Moreover, $\mathit{v}$ is the priorities vector$\mathit{v}=\left\{{\mathit{v}}_{\mathbf{1}},\dots ,{\mathit{v}}_{\mathit{i}},\dots , {\mathit{v}}_{\mathit{j}},\dots , {\mathit{v}}_{\mathit{n}}\right\}$ (see e.g., D’Apuzzo et al. 2007). Considering that the dominance coefficient of each pair of errors ${\mathit{e}}_{\mathit{i}\mathit{j}}=\frac{{\mathit{v}}_{\mathit{i}}}{{\mathit{v}}_{\mathit{j}}}$ is a ratio of their respective weights, is verified that$\mathit{A}·\mathit{v}=\mathit{n}·\mathit{v}$. In fact:

$$\left[\begin{array}{ccc}\frac{{\mathit{v}}_{\mathbf{1}}}{{\mathit{v}}_{\mathbf{1}}}& \dots & \frac{{\mathit{v}}_{\mathbf{1}}}{{\mathit{v}}_{\mathit{n}}}\\ ⋮& \ddots & ⋮\\ \frac{{\mathit{v}}_{\mathit{n}}}{{\mathit{v}}_{\mathbf{1}}}& \dots & \frac{{\mathit{v}}_{\mathit{n}}}{{\mathit{v}}_{\mathit{n}\mathit{j}}}\end{array}\right]\left[\begin{array}{c}{\mathit{v}}_{\mathbf{1}}\\ ⋮\\ {\mathit{v}}_{\mathit{n}}\end{array}\right]=\mathit{n}\left[\begin{array}{c}{\mathit{v}}_{\mathbf{1}}\\ ⋮\\ {\mathit{v}}_{\mathit{n}}\end{array}\right]$$

For the algebra of the matrices $\mathit{v}$ it turns out to be an eigenvector for the matrix $\mathit{A}$ with eigenvalue $\mathit{n}.$ Considering that $\left\{{\mathit{\lambda }}_{\mathbf{1}},\dots ,{\mathit{\lambda }}_{\mathit{n}}\right\}$ are $\mathit{n}$ eigenvalue of $\mathit{A}$, with ${\mathit{A}}_{\mathit{n}\mathit{x}\mathit{n}}$ = ${\mathit{e}}_{\mathit{i}\mathit{j}}$, if ${\mathit{e}}_{\mathit{i}\mathit{j}}=\mathbf{1}$ $\forall \mathit{i}=\mathit{j}$ so ${\displaystyle \sum }_{\mathit{i}=\mathbf{1}}^{\mathit{n}}{\mathit{\lambda }}_{\mathit{i}}=\mathit{n}$, principal eigenvalue ${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}=\mathit{n}$. At this point, it will be sufficient to calculate the vector that satisfies the equation (3). That is, it will be sufficient to determine the principal eigenvector associated with the eigenvalue ${\mathit{\lambda }}_{\mathit{m}\mathit{a}\mathit{x}}$ of the matrix $\mathit{A}$ and subsequently normalize $\mathit{v}$ on the sum of its elements (Saaty 2003; Ishizaka and Nemery 2013).

$\mathit{l}{\mathit{p}}_{\mathit{i}}$ is obtained with a linear interpolation:

$$\mathit{l}{\mathit{p}}_{\mathit{i}}={\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}+\frac{{\mathit{h}}_{{\mathit{l}}_{\mathit{j}+\mathbf{1}}}-{\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}}{{\mathit{l}}_{\mathit{j}+\mathbf{1}}-{\mathit{l}}_{\mathit{j}}}·({\mathit{r}}_{\mathit{j}}\left({\mathit{e}}_{\mathit{i}}\right)-{\mathit{l}}_{\mathit{j}})$$

(4)

in which:

${\mathit{l}}_{\mathit{j}}$ represents a reference point that allows us to reduce the pairwise comparison (Abastante et al. 2019) between the errors. We consider as reference points the same points that partition the data interval by equal parts (Abastante et al. 2018);

${\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}$ is the priority associated to reference point ${\mathit{l}}_{\mathit{j}}$ obtained with the eigenvalue method (formula (3));

${\mathit{r}}_{\mathit{j}}\left({\mathit{e}}_{\mathit{i}}\right)$ represents the evaluation of the error ${\mathit{e}}_{\mathit{i}}$ with respect to criterion ${\mathit{g}}_{\mathit{j}}$;

${\mathit{h}}_{{\mathit{l}}_{\mathit{j}+\mathbf{1}}}-{\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}$ determinate the weighted difference between two reference points;

${\mathit{l}}_{\mathit{j}+\mathbf{1}}-{\mathit{l}}_{\mathit{j}}$ represents the difference between two reference points.

We highlight that ${\mathit{p}}_{\mathit{j}}$ is defined by means of the PCM built with verbal evaluation by DMs using Saaty Scale (Saaty 1977); instead, $\mathit{l}{\mathit{p}}_{\mathit{i}}$ is determinated with a linear interpolation formula, this allows to compare the real performance of the errors ${\mathit{r}}_{\mathit{j}}\left({\mathit{e}}_{\mathit{i}}\right)$ respect the value of reference points ${\mathit{l}}_{\mathit{j}}$ defined with the PCMs.

4. Results

4.1. Description of Case Study

Our testing experience has been carried out in an international company operating in the automotive sector. The plant under analysis is located in southern Italy and carries out the assembly of the components on the engines. The production process of the plant is characterized by 9 processes:

−

Process 1: distribution shaft;

−

Process 2: cast iron;

−

Process 3: base crankshaft;

−

Process 4: base all;

−

Process 5: cylinder head;

−

Process 6: short block;

−

Process 7: cylinder head assembly;

−

Process 8: long block;

−

Process 9: picking.

In each process, operations are performed on the engine before it moves on to the next process. The main objective of the company is to guarantee the efficiency of the processes by reducing the errors that occur in the plant. The processes considered are impacted by 19 main errors categories; by way of example, a category of error can be considered a tightening operation (for reasons of confidentiality, the names of the other categories of errors are not reported). Each error can occur in one process or in multiple processes. The data are systematically collected in matrices by the company as errors occur in each process. We analyzed the errors related to the 9 processes that occur in two consecutive years.

Table 1. Errors found in processes in 2019 and 2020.

	2019									2020
	Process 1	Process 2	Process 3	Process 4	Process 5	Process 6	Process 7	Process 8	Process 9	Process 1	Process 2	Process 3	Process 4	Process 5	Process 6	Process 7	Process 8	Process 9
EC 1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	0	0
EC 2	0	0	0	0	0	2	0	0	0	0	0	0	0	0	0	4	0	0
EC 3	0	0	0	0	3	0	0	0	0	0	0	0	0	0	1	0	0	0
EC 4	0	0	0	0	0	0	66	0	0	0	0	0	0	0	0	1	0	0
EC 5	0	0	0	0	0	0	2	0	0	0	0	0	0	27	0	0	0	0
EC 6	0	0	0	0	2	0	2	0	0	0	0	0	0	0	0	4	0	0
EC 7	0	0	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	0
EC 8	6	0	0	4	0	0	22	0	0	134	0	0	0	5	17	0	0	350
EC 9	0	7	0	0	0	0	0	0	0	0	0	0	0	0	35	34	30	0
EC 10	0	0	0	0	0	2	0	0	5	0	0	0	0	0	0	4	0	0
EC 11	38	0	0	0	0	0	0	2	0	22	0	0	0	1	0	18	0	15
EC 12	0	0	0	0	0	0	7	0	2	0	0	0	0	0	0	1	2	0
EC 13	0	0	0	0	0	0	11	0	0	0	0	0	0	0	0	1	0	1
EC 14	0	0	0	0	0	1	0	0	0	0	0	0	0	0	3	0	0	0
EC 15	0	0	0	0	2	1	0	0	0	0	0	0	6	0	0	4	0	0
EC 16	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	2
EC 17	0	0	3	1	5	0	14	0	0	0	0	3	0	0	0	70	100	3
EC 18	1	0	0	0	0	0	11	0	0	0	0	0	0	0	0	3	42	0
EC 19	0	5	0	0	0	0	0	11	0	0	10	0	0	0	0	0	0	0
TOTAL	45	12	4	5	12	6	137	13	7	156	10	4	6	33	56	145	174	371

shows the data relating to the frequency of each Category of Error (EC) for each process considered for the 2019 and for 2020.

In complex realities such as the automotive production plants, the knowledge of the decision maker on business problems is essential to be able to interpret and correct the errors that occur in the company; when talking about the efficiency of production processes means not only reducing errors but also knowing and evaluating them individually so that they can be faced and prevented from repeating themselves. Upon detecting any errors in the processes, the company provides for their evaluation taking into account four criteria: frequency, cost, detection and severity. It should be noted that the criteria are identified and normally used by the company for the evaluation of errors. In this sense, we can assume they are suitable for the adoption within our approach.

Error evaluation aims to define a priority for each error. This allows us to analyze the efficiency, also taking into account the type of error that occurs in it (see, Fattoruso et al. 2022). On the basis of these evaluations, we have built a procedure for the weights definition with an PAHP approach which foresees the direct involvement of the decision maker.

In our case study it was built with the involvement of the control manager on the production process that from here on we will call DM for simplicity. The control manager has the task of verifying the correct functioning of each process and of detecting, analyzing and defining corrective actions for the errors that occur in it.

4.2. Solution Analysis

In order to calculate the weights ${\mathit{w}}_{\mathit{i}}$ for each error we have applied the methodology described in Section 2.

The first step is the determination of the criteria priority ${\mathit{p}}_{\mathit{j}}$. For this purpose, we have built with the DM a PCM (

Table 2. Pairwise comparison matrix between criteria.

	Frequency	Cost	Detection	Severity	${\mathit{p}}_{\mathit{j}}$
Frequency	1	1/2	1/5	1/9	0.046
Cost	2	1	1/3	1/9	0.076
Detection	5	3	1	1/9	0.143
Severity	9	9	9	1	0.736
				Consistency Index	0.09

), obtaining the priority vector by applying formula (3). For the construction of the PCM shown in Table 2, we asked the DM to compare the criteria in pairs, expressing his preferences, using the Saaty scale (Saaty 2001).

The second step for the construction of ${\mathit{w}}_{\mathit{i}}$ is the identification of errors local priority$\mathit{l}{\mathit{p}}_{\mathit{i}}$. The evaluation ${\mathit{r}}_{\mathit{j}}\left({\mathit{e}}_{\mathit{i}}\right)$ has been provided by the company (we report the evaluations in

Table A1. Evaluation ${\mathit{r}}_{\mathit{j}}\left({\mathit{e}}_{\mathit{i}}\right)$ for 2019 and 2020.

	2019				2020
	Frequency	Cost	Detection	Severity	Frequency	Cost	Detection	Severity
EC 1	1	5	15	5	1	5	10	3
EC 2	2	5	15	5	4	5	25	5
EC 3	3	5	15	4	1	5	25	4
EC 4	66	1	25	4	1	5	25	5
EC 5	2	2	10	5	27	5	2	2
EC 6	4	2	25	4	4	5	10	3
EC 7	2	2	10	4	1	5	10	4
EC 8	32	2	10	4	506	4	25	4
EC 9	7	1	10	5	99	5	3	3
EC 10	7	1	10	4	4	5	10	3
EC 11	40	1	3	3	56	5	10	3
EC 12	9	1	3	4	3	5	1	4
EC 13	11	1	3	4	2	5	25	5
EC 14	1	1	3	5	3	5	10	3
EC 15	3	1	3	4	10	5	10	4
EC 16	1	1	3	4	2	5	10	3
EC 17	23	1	3	3	176	5	10	5
EC 18	12	1	3	3	45	5	10	5
EC 19	16	1	25	4	10	5	25	5

in Appendix A). Based on the number of errors to be analyzed as suggested by Abastante et al. (2019) we have identified 6 reference points (${\mathit{l}}_{\mathit{j}}$) (

Table A2. Reference points ${\mathit{l}}_{\mathit{j}}$ for 2019 and 2020.

	2019				2020
	Frequency	Cost	Detection	Severity	Frequency	Cost	Detection	Severity
$l_1$	0	0	0	0	0	0	0	0
$l_2$	13.2	1.5	1.5	1	101.2	1.5	1.5	1
$l_3$	26.4	2.5	5	2	202.4	2.5	5	2
$l_4$	39.6	3	15	3	303.6	3	15	3
$l_5$	52.8	4	20	4	404.8	4	20	4
$l_6$	66	5	25	5	506	5	25	5

in Appendix A) and we built the PCMs for each considered criterion by deriving the priority ${\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}$(we show the priorities ${\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}$ in

Table A3. Priorities ${\mathit{h}}_{{\mathit{l}}_{\mathit{j}}}$.

	Frequency	Cost	Detection	Severity
$l_1$	0.032	0.043	0.028	0.043
$l_2$	0.056	0.064	0.053	0.064
$l_3$	0.099	0.101	0.104	0.101
$l_4$	0.170	0.16	0.163	0.16
$l_5$	0.251	0.25	0.267	0.25
$l_6$	0.391	0.382	0.385	0.382

in Appendix A) with formula (3). Thus, applying formula (4) we derived errors in local priority showed in

Table 3. Errors local priority for 2019 and 2020.

	2019				2020
	Frequency	Cost	Detection	Severity	Frequency	Cost	Detection	Severity
EC 1	0.034	0.382	0.163	0.382	0.032	0.382	0.134	0.118
EC 2	0.036	0.382	0.163	0.382	0.033	0.382	0.222	0.382
EC 3	0.037	0.382	0.163	0.250	0.032	0.382	0.222	0.250
EC 4	0.391	0.057	0.385	0.250	0.308	0.113	0.385	0.340
EC 5	0.036	0.083	0.134	0.382	0.038	0.194	0.086	0.014
EC 6	0.039	0.083	0.385	0.250	0.033	0.194	0.031	0.118
EC 7	0.036	0.083	0.134	0.250	0.032	0.194	0.134	0.250
EC 8	0.129	0.083	0.134	0.250	0.312	0.157	0.222	0.250
EC 9	0.045	0.057	0.134	0.382	0.055	0.113	0.092	0.118
EC 10	0.045	0.057	0.134	0.250	0.033	0.113	0.134	0.118
EC 11	0.172	0.057	0.075	0.160	0.028	0.113	0.177	0.160
EC 12	0.048	0.057	0.075	0.250	0.033	0.113	0.046	0.250
EC 13	0.052	0.057	0.075	0.250	0.032	0.113	0.395	0.382
EC 14	0.034	0.057	0.075	0.382	0.033	0.113	0.177	0.118
EC 15	0.037	0.057	0.075	0.250	0.034	0.113	0.177	0.250
EC 16	0.034	0.057	0.075	0.250	0.032	0.113	0.177	0.118
EC 17	0.088	0.057	0.075	0.160	0.088	0.113	0.177	0.278
EC 18	0.054	0.057	0.075	0.160	0.043	0.113	0.177	0.278
EC 19	0.065	0.057	0.385	0.250	0.017	0.113	0.385	0.382

. In

Table 4. Errors local priority for 2019 and 2020.

	EC 1	EC 2	EC 3	EC 4	EC 5	EC 6	EC 7	EC 8	EC 9	EC 10	EC 11	EC 12	EC 13	EC 14	EC 15	EC 16	EC 17	EC 18	EC 19
${\mathit{w}}_{\mathit{i}}$ for 2019	0.335	0.335	0.238	0.261	0.308	0.247	0.211	0.215	0.307	0.209	0.141	0.201	0.201	0.298	0.201	0.201	0.137	0.135	0.246
${\mathit{w}}_{\mathit{i}}$ for 2020	0.136	0.343	0.246	0.300	0.019	0.108	0.219	0.242	0.111	0.116	0.150	0.201	0.348	0.122	0.219	0.122	0.243	0.240	0.346

we report the weight ${\mathit{w}}_{\mathit{i}}$ obtained with formula (2).

At this point, we have defined the overall efficiency of the processes measured by the composite indicator defined with the formula (1). The CI values are shown in

Table 5. Composite indicator for 2019 and 2020.

	Process 1	Process 2	Process 3	Process 4	Process 5	Process 6	Process 7	Process 8	Process 9
2019	0.781	0.891	0.980	0.968	0.926	0.949	0	0.903	0.953
2020	0.595	0.961	0.989	0.985	0.979	0.902	0.679	0.569	0

.

As we can see, processes 3 and 4 are the more efficient in both 2019 and 2020. Some considerations can be made also on the variations from one year to the other, in order to evaluate the effectiveness of decisions made on the process’s execution. For example, for process 7 it is clear an improvement of the efficiency from 2019 to 2020, maybe thanks to some corrective actions performed on this process. On the other hand, process 9 has registered the worst variation, calling for an accurate analysis of the causes of the errors and for some significant adjustments.

In order to better analyze the evolution over time of processes performance, we have calculated the Malmquist index (see, e.g., Malmquist 1953; Fare et al. 1994; Tone 2004), a common measure that has been widely adopted in several application domains (Wang et al. 2013, 2014). Within our computational experience, we have considered the following Malmquist Composite Index (MCI) to measure the variations of the CI for each process $s$ from year $t$ to year $t+1$:

$$MC{I}_{t,s}^{t+1}={\left[\frac{C{I}_s^t\left(t+1\right)}{C{I}_s^t\left(t\right)}·\frac{C{I}_s^{t+1}\left(t+1\right)}{C{I}_s^{t+1}\left(t\right)}\right]}^{1/2}$$

where $C{I}_s^x\left(y\right)$ represents the CI evaluation performed in year $y$ with the weights defined for year $x$. The Malmquist index values for 2019 and 2020 are reported in

Table 6. Malmquist composite indicator for 2019 and 2020.

	Process 1	Process 2	Process 3	Process 4	Process 5	Process 6	Process 7	Process 8	Process 9
MCI	0.751	1.059	1.014	1.018	0.999	0.889	$+\infty$	0.668	0

.

Since MCI is the geometric mean of the ratios of CI values of two consecutive years, the value it can assume is in the $\left[0,+\infty \right]$ range. However, a MCI lower than 1 stands for a worsening of the process efficiency from one year to the next one, while a value larger than 1 shows an improvement. The analysis of data in Table 6 confirms that process 7, which was completely inefficient in 2019, has the best improvement in 2020, while process 9 had a breakdown. More generally, this index allows the dynamic monitoring of performance evolution over time.

4.3. Comparison with Benchmark Models

The efficiency evaluation obtained with the proposed method (MCDA model), which takes into account the expert judgements of DMs about the impact of errors, has been compared with the results generated by two DEA approaches (see, Hatefi and Torabi 2010). The first one considers the Best Possible Weights (BPW model) set for the CI evaluation of each process. According to the classical DEA framework, the weights are obtained by solving a different input-oriented optimization model each time, having the maximization of the efficiency of a different DMU as an objective. Even if a judgement of efficiency for a DMU could be favoured by the specific choice of the weights, a result of inefficiency is clearly indisputable. The second model adopts a Common Weights set (CW model) for all the DMUs. This approach contrasts with classical DEA framework (Karsak and Ahiska 2008), but has the aim of allowing a more “fair” basis for the CIs calculation, and thus for the DMUs comparison, and of reducing the number of DMUs that result efficiency. The set of weights is obtained by solving a unique optimization model, having the objective of minimizing the maximum value of inefficiency of all the DMUs (see model (6) in Hatefi and Torabi 2010). In the following

Table 7. Composite indicator for 2020 for MCDA, BPW and CW model.

	Process 1	Process 2	Process 3	Process 4	Process 5	Process 6	Process 7	Process 8	Process 9
MCDA	0.595	0.961	0.989	0.985	0.979	0.902	0.679	0.569	0
BPW	1	1	1	1	1	1	1	1	1
CW	0.71	1	1	0.71	1	1	1	0.71	1

, we report the efficiency values obtained with our MCDA approach and the benchmark models described.

As we can see, the two “objective” models, which define the weights just based on numerical considerations, fail to distinguish the processes and provide poorly significant evaluations of their efficiency. In particular, the BPW model labeled all the processes as efficient, even if several errors are registered. In addition, CW model results show some inconsistency, since for example process 9 is considered efficient even if it presents the larger number of errors (see last row of Table 1).

The inconsistency of the results obtained with these two models is mainly due to the not so high overall frequency of errors on the processes and, more specifically, because for each process just a small subset of possible errors is relevant. For this reason, the weights definition by means of an optimization step can lead to values that are poorly constrained, and thus less representative of the real impact of the errors.

5. Conclusions and Discussion

This paper proposes an innovative composite indicator constructed by integration of a new multicriteria method as Parsimonious AHP and Data Envelopment Analysis. The analysis of process efficiency in the automotive sector is based on the occurrence of errors and on their impact in a production process. However, errors are not all the same and require a priori multi-criteria evaluation to define their priority.

The methodological approach we propose aims at providing a controlling tool for processes’ efficiency that can take into account the opinion on the error impact of domain experts, with the aim of having a more realistic and effective decision support. This paper shows how the combination of Parsimonious AHP and DEA can be useful for analyzing the efficiency of production processes in the automotive sector. In particular, the use of PAHP with DEA allows exploiting all the main advantages inherent in multicriteria methods including the involvement of the decision maker in the analysis of the problem and in the definition of the priorities of criteria and errors. The involvement of the DM in the construction of multicriteria methodologies allows in detail to exploit the knowledge of errors and internal processes and the dynamics that occur in them. In this way, with the adoption of multicriteria methods, it allows to obtain coherent and truthful priority of errors with respect to the context in which they occur. In this sense, the integration of PAHP with DEA returns a more consistent analysis of the efficiency of the processes to the environment under analysis, compared to the classic DEA approaches. This allows for greater confidence in the results and prevents those responsible for controlling the processes from carrying out further analyzes on the relevance of the results obtained, as could be the case, for example, using purely classical DEA approaches.

In particular, the results obtained on a real-life test case, based on the automotive production domain, show that the PAHP-DEA method allows having clear evidence of the efficiency level of each process and the overall impact of errors on the entire processes set. An accurate analysis of results can also lead to a possible revision of the criteria adopted to define the weights associated to errors, in order to have a significant comparison framework. Moreover, the efficiency analysis can also be carried out in a dynamic fashion. DMs can observe the efficiency evolution of each process, also by means of the Malmquist index, so to have a measure of the effectiveness of actions aimed at reducing the number of errors and their impact on the processes.

Finally, the comparison with other benchmark DEA models highlights the effectiveness of the proposed approach to serve as an efficiency controlling tool, in particular when the relation matrix between the sub-indicators (errors in our test case) and the entity under evaluation (production processes) is quite sparse.

The composite indicator determined as an integration of these methods is completely innovative for the scientific field, but also for the company. In fact, even if the company has carried out the analysis and data collection, process efficiency analyzes have never been carried out using the integration of multi-criteria analysis approaches and Data Envelopment Analysis. The methodological framework we have defined has aroused great interest by DMs of the plant used as a test case, so that they are considering its implementation and integration within the IT systems and the introduction of the efficiency analysis among the standard quality assurance procedures of the company.

From a managerial insights standpoint, the DM can benefit in several ways by the adoption of such a control tool, as already confirmed by the computational experience described early in this section. Here we briefly summarize the main advantages for the DM in the production planning and control processes:

It allows direct involvement of the DM in all phases of the construction of the multi-criteria method. This allows for the definition of a double advantage: the first one is to build a method that focuses on the crucial aspects related to the production process by characterizing and analyzing in detail the errors that make it up and defining the criteria useful for their evaluation; the second one is related to a greater awareness of the evaluations expressed for the determination of the priorities of criteria and errors and a deeper consciousness of the production context is acquired. The active involvement of the DM in the construction of the multicriteria method plays a decisive role in the understanding of the methodology and in its integration as a new control tool. Furthermore, it should be noted that once the method has been built, it is extremely flexible (introduction of new criteria, alternatives or modification of the field of application) this allows the DM to adapt the model over time according to the new needs declared prevailing by the company.

The possibility to have a fair measure of the performance of each production process, also in a multiperiod fashion, can provide precise monitoring of processes’ performance and assessment of the effectiveness of corrective actions to adopt on the sources of errors.

The effectiveness due to the adoption of an efficiency evaluation approach is by far enhanced by its integration with the plant IoT infrastructure. The availability of data about errors in a “quasi” real-time way allows for a greater frequency in monitoring, without necessarily having to wait for the end of a predetermined period. In this way, the possibility of intervention is almost immediate, allowing a clear improvement in overall efficiency.

The proposed approach can also be extended to other sectors as e.g., the air transport sector (see e.g., Baltazar et al. 2014) or public administration sector (see e.g., Longaray et al. 2018).

Future research directions will include the modelling of uncertainty in some of the parameters and in the PCM and the definition of Stochastic Programming models for the weights optimization, which can include risk measures on the efficiency level variability.

We highlight that this study had some limitations, which need to be considered in the analysis of its results. First, we’ve only tested our approach for two consecutive years, so its results may not be broadly generalizable. We plan in the future to test the proposed approach on historical and current data.

However, we believe that, after a preliminary computational experience, this work is quite consistent with the study objectives and the nature of the research questions. In fact, we believe that the paper can provide insights that could be important to advance the theory.