Multicriteria Ranking for the Efficient and Effective Assessment of Police Departments

The nonparametric assessment of police efficiency and effectiveness is challenging due to the stochastic nature of criminal behavior and the subjective dependence on multiple decision criteria, leading to different prospects depending on the regulation, necessity, or organizational objective. There is a trade-off between sustainable efficiency and effectiveness in many police performance assessments, because many departments can be crime-specialized or cannot reproduce good results effectively on more severe or complex occurrences. This study aims to provide a non-compensatory ranking classification combining Conditional Frontier Analysis with the PROMETHEE II methodology for the multidimensional efficiency and effectiveness analysis of police. The results on Pernambuco (Brazil) Police departments offer interesting perspectives for public administrations concerning prioritizations of units based on the mitigation of resources and strategic objectives.


Introduction
The sustainable development of a society requires the optimal usage of resources for the provision of goods and services and the ability to reach the desired social goals. Efficiency can be defined as the capacity to avoid wasting materials, resources, efforts, or time to produce a result or outcome. This concept is strictly related to sustainability. On the other hand, effectiveness can be defined as the ability to produce a desired result. This concept is strictly related to quality. In the simple words of Peter Drucker, efficiency is doing things right, while effectiveness is doing the right things. These two perspectives are not always walking in the same direction and their potential conflict can jeopardize some of the promising prospects of sustainable service provisions, especially considering public administrations.
There is a recurrent trade-off between quality and efficiency in many empirical assessments [1][2][3]. Lo Storto [4], investigating the relationship between efficiency and effectiveness of public expenditure in 108 major Italian municipalities, suggests shreds of evidence for this trade-off, involving public service quality indicators (expenditure effectiveness) and DEA measures for cost-efficiencies. Nepomuceno et al. [5], investigating 88 public and private health service units in Pernambuco, Brazil, also offer support in addition to this discussion. According to the authors, most hospitalization-efficient units are crowded public hospitals working at full capacity most of the year, which can only meet all the demand for hospitalizations (the output in the analysis) by compromising the service's quality.
Some composite indicators, such as the Civil Society Organization Sustainability Index developed by the United States Agency for International Development, offer an ergy management, social science, medicine, agriculture, education, design, government, and sports.
The choice of the PROMETHEE II approach in this application was made due to the (i) methodology characteristic providing a quantitative measure (the Outranking Net Flow) for aggregating different decision perspectives to support the ranking of service units (see the methodology described in the next section), (ii) simplicity of its implementation that is particularly helpful when being explained to policymakers and applied in combination with other methods, and (iii) wide application of the approach in different fields and contexts both from academics and practitioners [26][27][28].
In this paper, we combine robust directional efficiency measures [15,18,29] for policing, conditional to crime as an exogenous factor, with the PROMETHEE outranking for a complete classification of Pernambuco's municipalities based on the sustainable efficiency of each police department in solving three types of crimes, violent crimes (CVLI-Crimes Violentos Letais e Intencionais), street robberies (mugging), and carjacking (or more generally motor vehicle theft and robbery), using officers as input, and based on the effectiveness of reaching the Pact for Life state goal on reducing homicides [15,30,31]. To the best of our knowledge, this is the first application of this robust and combined methodology to assess the efficiency and effectiveness of police units. Our methodology relies on recent advances in conditional frontier analysis used with a traditional PROMETHEE methodology for outranking decision units.
Some assessments of crime and policing in Pernambuco used statistical, multicriteria, and geospatial tools for ranking, clustering, and classifying units and regions according to the vulnerability to homicides [32][33][34], preference learning [35,36], property crimes [37,38], and investigating the violent behavior in Pernambuco [39]. Despite providing valuable support for policymakers, to the best of our knowledge, ranking regions or police units for public security purposes based on a multicriteria combination of nonparametric robust estimators for technical efficiency with measures for effectiveness of results were not featured in the current literature. This combined methodology is presented in detail in the next section, highlighting the potentials not limited to this application. The third section is dedicated to evaluating and discussing the compensatory and non-compensatory rankings generated from different perspectives, which can aid substantial subsidies for strategic decision-making. The conclusion summarizes the proposal, empirical evaluation, and contributions of the paper.

Methodology
Many DEA ranking methods in the productive and efficiency analysis literature are considered post-analysis approaches [13]. The framework illustrated in Figure 1 can be situated in this classification. Four sub-ranks are constructed through pairwise comparisons. Compensations between efficiency and effectiveness are restricted by the imposition of vetoes for clustering effective/ineffective and efficient/inefficient alternatives (municipalities). The municipality is top-ranked when it is sufficiently effective according to the predefined objective, and efficient in using the available resources to produce clear-ups for the specified felonies and misdemeanors. The second sub-rank has effective but not efficient municipalities, i.e., excellent efficiency prospects cannot offset poor effectiveness. If the municipality is efficient in using the available resources to solve crimes but is not effective in reaching the specified institutional goal, it is located in the third sub-rank with similar municipalities. The last sub-rank had both ineffective and inefficient units. The PROMETHEE II net flow coefficient outranks the units in each sub-rank of this framework.

Effectiveness: The Pact for Life Initiative
The Pact for Life program (Programa Pacto pela Vida) [30,31] was Pernambuco's state policy for police repression and social prevention of crime and violence. The initiative, utilizing preventive and repressive actions, aims to reduce impunity of violent crime through the integration of the public security system, readjustment and modernization of processes, protocols, and routines directly related to the police operational capacity and the criminal justice system, consolidation and integration of information, dissemination and democratization of data, and strategic training of new police officers. Focusing on reducing homicides prior to any other criminal occurrence, the Pact for Life program was one of the most successful state-level projects in their first years of execution.
The program's goal is to reduce homicides by 12%. Pernambuco's municipalities are considered effective when they reach this coefficient. As crime is a stochastic concept which depends on many objective and subjective determinants which are out of the police department field of action, the interesting perspective in this program is trying to integrate the many sectors and public institutions (police courts, justice courts, the prison system, and the community) which are part of the solution. In this analysis, the effectiveness veto of 12% was used to improve DEA's discriminant power, supporting a more appropriate ranking of both efficient and inefficient police units in the state.

Efficiency: Conditional Frontier Analysis
The advances in nonparametric robust estimators [16][17][18], besides the benefit of not imposing a functional specification for the frontier projection, are also not affected by extreme observations and outlier data. This is due to the projection of partial frontiers of order-m or order-α which do not envelop all data points, being less affected by extreme data values [17]. In this particular application, a directional version of the FDH was provided, in which results are robust to outliers and extreme values.
Conditional frontier analysis [17,18], in its directional version [40,41], is part of the nonparametric robust estimators proposed to overcome some drawbacks in the traditional DEA measures for technical efficiency. The application of this methodology permits assessment of police performance by considering criminal data as exogenous variables (Z) that were not under the service units' control. Consider a particular DMU j from a set of m decision units using i = 1, 2, 3, …, n inputs to produce r = 1, 2, 3, …, s outputs. The directional efficiency is obtained by choosing a feasible direction g(x,y) ≥ 0 for contracting

Effectiveness: The Pact for Life Initiative
The Pact for Life program (Programa Pacto pela Vida) [30,31] was Pernambuco's state policy for police repression and social prevention of crime and violence. The initiative, utilizing preventive and repressive actions, aims to reduce impunity of violent crime through the integration of the public security system, readjustment and modernization of processes, protocols, and routines directly related to the police operational capacity and the criminal justice system, consolidation and integration of information, dissemination and democratization of data, and strategic training of new police officers. Focusing on reducing homicides prior to any other criminal occurrence, the Pact for Life program was one of the most successful state-level projects in their first years of execution.
The program's goal is to reduce homicides by 12%. Pernambuco's municipalities are considered effective when they reach this coefficient. As crime is a stochastic concept which depends on many objective and subjective determinants which are out of the police department field of action, the interesting perspective in this program is trying to integrate the many sectors and public institutions (police courts, justice courts, the prison system, and the community) which are part of the solution. In this analysis, the effectiveness veto of 12% was used to improve DEA's discriminant power, supporting a more appropriate ranking of both efficient and inefficient police units in the state.

Efficiency: Conditional Frontier Analysis
The advances in nonparametric robust estimators [16][17][18], besides the benefit of not imposing a functional specification for the frontier projection, are also not affected by extreme observations and outlier data. This is due to the projection of partial frontiers of order-m or order-α which do not envelop all data points, being less affected by extreme data values [17]. In this particular application, a directional version of the FDH was provided, in which results are robust to outliers and extreme values.
Conditional frontier analysis [17,18], in its directional version [40,41], is part of the nonparametric robust estimators proposed to overcome some drawbacks in the traditional DEA measures for technical efficiency. The application of this methodology permits assessment of police performance by considering criminal data as exogenous variables (Z) that were not under the service units' control. Consider a particular DMU j from a set of m decision units using i = 1, 2, 3, . . . , n inputs to produce r = 1, 2, 3, . . . , s outputs. The directional efficiency is obtained by choosing a feasible direction g (x,y) ≥ 0 for contracting resources (inputs) and/or expanding results (outputs) to reach the industry efficient frontier: The results for efficiency projections depend on the choice of the directional input vectors, which aim to contract the production resources, and directional output vectors, which aim to expand the products. Thus, the overall evaluation is sensitive to the opted directions. From an economic point of view, this is consistent with the optimization behavior of service units (maximizing results or reducing production costs). Nepomuceno et al. [42] offer a brief discussion on the choice of sustainable directions, and a review of methodologies for selecting directions in a non-parametric framework can be found in the work of Wang et al. [43]. For instance, directions can be defined exogenously, including subjective preferences or conditional techniques [42,44,45], or they can be defined endogenously, by optimization approaches [43,46,47]. While the former has the benefit of flexibility for decision-making, the latter has the adherence of statistical properties and theoretical support.
The choice for the data direction g (x,y) = (X,Y) or for the unit vector g(x,y) = (1, 1) are the most appropriate in many circumstances due to the simplicity of underlying aggregations. In this assessment, we opt for the unit vector. As this assessment can be considered a twostep methodology (first as an efficiency application, then prioritization), choosing another potential flexible or optimized direction would require double aggregations, and the results in the prioritization phase (for constructing the effective-efficient sub-ranks) would be biased. Exogenous factors can affect police production by changing the input/output configuration or by affecting the distribution of inefficiencies. Including potential exogenous determinants of efficiency is based on the premise that the joint probability distribution H (XY) = prob (X ≤ x, Y ≥ y) when conditional to Z = z can define an attainable production, such that x can produce y when Z = z. For the directional efficiency, the conditional directional distance function is defined as [35,36]: is the joint conditional probability that a given unit j with production (X, Y) dominates the unit under evaluation with production configuration (x, y). This is empirically developed using a non-parametric estimator that smooths the multiple exogenous factors Z in the neighborhood of z from a sample of q = 1, 2, 3, . . . , p observations based on an appropriate kernel estimation and choice for a bandwidth vector. Three models were developed to derive FDH directional efficiency measures which compose the decision criteria in the ranking of municipalities: a model for violent crimes clear-ups as output (CVLI-Crimes Violentos Letais e Intencionais), a model for street mugging as output (Trans-Assalto a Transeúnte), and the last one for carjacking as output (Veíc-Roubo de Veículos), all conditional to the occurrences, using sworn and administrative officers as input.

Outranking: PROMETHEE II Net Flow
According to Brans et al. [22], the PROMETHEE method offers a preference function for each decision criterion in the elicitation process and prioritizes multiple alternatives. The intensity of preference for an alternative x over another option y, P i (x,y), was developed through differences in performance levels on that criterion for the two alternatives f i(x) − f i(y) ∀ i = 1, 2, 3 . . . I decision criteria. This takes on values between 0 and 1. The decision-maker defines a standard function shape (usual, u-shape, level, linear, or Gaussian), specifying potential parameters according to their evaluation. The preference index P (x,y) is defined as a weighted average of preferences on the individual criteria: This weight elicitation represents a notion of global importance in the decision-maker's perspective instead of scaling factors (trade-offs). When the decision-maker cannot provide or is not willing to provide details on how critical each decision criteria is, an alternative is using Rank Order Centroid (ROC) weights [48] for evaluating criteria with partial information about attribute weights. In this methodology, ranking the least to the most important criteria in the decision-maker's perspective is sufficient to obtain the optimal weights to be considered in the pairwise comparisons for definitions of outranking flows. The weights for each criterion are determined by: where L is the total of criteria from w = k to L. After defining weights and potential threshold parameters (for preference and indifference comparisons), the PROMETHEE preference index provides a valued outranking relation for ordering alternatives. The positive outranking flow and the negative outranking flow (over all alternatives under consideration) are defined as follows [24]: The positive outranking flow for x: The positive outranking flow expresses the scope of how much x outranks all other alternatives. The negative outranking flow expresses the scope to which all other options outrank x. A complete order of alternatives was derived from the "net flow" for each alternative, defined as: Thus, an alternative x outranks y if Q(x) > Q(y), and it is indifferent when Q(x) = Q(y). This complete preorder is more disputable than partial orders derived from the individual positive and negative flows due to some information loss. Nevertheless, they are attractive in providing a DEA post-analysis non-compensatory ranking of DMUs.

Data, Application, and Discussion
The policing structure in Brazil is composed of four instances: the Military Police, the Civil Police, the Federal Police, and Federal Highway Police. This assessment regards the production of public security produced by the Civil Police, which aims at preventing, suppressing, and investigating crimes of all types, performing judicial investigative functions directed by police delegates.
Data regarding the number of police officers (input) and inquiries with the definition of responsibility (clear-ups) for three types of felonies (output), and the corresponding occurrences (violent crime, street mugging and carjack) in 145 of the 185 Pernambuco cities were provided by the Secretariat for Social Defense (SDS-PE). The criminal occurrences were the environmental factors conditioning the directional efficiency of the police departments. Adequacy of this data can find support in similar assessments of police efficiency [15]. Table 1 and Figure 1 summarize the main descriptive data information. Pernambuco had a total of 1430 sworn officers and administrative staff as inputs in this application. The high number of human resources comes from the city Cabo de Santo Agostinho, in the coastal region, which also reports higher occurrences for violent crime (198 incidents) and street mugging (2198) than other cities in this assessment. There are a smaller number of officers in Camocim de São Felix, in the countryside region. The clear-up rates for Pernambuco as a whole, according to Table 1, are 41.72% for violent crimes, 6.38% for street mugging, and about 2.93% for carjacking, with a global efficiency of 58.08% considering an input-oriented variable returns to scale application. Figures 2 and 3 illustrate this description in notched boxplots. There is strong evidence for median equality, as the confidence intervals represented by the notches overlaps those of violent crime and mugging. We adopted the usual standard function shape for prioritizing units using the PROMETHEE method. In this case, we do not consider indifference or preference thresholds during the elicitation process. Compensations were considered with the effectiveness veto threshold. ROC weights were used considering the four decision metrics from Nepomuceno et al. [5]: Pact for Life effectiveness in reducing 12% homicides (the most important criterion, w = 0.521), relative directional efficiency in violent crimes (CVLI_Rel.Ineff., the second most important criterion, w = 0.271), relative directional efficiency in carjacking (Veic_Rel.Ineff., the third most important criterion, w = 0.146), and relative directional efficiency in street mugging (Trans_Rel.Ineff., the least important criterion, w = 0.062).   Pernambuco had a total of 1430 sworn officers and administrative staff as inputs in this application. The high number of human resources comes from the city Cabo de Santo Agostinho, in the coastal region, which also reports higher occurrences for violent crime (198 incidents) and street mugging (2198) than other cities in this assessment. There are a smaller number of officers in Camocim de São Felix, in the countryside region. The clearup rates for Pernambuco as a whole, according to Table 1, are 41.72% for violent crimes, 6.38% for street mugging, and about 2.93% for carjacking, with a global efficiency of 58.08% considering an input-oriented variable returns to scale application. Figures 2 and 3 illustrate this description in notched boxplots. There is strong evidence for median equality, as the confidence intervals represented by the notches overlaps those of violent crime and mugging.   We adopted the usual standard function shape for prioritizing units using the PRO-METHEE method. In this case, we do not consider indifference or preference thresholds during the elicitation process. Compensations were considered with the effectiveness veto threshold. ROC weights were used considering the four decision metrics from Nepomuceno et al. [5]: Pact for Life effectiveness in reducing 12% homicides (the most important criterion, w = 0.521), relative directional efficiency in violent crimes (CVLI_Rel.Ineff., the second most important criterion, w = 0.271), relative directional efficiency in carjacking (Veic_Rel.Ineff., the third most important criterion, w = 0.146), and relative directional efficiency in street mugging (Trans_Rel.Ineff., the least important criterion, w = 0.062).    We adopted the usual standard function shape for prioritizing units using the PRO-METHEE method. In this case, we do not consider indifference or preference thresholds during the elicitation process. Compensations were considered with the effectiveness veto threshold. ROC weights were used considering the four decision metrics from Nepomuceno et al. [5]: Pact for Life effectiveness in reducing 12% homicides (the most important criterion, w = 0.521), relative directional efficiency in violent crimes (CVLI_Rel.Ineff., the second most important criterion, w = 0.271), relative directional efficiency in carjacking (Veic_Rel.Ineff., the third most important criterion, w = 0.146), and relative directional efficiency in street mugging (Trans_Rel.Ineff., the least important criterion, w = 0.062). Figure 4 illustrates the classification of the police units sensitive to weighting changes according to this ROC weights configuration (L = 4), equal weights (L = 4), ROC weights considering only measures for the police efficiency (L = 3), and scenarios comparisons.  The panels of Figure 4 report rankings that are sensitive to weighting definitions. When comparing a traditional efficiency-based ranking (L3) to a complete classification considering effectiveness, one can see some significant ranking reversals, such as Cumaru, Terezinha, and Joaquim Nabuco. Cumaru was the first in absolute efficiency and effectiveness (L4). It was located at the eleventh position in a ranking considering only efficiency (L3). Terezinha, the sixth in absolute efficiency and effectiveness (L4) was located at the 24th position considering only efficiency criteria (L3) (18 position change). Joaquim Nabuco, from the ninth position in efficiency and effectiveness (L4), was located in position 26 when only efficiency criteria were considered (L3). Several other ranking changes highlight the importance of assessing this scientific soundness, combining sustainable efficiency measures with effectiveness prospects. Table 2 provides some useful information comparing PROMETHEE ranking reversals sensitive to the three types of weighting definitions. The ranking of L4 for the effective units (both efficient and inefficient) was the benchmark for comparing the classification changes. The following Tables 3-5 report the overall policing effectiveness-efficiency application for a non-compensatory ranking of 145 of Pernambuco's municipalities described in the methodology. According to the results, none of the three completely efficient units (i.e., efficient in all three output models) were ineffective. For this reason, we have three subrankings instead of four, as illustrated in Figure 1. The tables provide information on the non-compensatory compared to the compensatory ranking position, i.e., when the municipalities were all outranked in the same group without the imposition of effectiveness or efficiency vetoes. The net flow parameter was used to outrank the units in each sub-rank.    The effectiveness was measured by how much the municipality reached the target of 12% reduction in homicides (more is preferable, but 0.12 is sufficient). The last column for the relative inefficiency aggregates each unit's relative inefficiency scores for all three models considering the slacks (less is preferable, and zero means the unit is efficient in all three models, with no slack for police officers). It was interesting how different the non-compensatory top-ranked municipalities would feature in a compensatory evaluation. Jucati, the first top-ranked municipality, was a small city in the agreste pernambucano (rural/wasteland region) of about 11,000 residents and a population density of 87.92 per km 2 . It had 4 officers as input along the year, 4 homicide occurrences (all solved), 12 street mugging (10 solved) and 15 carjackings (8 recovered). The municipality reduced from 9 homicides in 2015 to 6 homicides in 2016 (about 33% reduction) and from 6 homicides in 2016 to 4 homicides in 2017 (about 33% reduction).
When compared to the first effective but not efficient unit (Cumaru, Table 4) we can observe the compensation effect; as Cumaru, another small city in Pernambuco, could reduce the homicides entirely in the year of evaluation (from 2 to 0, 100% effectiveness, w = 0.5208333), this more than compensated a poor efficiency performance (25% relative inefficiency), locating this municipality at the second position in the compensatory ranking, and Jucati at the 18th position. Compensations of this nature can be observed all over the rankings. Non-compensatory/compensatory ranking inversions were even bigger for Saloá (2 compared to 37) and Camocim de São Félix (3 compared to 69). The non-compensatory ranking of units in this assessment tends to provide a fairer evaluation in line with what is expected by the policymaker. Effective and Efficient Units are shown in Table 3, Effective and Inefficient Units. in Table 4 and Ineffective and Inefficient Units in Table 5.

Conclusions
In many problems involving human resources' strategic management and their consequences, it is essential to consider ranking alternatives in a sustainable perspective. This paper shows that each particular ranking reflects specific values, preferences, and criteria for specific design purposes. We highlight the importance of a fair non-compensatory judgment on police units' performance using multiple decision criteria based on efficiency and effectiveness. The proposed methodology may be applied and extended to many other contexts, areas of economic activities, policies, and empirical scenarios to include potential environmental non-discretionary determinants of productivity, robust to extreme data and super-efficient frontier projections. Different types of felonies have different degrees of severity, complexity, and social importance. Traditional DEA measures for ranking technical efficiency fail to consider those clear-up specifications. The resulting rankings of police units are strongly affected by this limitation. In this work, we have combined recent advances in robust non-parametric frontier estimation with multiple criteria outranking to provide a more realistic and valuable public security ranking of municipalities. By including crime as an exogenous factor affecting the regular distribution of efficiencies in three police performance models, we derived more robust technical efficiency measures to be used with effectiveness goals as decision criteria for classifying the police units. Similar "efficient and effective" rankings may be obtained by applying this paper's methodology to assess other regions or other fields of analysis.
Additional analyses that could be interesting to carry out include considering intervals of efficiency or the inclusion of indifference and preference thresholds in the decision matrix based on an elicitation of the society's values reflected by a public authority. Intervals of efficiency, instead of an absolute measure of efficiency, could be an alternative for permitting service units to discount relatively small and irrelevant differences in the efficiency score that would drastically lead to ranking reversal because of eventual unconventional weighting structures. Efficiency-based models for re-allocating sworn officers and administrative police staff based on time-series data [49,50] and inverse frontier methods [51] might offer an additional post-analysis value added to identify efficient strategies to improve the current police productivity. Another interesting analysis would be to explore other multicriteria decision approaches [26,27] to check the robustness of the results obtained by applying the PROMETHEE II approach. All these analyses are left for future research.