Multi-Criteria Analysis in Compound Decision Processes : The AHP and the Architectural Competition for the Chamber of Deputies in Rome ( Italy )

In 1967, a national architectural competition was released for a preliminary project proposal, aimed at the realization of the new building for the Chamber of Deputies in Rome. The outcomes of that competition were unusual: eighteen projects were declared joint winners, and no winner was consequently selected. With reference to that event, this research aims to examine the usefulness of the evaluation tools that are currently employed and the positive effects that one of these techniques would have had, as support for the identification of the “winner” project, are highlighted. Therefore, an hypothetical examination/adjustment of the decision process of that competition through the Analytic Hierarchy Process (AHP) is developed, analyzing the outputs obtained by the implementations of this technique on the final decision. In addition to confirming the usefulness of the evaluation tools for compound and conflicting decision processes, the results of this experiment led to a further understanding of the socio-cultural dynamics related to the original outcomes of the competition analyzed.


Introduction
In 1967, a national architectural competition was released for the preliminary project proposal, finalized to the construction of the new building for the Chamber of Deputies in Rome.The competition's announcement asked for a project idea for an enlargement of "Palazzo Montecitorio", which is still the headquarters of the Chamber of Deputies, in order to realize the project on an adjacent area between Piazza Montecitorio, Via della Missione and Via Campo di Marzio (Figure 1).
The outcomes of that competition were unusual: eighteen joint winner projects were announced, and no winner was consequently selected.This event left a deep mark in the history of competitions for architectural projects of public works in Italy, preventing-as a matter of fact-any chance to intervene on the old town center of Rome over the 20th century.
In order to fulfill a general analysis of the state-of-the-art of planning in the old town centers in Italy, the study of the announcement in 1967 and its outcomes was a starting point for the considerations of this research.Firstly, the projects submitted to the competition in 1967 were analyzed.In consideration of the requirements of the competition, eleven of them were selected (Table 1), which particularly expressed the artistic and technical features of the ages of the competition.Through the implementation of the Analytic Hierarchy Process (AHP), a hypothetical examination/adjustment of the decision process of that competition on the eleven selected projects was developed, analyzing the effects of the use of this evaluation technique on the final decision.
Buildings 2017, 7, 38 2 of 17 competition.Through the implementation of the Analytic Hierarchy Process (AHP), a hypothetical examination/adjustment of the decision process of that competition on the eleven selected projects was developed, analyzing the effects of the use of this evaluation technique on the final decision.The competition for the enlargement of the headquarters of the Parliament had been released before the use of the evaluation tools for the selection of projects for public works, which has been spreading in Italy only from the 1980s [1]; this research does not aim to highlight the limitations of the announcement in 1967 and the decision process that followed.The objective is to study the usefulness of the evaluation tools that may support the decision making and the positive effects that the application of one of them would have had on the selection of the "winner" project.
Hence, as a preamble of this work, it is important to consider that: the research is a technical simulation that was developed regardless of the 50-year time-gap, on the analysis of materials collected in the archives and bibliographic sources, taking into account the in-depth scientific studies that have been improved during this long-time period;  the critical review of the outcomes of the competition does not aim to be an end in itself, but it is meant to play a positive role in the orientation, definition and selection of the projects for public  The competition for the enlargement of the headquarters of the Parliament had been released before the use of the evaluation tools for the selection of projects for public works, which has been spreading in Italy only from the 1980s [1]; this research does not aim to highlight the limitations of the announcement in 1967 and the decision process that followed.The objective is to study the usefulness of the evaluation tools that may support the decision making and the positive effects that the application of one of them would have had on the selection of the "winner" project.
Hence, as a preamble of this work, it is important to consider that: • the research is a technical simulation that was developed regardless of the 50-year time-gap, on the analysis of materials collected in the archives and bibliographic sources, taking into account the in-depth scientific studies that have been improved during this long-time period; • the critical review of the outcomes of the competition does not aim to be an end in itself, but it is meant to play a positive role in the orientation, definition and selection of the projects for public works in old town centers, particularly focusing on the establishment of the criteria for the evaluation of project proposals to be analyzed.
The paper continues with the procedure already used for the starting surveys, the establishment of the evaluation scheme and the operational proposal for the case study.In particular, Section 2 reports on the collection of the data related to the competition's announcement and a close examination of it and its outcomes.Section 3 outlines a general scheme for the technical selection of the most effective multi-criteria evaluation.Section 4 explains the AHP, i.e., the evaluation technique chosen for this research.Section 5 shows the application of the AHP to the case study and a commentary of the outcomes.

The Competition in 1967
The competition's announcement was published on 17 May 1966, providing a submission deadline in sixty days.However, the deadline was postponed to 31 January 1967, maybe because of some incongruities in the function charges, the degree of difficulty of the materials required and pressures from the political and cultural background of that period, and the final decision of the Jury was announced on 30 May 1967.
The analysis developed on the materials collected from the archive of the Chamber of Deputies show that this competition aimed at solving some issues, e.g., a lack of space and static complications of Palazzo Montecitorio, along with the need to accommodate the adjacent area that, in its indecorous conditions, was a real offence for the historic district of Rome.
Although several proposals for the renovation of this area had been put forward during the previous 20 years, the analysis of the requirements to include in the competition's announcement were almost contradictory.The archive materials showed some comparative charts with the headquarters of government institutions of other countries (i.e., the government palace of Jerusalem that was completed a few years earlier) were too different in their dimension and function features, as well as for their context compared with the Italian one.Furthermore, the project proposals had to undergo a very strict evaluation procedure: after a qualifying examination, the suitable projects would have been listed in a ranking and the winner would have been selected among them.In addition, a note of the competition's announcement specified that "joint prizes were not admitted" (art.11 of the competition's announcement), but this rule was paradoxically disregarded.
Sixty-four projects were submitted before the scheduled deadline, whereas sixty-two projects proved suitable after the preliminary examination according to the requirements specified in the competition's announcement, art.7 (materials required) and arts. 1 and 2 (general rules regarding dimensions and functions).Subsequently, the General Commission, appointed by the Ministry of Cultural Heritage that was in charge at that time, analyzed and assessed the projects in different moments during a 750 m exhibition.
The records of the Committee, analyzed from the archive consulted, show that only eleven "more deserving" projects passed the preliminary examination.The others were discarded because they were judged as less deserving and characterized by deficiencies that did not allow them to pass the minimum suitability standards set by the competition's announcement.Subsequently, for some reasons that are not specified in the records, fourteen projects were re-admitted and, later, nine additional projects were stated "suitable subject to further examination".In the end, thirty-four proposals were admitted to the final evaluation.
The thirty-four finalist projects were analyzed by three sub-committees, branches of the General Committee, specifically appointed to evaluate some particular aspects: the "urban-architectural" committee, the "functionality-relevance" committee and the "static-technological" committee.The three sub-committees expressed their assessments according to three evaluation levels (A = Excellent, B = Good, C = Sufficient), but giving different scores to each one (Table 2) in relation to the importance attributed to the particular aspects considered by each sub-committee.Table 2 highlights that the urban-architectural committee had more influence than the other two, of equal relevance between them.According to their marks, twenty-one projects were selected as "more deserving", classed in three categories (first group: five projects; second group: thirteen projects; third group: three projects).
After a final review, carried out jointly by the three sub-committees, eighteen projects of the first two groups were awarded as joint-winners.As a matter of fact, this decision meant a non-selection of a winner project.

Multi-Criteria Analysis
This research has started after the analysis of the results related to the architectural competition in 1967, identifying the most important steps of the decision-making process used at that time, retracing the same steps and reframing them with one of the most applied decision-making tool-the AHP-to test its usefulness for the 1967 competition.
According to Regulation 2004/18/EC of the European Parliament, as regards the coordination of procedures for the award of contracts for public works (art.46), all Member States are required to structure tenders with objective parameters, in order to ensure fair and transparent comparison of different offers.However, no practical guidelines for coding and simplifying decision-making techniques of Member States are given.
In Italy, following the European rules, the Implementary Regulation (Presidential Decree of 5 October 2010, No. 207, Annex G) of the Legislative Decree No. 163/2006 offers a list of techniques that can be used to evaluate the most economically advantageous tender.However, there is a lack of information about methods related to several practical cases and how to implement them.For these reasons, the Implementary Regulation and its guidelines have been analyzed, focusing on the strategic role of the multi-criteria analysis and the choice of the most suitable method for the 1967 call.
Multi-criteria techniques identify a sector of the Decision Analysis (DA) that investigates methods and tools aimed at the simplification, starting from different criteria, of the decisional process related to the purpose of the choice, the classification, or the order of the options [2,3].Nowadays, the most used DA models are based on tools to support choices with multiple criteria: Multiple-Criteria Decision Analysis (MCDA) or Multiple-Criteria Decision Making (MCDM).
These techniques overcame the classical methods, considered too strict with their linear optimization for single criterion and unsuitable for complex decisions.
Since the late 1960s, many MCDM methods have been proposed [4,5], in order to measure and prioritize the tangible and the intangible criteria to be used in the choice of the best solution among the available alternatives in relation to the assessment problem.
Multi-criteria analyses are by definition interdisciplinary tools, so it is not possible to classify them within definitive categories.The reference literature offers several taxonomies based on different factors [6][7][8][9].
Over time, many of these methods have been further deepened and developed by the researchers.Therefore, the choice of the MCDM method can also be configured as a decision-making problem that needs to be answered [10,11].
Although the variety of multi-criteria techniques can be considered a strength, it is necessary to pay attention in choosing the evaluation method: every tool has technical limits that could compromise the final decision [12].

Analytic Hierarchy Process
The paper focuses on the choice among design alternatives that need to satisfy several criteria.Therefore, regarding the project options:

•
it is not possible to define objective parameters in order to set specific functions because the typology of call for ideas is a way to collect general guidelines about technical-functional and economic details; • it is not possible to create indifference threshold related to qualitative values of the architectural and urban elements; • the global evaluation needs to be a synthesis of all partial values collected through several criteria, and, at the same time, there is a need to understand how a single criterion influences the final decision, avoiding aberration for the goal.
Considering the premises, the AHP, developed by Thomas Saaty [13], resulted as the most suitable method for this research.
The application of the AHP allows to decompose the decisional problem, structuring hierarchic levels that allow to deal with the decision, reducing its complexity.Through the composition of a tree diagram, the general problem is divided in always smaller sets of the hierarchical structure, and the local priority between them is defined by pairwise comparisons.The minimum number of compulsory components of the hierarchy for structuring the AHP model is three: the goal, the alternatives to reach the goal, and the criteria-the level in the middle-to evaluate how the alternatives can satisfy the goal for each specific criterion.
Using the psycho-mathematical principle of pairwise comparison, every criterion is compared with the others to define the priority related to the element of the upper level that is used as reference for the comparison [14].In this way, it is possible to avoid the evaluation of the whole group of criteria/alternatives comparing them in couple.The comparison can be done using verbal expressions that are familiar also for a non-expert decision-maker.There are several technical theories for the translation of these verbal values into numbers [15]; however, the most used one is the Saaty's semantic scale [16] (Table 3).Pairwise comparisons and their related values are used to fill matrixes that allow the calculation of the global priority.This matrixes are squared, invertible, with order n (n represents the number of criteria to be compared), and all the values of the comparison are positive.
The AHP is extensively used in numerous disciplines [4,5,17] and for various applications [18][19][20][21], but it is rarely applied in the field that in the present paper has been dealt with: in fact, the target of the research concerns the investigation of the aspects for which the AHP can be an appropriate evaluation tool for choosing the best solution as regards the expansion of a public building characterized by high symbolic values (i.e., the Chamber of Deputies) and located in a historical and consolidated urban area (i.e., the historic center of Rome).The AHP allows for choosing the most suitable alternative based on quantitative and qualitative evaluations, obtaining a multi-criteria ranking for reaching the goal [22].Furthermore, the process can be improved and supported by consistency analysis on the evaluations expressed by the decision-maker or the group of decision-makers and the sensitivity analysis allows to check the influence of each criteria on final decision.
The AHP fundamental steps are: (1) structuring the hierarchy and its levels; (2) structuring of the pairwise comparison matrix for criteria and the definition of their weights; (3) structuring of the pairwise comparison matrix for alternatives related to criteria; (4) aggregation of the values and synthesis for the definition of the best alternative.
In the present paper, for the structuring of the pairwise comparison matrix (step (3)) will be applied: (i) for quantitative criteria analysis, a full aggregation approach; (ii) for qualitative criteria analysis, to solve the problem of a large number of pairwise evaluations, the traditional AHP procedure has been implemented according to the cluster approach and pivots approach [23] as synthesized by Ishizaka and Nemery [5].

The Case Study
All phases of AHP are developed and referenced to the case of the 1967 competition.

Decisional Hierarchy Definition
Having chosen AHP among different multi-criteria techniques, the first step has been to reconstruct the decisional process according to the most recent European guidelines [24,25], in order to find macro-categories of criteria to evaluate qualitatively and quantitatively all the different project proposals.
All of those data and operative suggestions collected through archive and bibliographic analysis, converged in a Strengths, Weaknesses, Opportunities, and Threats analysis-SWOT analysis (Table 4)-in order to define the strengths and the weaknesses of the '67 brief and, at the same time, to discuss risks and opportunities as references to create a new generic brief document.The SWOT analysis allows to synthetically describe the main characteristics of the initiative and of the context in which it is realized and to analyze alternative scenarios.Through the SWOT analysis, it is possible to examine the context in which the investment is located and to highlight the internal factors to be reached (forces) or to be compensated (weaknesses), as well as the favorable (opportunities) or unfavorable (threats) external factors [26].Even without an analytic aim, the SWOT analysis is an essential instrument to specify roles of criteria [26,27].In fact, with the definition of further sub-criteria, it is possible to limit the operative doubts that could emerge in the subsequent stages of technical simulation, since it is necessary to deal with qualitative aspects that might be influenced by individual perception.The next step concerns the structuring of elements in the AHP, setting up the hierarchy of goal-criteria-alternatives and proceeding with the decisional process.Step V-Critical analysis following the publication of the results . great contribution to design reflection .the eighteen first place "ex aequo" .the failure of the public competition .pursuing the same projectual goal through the support of the decision analysis Step VI-Analysis of the subsequent actions to the 1967 call .sterility of temporary solutions .very onerous leases to occupy the surrounding properties . estimated expenditures in proportion to the cost of the surrounding local rents .risks of archeological preexistences .waste of public resources for functional solutions that are unmanageable

Structuring of Hierarchy
The application of the AHP needs the definition of priorities for all the inferior elements in reference to higher elements which have effects on evaluation results [28].Therefore, in Figure 2, elements are schematized: Goal (level 1): definition of the best design solution to ensure a coherent integration with the surrounding area; Criteria (level 2), with relative sub-criteria: Alternatives (level 3), the 11 selected projects (see Table 1).
The sub-criteria in the present case have been considered only to allow the better interpretation of the criteria [29], but they have been not considered when making comparisons.In fact, once the guidelines have been defined, pairwise comparisons have been made only between five main criteria.
Concerning the specific order in composing matrixes, it is preferable to begin with comparisons between criteria to define their weights and, at a later time, with comparisons between alternatives [30].
In order to simulate ex novo the decisional process, it was necessary to build a hypothetical Commission.Structured for this AHP model application, the Commission has been composed as experts by 12 designers and four design professors.These experts have been interviewed (i) to assign the weight distribution to the criteria; and (ii) to develop comparisons among different project proposals related to several criteria.Alternatives (level 3), the 11 selected projects (see Table 1).The sub-criteria in the present case have been considered only to allow the better interpretation of the criteria [29], but they have been not considered when making comparisons.In fact, once the guidelines have been defined, pairwise comparisons have been made only between five main criteria.

Goal
Concerning the specific order in composing matrixes, it is preferable to begin with comparisons between criteria to define their weights and, at a later time, with comparisons between alternatives [30].
In order to simulate ex novo the decisional process, it was necessary to build a hypothetical Commission.Structured for this AHP model application, the Commission has been composed as experts by 12 designers and four design professors.These experts have been interviewed (i) to assign the weight distribution to the criteria; and (ii) to develop comparisons among different project proposals related to several criteria.
The weights for the criteria-considering they were only five-have been assigned throughout an open debate.The weights for the project proposals-considering they were 11-have been defined with two different approaches: (i) for quantitative criteria (1.Relevance to the rules; 5. Economic convenience), alternatives have been ordered collecting numerical values with a process of data check.The weights for the criteria-considering they were only five-have been assigned throughout an open debate.The weights for the project proposals-considering they were 11-have been defined with two different approaches: (i) for quantitative criteria (1.Relevance to the rules; 5. Economic convenience), alternatives have been ordered collecting numerical values with a process of data check.(ii) for each of the qualitative criteria (2.Technical feasibility; 3. Architectural quality; 4. Urban Strategy), the alternatives have been pre-ordered using a geometrical mean of votes given by each member of the decision group.Thereafter, the alternatives have been divided into two smaller groups and weighted again throughout an open debate.

Construction of the Pairwise Comparison Matrix for the Definition of the Relative Weights
The ranking of the criteria in relation to the goal has been developed by answering the following question: "which, and by how much, criterion is more important to choose among alternatives?" The matrix of criteria is made up of values a ij (related to the i-th row and the j-th column), which define the relative importance of the i-th criterion over the j-th criterion.
The preference matrix can represent and analyze comparisons' results, based on the semantic scale by Saaty [31], which relates verbal expressions with numeric point-based values (from 1 to 9 in both directions based on the value of preference on one or the other criteria in every comparison, Figure 3), before creating the criteria matrix.
The latter represents the classic way of synthesizing comparisons among the criteria and allows to define the classification and the absolute weight of each criterion.
The influence analysis of each criterion based on the fulfillment of the goal makes clear that, since the theme was a competition for architectural ideas, the criteria characterized by the highest weight were "architectural quality" and "urban strategy".
There are several methodologies for the definition of the local weights of the criteria: the approximate method, the eigenvalues method and the geometric mean method.
In the case study presented, the eigenvalues method has been chosen, which allows for better management of problems connected with inconsistent matrices.
In case of inconsistencies, in order to define the value through which transitivity cannot be applied, the Consistency Index (CI) is used.The Consistency Ratio (CR) allows instead to determine the limits of acceptability of those values.
The influence analysis of each criterion based on the fulfillment of the goal makes clear that, since the theme was a competition for architectural ideas, the criteria characterized by the highest weight were "architectural quality" and "urban strategy".
There are several methodologies for the definition of the local weights of the criteria: the approximate method, the eigenvalues method and the geometric mean method.
In the case study presented, the eigenvalues method has been chosen, which allows for better management of problems connected with inconsistent matrices.
In case of inconsistencies, in order to define the value through which transitivity cannot be applied, the Consistency Index (CI) is used.The Consistency Ratio (CR) allows instead to determine the limits of acceptability of those values.

Construction of Evaluation Matrices of Alternatives in Relation with the Criteria
The construction of the evaluation matrices has been arranged following two different approaches: (i) a full aggregation approach for the quantitative criteria; (ii) the AHP traditional approach.
For the quantitative criteria, a numeral data check has been utilized; for the qualitative criteria, a double comparison between alternatives has been used.In order to avoid burdening the writing, the operating procedure in attributing a weight to every single criterion using a quantitative one (1.Relevance to the rules) and a qualitative one (3.Architectural quality) is explained.
For the qualitative criteria, the local arrangement of alternatives is connected with the question: "which of the two alternatives and how much better satisfied the specific criterion?"For the arrangement of the alternatives in relation with the criteria, the procedure and the rules to be followed are already defined with those to compare the criteria themselves.

Construction of Evaluation Matrices of Alternatives in Relation with the Criteria
The construction of the evaluation matrices has been arranged following two different approaches: (i) a full aggregation approach for the quantitative criteria; (ii) the AHP traditional approach.
For the quantitative criteria, a numeral data check has been utilized; for the qualitative criteria, a double comparison between alternatives has been used.In order to avoid burdening the writing, the operating procedure in attributing a weight to every single criterion using a quantitative one (1.Relevance to the rules) and a qualitative one (3.Architectural quality) is explained.
For the qualitative criteria, the local arrangement of alternatives is connected with the question: "which of the two alternatives and how much better satisfied the specific criterion?"For the arrangement of the alternatives in relation with the criteria, the procedure and the rules to be followed are already defined with those to compare the criteria themselves.
Taking into account several alternatives and having defined B k as the generic matrix of pairwise comparisons among the alternatives, there will be a number of alternatives matrix as the rank of the A matrix (which is five), one for each criterion.In this way, the vector of priority p can be calculated for each of the matrices B k , with k [1 ÷ n]: After the setting of matrix, there will be: the vector of weights r, made up of n elements r j , n vectors of priorities p k , each of them composed by m elements p kj .
Analyzing "criterion 1. Relevance to the rules", the original data collected from the 1967 Committee-regarding the materials requested in articles for general prescription (arts.1, 2, 7) and specific ones (art.4) of the original call-have been transformed in numerical values.In Table 5, considering the example of art.7, the submitted materials have been reported (with y = yes; n = no) and have been transformed in numbers (from 1 for all the materials submitted to 0 for missing materials) in Table 6.Successively, values for art.7 have been added to the ones collected for the other articles and ordered in Table 7.
The comparison of the alternatives for the qualitative criteria have been developed through the "classical" pairwise comparison.In this paragraph, an example is illustrated, related to the results of the pairwise comparison for the "criterion 3. Architectural Quality", specifying how the problem of a large number of alternatives has been faced.In Table 9, the priority values of the first cluster have been reported.Then, a pairwise comparison has been made for the second cluster starting from the Pierino e il lupo project.At the end of the process, the priority values of the second cluster have been also reported in Table 9 and proportioned to the ones of the first cluster, obtaining the final values for "criterion 3. Architectural Quality".The same process has been applied for the other qualitative criteria.The last step of the AHP involves the synthesis of the results.Using the "classic" additive approach, it is possible to obtain the global priority (P k ) of the k-th alternative as the sum of the products of its local priorities p k,i with respect to i-th criterion for r i weight of the i-th criterion: In this way, it is possible to sort out the alternatives to reach the goal.
Regarding the case study, the final results obtained from the AHP application for choosing the best project are useful to better understand the socio-cultural dynamics of the original results (Figure 4).
With the AHP application, the WW project resulted in being the most relevant to the Goal, considering the same result gained by the WW project within the eighteen best projects, it is possible to recognize the work done by the Committee in the previous contest.
In that sense, the hypothetic use of the AHP in the 1967 evaluation could have ensured a greater objectivity in the process of assessing the best project and a significant limitation of subsequent technical doubts.
The Martedì and the Buleuterion projects, which were among the nine re-proposed projects, represent the second and third best alternative, therefore confirming the repêchage decision.
The sensitivity analysis helped interpret the results related to the Campo Marzio, Curia Innocenziana and Fontana Grande projects: despite being the best for "criterion 2. Technical Feasibility", these projects suffered a "downgrading" in the final standings for the scores obtained with respect to "criterion 3. Architectural Quality" and "criterion 4. Urban Strategy".Similarly, the Mac3 project, although ranking among the five "first-class" projects in 1967, results as the last one in the simulation with the AHP, resulting in being not valid enough with respect to all the criteria.
However, it should be noted that, although it is a call for proposals, given the importance of the function to be designed and its location in the heart of Rome, more defined and calibrated rules should have been contemplated according to the specific urban, historical and architectural context.Economic guidelines would have allowed for determining the financial feasibility of the proposed technical solutions.

Conclusions
The achieved results are in line with the objective of the paper: it was shown that a greater attention to the definition of the features that characterize the quality of the project in the decision-making steps leading to the identification of the best solution would bring technical and cultural benefits, capable of guaranteeing more solid choices also in the recovery process of degraded urban areas in the presence of high historical-artistic value buildings neighboring to the said site [32][33][34].
In summary, the conditions of degradation and neglect in which the project area still pours, adjacent to one of the most important institutional offices of the Italian State, shows, more than anything else, the importance and urgency to define (ex ante) and follow (ex post), in the process of drawing up and evaluating projects, consistent and adapted criteria to the Goal to be pursued, as well as it seems necessary to use adequate tools to support decisions in order to evaluate alternative proposals for achieving the Goal [35][36][37][38].
What has emerged from the research leads to reflection on the need to rationalize the current modus operandi in the public works sector, particularly for those jobs that are responsible for the realization of architectural works.The starting point can only be the development of a call for proposals,

Figure 1 .
Figure 1.Aerial view of Montecitorio with the project area marked in red.

Figure 1 .
Figure 1.Aerial view of Montecitorio with the project area marked in red.

Figure 3 .
Figure 3. Matrix of pairwise comparison of the criteria.
Figure 4. Final results

Table 1 .
Selected projects in the architectural competition.

Table 1 .
Selected projects in the architectural competition.

Table 2 .
Marks according to the evaluation levels of the sub-committees.

of the Available Information In Relation to 1967 Call In Relation to the General Composition of the Calls
Matrix of pairwise comparison of the criteria.

Table 6 .
Example of the quantitative criterion: "Relevance to the rules (art.7)"-translation in numbers of data of 1967 committee.

Table 8 .
Example of the qualitative criterion "architectural quality"-heuristic method to pre-order the alternatives.

Table 9 .
Example of the qualitative criterion "architectural quality"-clusters and pivot element.