A Hybrid Analytic Hierarchy Process and Likert Scale Approach for the Quality Assessment of Medical Education Programs

Abstract

The quality assessment of training courses is of utmost importance in the medical education field to improve the quality of the training. This work proposes a hybrid multicriteria decision-making approach based on two methodologies, a Likert scale (LS) and the analytic hierarchy process (AHP), for the quality assessment of medical education programs. On one hand, the qualitative LS method was adopted to estimate the degree of consensus on specific topics; on the other hand, the quantitative AHP technique was employed to prioritize parameters involved in complex decision-making problems. The approach was validated in a real scenario for evaluating healthcare training activities carried out at the Centre of Biotechnology of the National Hospital A.O.R.N. “A. Cardarelli” of Naples (Italy). The rational combination of the two methodologies proved to be a promising decision-making tool for decision makers to identify those aspects of a medical education program characterized by a lower user satisfaction degree (revealed by the LS) and a higher priority degree (revealed by the AHP), potentially suggesting strategies to increase the quality of the service provided and to reduce the waste of resources. The results show how this hybrid approach can provide decision makers with helpful information to select the most important characteristics of the delivered education program and to possibly improve the weakest ones, thus enhancing the whole quality of the training courses.

1. Introduction

It is known that, in recent decades, in all the major countries of the world, a high level of knowledge relating to theory, practice and communication in the healthcare field is expected. This means that training courses are provided for each specialist medical area and for all healthcare professions. Moreover, in health environments too, services’ quality and cost saving are becoming increasingly important. It is quite difficult to precisely define the quality concept, and the difficulty increases in health services. In the literature, different qualities’ features are faced, which are intercorrelated and can be treated as converging to the definition of a quality triangle (shown in Figure 1).

In any case, basically, we can say that “quality” indicates the degree of correspondence between the customer’s wishes and their fulfillment with the product or service offered. Obviously, to assess this correspondence, an evaluation tool is necessary.

The aim of educational evaluation concerns the analysis, the interpretation and the judgment of all the main characteristics of a training course for both direct recipients, i.e., the learners who attend the courses, and the educators, lecturers, teachers and organizers. Indeed, the course assessment aims to measure and monitor those key variables representing the efficiency, the effectiveness and the overall quality of education programs.

In the literature [1,2], the distinction between training evaluation’s target areas in terms of its design and content as well as its effects on and result for the organization’s and learner’s improvement is provided. A relationship of cause–effect can be identified among target areas, in which the learner’s acceptance could affect the organizational work.

The popularity rating can be considered a particular method of a user satisfaction survey to assess the internal efficiency of the intervention, the quality of training and the effectiveness of the teaching/learning path. The approval correlates with the individual perception of the experience (perceived quality) and, therefore, relies on qualitative methods and tools. Popularity can be monitored during the evaluation process or after the conclusion of the course. It should be noted that the above-mentioned assessment provides information at a low cost, and it is a useful key for more complex assessments. It is the “oldest” evaluation type, as it coincides with the birth of the education and training systems. Indeed, a very important variable in the evaluation of training effectiveness is the goals–results match of the entire training course or parts of it (modules, units, etc.) [3]. This is closely linked to the type of skills learned: basic, cognitive, professional, technical, instrumental, etc. Learning objects and training course appending affect the choice of methods and tools to use, mainly concerning the actual working modalities, i.e., checking whether the theoretical issues learned during the training courses are transferred to work situations—in other words, how far the skills, abilities and knowledge are useful to improve job performance—and the assessment of the organizational changes caused by the training process.

As mentioned, researchers are focused on the assessment and analysis of service quality for properly managing and organizing health resources due to the continuous increase in service demand in the medical education and clinical contexts, leading to a continuous update of physicians’ knowledge (also called continuing medical education (CME) [1]).

In this context, this paper, which is the extension of a previous study presented as a conference paper [4], aimed to describe a novel methodology to assess and monitor the quality of the medical training program offered by the Centre of Biotechnology of the National Hospital A.O.R.N. “A. Cardarelli” of Naples. The goal of this study was to propose a hybrid approach to verify whether the needs expressed by the learners correspond to the actual service priorities, i.e., those services needing improvements. In particular, the services’ quality was evaluated through the measurement of customer satisfaction and the prioritization of the interventions which, in turn, were evaluated combining two approaches, namely, a Likert scale (LS) and the analytic hierarchy process (AHP). Both methods have been widely used in the literature to develop and maintain a relevant plan for ongoing improvements in service quality. We adopted and combined these two techniques and show that, despite the fact both the AHP and LSs are widely used and recognized methodologies in several fields, if they are used separately, they can provide misleading conclusions, possibly compromising the decision-making process. Instead, if we combine them, we can provide decision makers with a wise trade-off between the voice of the customer (i.e., the needs expressed by the users, learners) and the voice of the process (i.e., those services that need to be improved). This could suggest better decisions regarding the prioritization of the improvement actions and a better organization of resources.

As far as the AHP is concerned, it is used across various fields to plan, to select the best alternatives, for resource allocations and for resolving conflicts and optimization [5], and it offers an efficient evaluation of training programs [6]. The AHP was developed by Saaty [7] as a multicriteria decision-making method to deal with complex problems with multiple subjective criteria.

The AHP performs pairwise comparisons to derive the relative importance of some predetermined items and assess alternatives to make the best decision. The prioritization mechanism is accomplished by assigning a number from a comparison scale developed by Saaty (Saaty’s scale) [8].

The AHP, as well as other tools based on multi-attribute utility functions such as the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), the analytic network process (ANP) and the Simple Multi-Attribute Rating Technique (SMART), has been applied in several contexts [9,10,11,12] including the fields of energy renewal [13,14,15], education [9,16,17], urban areas and mobility [18,19,20] and, more recently, health technology assessment [21,22,23,24,25,26,27] and quality improvement in the healthcare sector [11,28,29,30,31]. From the basic AHP principles, many other hybrid approaches for multicriteria decision making have been developed such as the fuzzy AHP [32,33,34,35], aimed at handling imprecise criteria with the use of fuzzy logic [34,36], the Interval Rough AHP–Multi-Attributive Border Approximation Area Comparison (IR-AHP-MABAC) [37], aimed at treating uncertainties in group multicriteria decision-making problems, and other hybrid approaches combining the AHP with other methodologies such as AHP-fuzzy TOPSIS, fuzzy AHP-PROMETHEE or AHP-Simple Additive Weighting [38,39,40]. In particular, the traditional AHP is among the most widely used approaches, and it has proved to be particularly suitable for cases with a limited number of criteria and in healthcare applications [41].

On the other hand, an LS is a psychometric scale widely used to evaluate questionnaires, e.g., it is employed for evaluating the satisfaction degree of users through the measurement of their opinions [29,42,43]. Questionnaires which apply the LS formula permit a choice based on an n-point scale. Participants are asked to express their judgment on an even-numbered scale, while an odd-numbered scale involves a score corresponding to indecision or neutrality. This method is reliable for obtaining measurements of training effectiveness and overall training impact at work [44]. However, evaluations carried out using an LS also have some disadvantages, such as biasness displayed by respondents in answering the questions honestly, the reproducibility of results and the validity of the results. Compared to other scales, such as the Saaty scale usually employed in the AHP method, an LS collects the absolute level of agreement/disagreement (generally on a five-point or seven-point scale) with respect to a finite number of items related to a specific object under examination [45]. Despite the fact it is a numeric scale, it is highly subjective, and it is used to represent the attitude of a person with respect to an object. On the other hand, Saaty’s scale introduces some objective elements in the evaluation through pairwise comparison, which collects relative measurements of the agreement/disagreement level with respect to an object by comparing it with another object. In this way, the Saaty scale aims to overcome the limitation in the use of absolute scales for psychometric assessment by replacing them with relative measurements by pairwise comparisons [46]. Despite this, an LS is helpful to highlight the positive/negative attitude of the user with respect to an item, and it is particularly able to assess the absolute satisfaction degree towards some aspects, in our case concerning the training courses.

In order to better analyze a generic training service, Grönroos (1988) [47] proposed the following specific criteria: professionalism and competence, attitude and behavior, accessibility and flexibility, ability to repair an error, reputation and credibility of the operator, reliability and trust. Parasuraman et al. (1985) [48] classified measurable variables in order to standardize the dimensions of perceived quality: tangible aspects, reliability, responsiveness, competence, courtesy, credibility, safety, accessibility, communication, understanding. In this work, dimensions defined based on those proposed by Grönroos (1988) and Parasuraman et al. (1985) were used together with numerical scales, in accordance with Saaty’s works [3,49,50], to quantify and compare the obtained results. This study extends our previous conference contribution [4] by widening the methodological aspects of the work and clarifying how the two proposed techniques (AHP and LS) are combined together. More specifically, an LS was used since it provides a well-known and versatile tool to assess user satisfaction, and, in our study, it was used to represent the degree of satisfaction of the learners with respect to the training courses; meanwhile, the AHP, among other multicriteria decision-making methods, appears to be more often applied to healthcare-related problems, also with some applications to medical education, and, in our study, we used it in combination with the LS to prioritize the improvement actions to enhance the quality of the training courses provided. Thus, in this manuscript, the hybrid and generalizable approach is described in depth from both a theoretical and practical perspective, the proposed novel workflow is presented and each step of the procedure is detailed. In addition, further results obtained by applying this hybrid approach to the real scenario are presented and widely discussed, and an extensive interpretation of the achievements is provided and discussed in view of the impact that the generalization of this approach could have on the medical training field and, more generally, in the quality assessment of education programs.

2. Materials and Methods

2.1. Proposed Methodology

For carrying out a customer satisfaction survey, the main steps to be followed are:

• Selection of the analysis dimensions;
• Sample choice;
• Questionnaire development;
• Questionnaire submission;
• Data processing.

The first step was introduced in the previous section. Obviously, the sample was chosen among people attending courses; in particular, the questionnaires were administered to a group of 110 students who attended training courses at the “Centre of Biotechnology of the National Hospital A.O.R.N. “A. Cardarelli” in the period from September 2013 to February 2014. The other steps will be described in the following.

However, the novel contribution of the proposed methodology is that the data processing is based on the combination of the AHP and LS tools and comparison of the obtained results to plan, if necessary, corrective actions.

Let us very briefly recall that the AHP is a multicriteria decision-making tool. According to the AHP, the general objective to be met is defined first, followed by the criteria to achieve this objective, the possible sub-criteria into which the criteria can possibly be broken down, etc., up to the lowest level of the hierarchy, or alternatives that must be prioritized (see Figure 2).

In particular, it comprises questionnaires for comparison of each element with the others and employs the geometric mean to arrive at the final solution, i.e., a priority list. The geometric AHP method has been largely used since it proved to be more efficient than the arithmetic method. From a mathematical point of view, indeed, the geometric mean is the only one that maintains the reciprocity of the matrix, i.e., preserves the matrix consistency [51,52,53,54].

On the other side, questions were designed using the LS forecast scores which represent agreement or disagreement (see, for example,

Table 1. Scores and meaning assigned to the items according to a 5-point Likert scale.

Score	Description
5	Completely agree
4	Agree
3	I do not know
2	Disagree
1	Completely disagree

).

This allowed us to numerically quantify, for each main need, the related satisfaction levels of the learners. The logical flow chart of the proposed methodological approach is reported in Figure 3.

As shown in Figure 3, the method adopted for the analysis of the perceived quality was the use of questionnaires; of course, the items defined in the questionnaires are related to each other.

2.2. Application of the Approach to a Case Study: AHP Approach

In accordance with the AHP model, firstly, the survey target was defined; the goal was to assess and analyze the service quality of the training courses provided by the Centre of Biotechnology of the National Hospital A.O.R.N. “A. Cardarelli” of Naples. Then, according to the dimensions identified by Grönroos (1988) [47] and Parasuraman et al. (1985) [48], three main dimensions of the service, representative of the training, and three corresponding sub-dimensions, which together make up the so-called dominance hierarchy, were identified, as reported in

Table 2. Definition of the hierarchy of dominance according to the AHP.

Hierarchy Level	Description
Objective level	Quality assessment of medical training courses
Criteria level	Structure			Teacher			Organization
Sub-criteria level	Location	Equipment	Comfort	Preparation	Interpersonal skills	Effectiveness	Cost	Course duration	Course content

.

In particular, structure, teacher and organization were chosen as the main dimensions (criteria). The following sub-dimensions (sub-criteria) are associated with the “structure” dimension:

• Location: the strategic location of the building with respect to the hospital structure and to the transport facilities nearby;
• Equipment: presence and suitability of the equipment provided with the training courses;
• Comfort: characteristics making the environment pleasant and comfortable.

The following sub-dimensions (sub-criteria) are associated with the “teacher” dimension:

• Preparation: adequacy of the level of training and preparation of the teacher;
• Interpersonal skills: ability to relate to learners by interpreting their requests and providing clear and comprehensive answers;
• Effectiveness: ability to deal with topics aimed at achieving the objectives of the course.

Finally, “organization” is the criterion that includes the general aspects that characterize the entire training process, which can be better defined through the following sub-criteria:

• Cost: costs sustained for the course;
• Duration: number of hours of the course;
• Content: topics and subjects of the course.

2.3. Planning of the LS Methodology

Before developing LS questionnaires, some criteria have to be followed for correctly establishing the items, as shown in the following list:

• Each item must be formulated in such a way that people with opposite attitudes give different responses;
• It is often helpful to present statements in an impersonal form;
• Statements must be concise and formulated with a simple language;
• Double negation sentences must be avoided;
• The items must be formulated half with a favorable attitude to the object and half with an unfavorable attitude.

Based on the above-listed criteria, a questionnaire composed of two parts was realized [4].

2.4. Data processing Using AHP

As mentioned, the AHP method involves the calculation of matrices of pairwise comparisons. In order to determine the matrices’ coefficients, a 9-point semantic Saaty scale [3] was employed to quantify qualitative judgments.

Based on the 9-point LS questionnaire, a comparison table was obtained for each couple of parent criteria (teacher–organization; teacher–structure; organization–structure).

A weighted geometric mean method (WGMM) was adopted to cluster the judgments as in the following equation:

$${{\displaystyle \prod }}_{k=1}^N{\left(a_k\right)}^{{\beta }_k},$$

(1)

which can be re-formulated by applying a constant weight equal to 1/N as

$${{\displaystyle \prod }}_{k=1}^N{\left(a_k\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N$}\right.}=\sqrt[N]{{{\displaystyle \prod }}_{k=1}^N{a}_k},$$

(2)

then, applying the logarithm, the computation is simplified in a sum:

$${10}^{\frac{1}{N}{{\displaystyle \sum }}_{\mathrm{k}=1}^N\log a_k},$$

(3)

Since non-integer values are obtained by applying the geometric mean method, we rounded it to the nearest integer referring to a specific degree of relative importance on the adopted 9-point semantic Saaty scale. The pairwise comparisons between the parent criteria are shown in

Table 3. Pairwise comparisons between parent criteria.

Teacher–Organization Pairwise Comparison		Teacher–Structure Pairwise Comparison		Organization–Structure Pairwise Comparison
Score	Number of Learners	Score	Number of Learners	Score	Number of Learners
1/9	2	1/9	0	1/9	1
1/8	0	1/8	0	1/8	0
1/7	5	1/7	6	1/7	2
1/6	0	1/6	0	1/6	0
1/5	10	1/5	0	1/5	4
1/4	0	1/4	0	1/4	0
1/3	14	1/3	3	1/3	12
1/2	0	1/2	1	1/2	0
1	36	1	28	1	42
2	0	2	0	2	0
3	19	3	10	3	11
4	0	4	0	4	1
5	13	5	27	5	16
6	0	6	0	6	0
7	7	7	8	7	12
8	1	8	1	8	0
9	3	9	26	9	9
WGM 1	1.18 ≈ 1	WGM	2.81 ≈ 3	WGM	1.67 ≈ 2

¹ WGM: weighted geometric mean calculated according to Equation (3).

.

Starting from Table 3 and from the weighted geometric mean values, it was possible to build the pairwise comparison matrix inherent to the three parent criteria, i.e., teacher, organization and structure (see

Table 4. Pairwise comparison matrices for parent criteria.

	Teacher	Organization	Structure
Teacher	1	1 1	3 1
Organization	1 2	1	2 1
Structure	1/3 2	1/2 2	1

¹ Equals the corresponding WGM, as reported in Table 3. ² Equals the reciprocal of the corresponding WGM (1/WGM), as reported in Table 3.

).

The data contained in the pairwise matrix (Table 4) were used to derive the order of the pairs between the elements of each matrix, i.e., a scale of values that expresses the final choice of priorities. The priority scale is a scale obtained using the eigenvalue method proposed by Saaty [3], and, here, it was implemented by developing a straightforward algorithm in the MatLab environment (Figure 4).

The same process was then iterated for each sub-criterion to build the corresponding pairwise comparison matrices (the pairwise comparison matrices for the sub-criteria are reported in the Results section together with the priority values). The next step was to calculate each sub-criterion’s global (or overall) weight, by multiplying the sub-criterion priorities by the corresponding parent criterion’s priority.

2.5. Matrix Consistency Check

The next step was to calculate the consistency index (CI) and consistency ratio (CR) for all matrices according to the following Equations (4) and (5):

$$CI=\frac{{\lambda }_{max}-n}{n-1},$$

(4)

$$CR=\frac{CI}{RI},$$

(5)

where ${\lambda }_{max}$ is the largest eigenvalue, $n$ is the matrix dimension and RI is the Saaty random consistency index, which depends on the matrix dimension, as reported by Saaty [50,55]; in our case, for a matrix size of 3, the RI is equal to 0.52.

2.6. Application of the Approach to a Case Study: Processing LS-Based Questionnaires

Concerning the Likert scales, the first step was to evaluate the percentage of learners who provided a specific response based on their satisfaction with the service quality provided, as shown in

Table 5. Percentages of response for each sub-criterion.

ITEM	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
Location	49.1	45.5	3.6	1.8	0
Equipment	33.6	58.2	6.4	1.8	0
Comfort	37.3	47.3	11.8	3.6	0
Preparation	56.4	39.1	3.6	0.9	0
Effectiveness	60.9	35.5	1.8	1.8	0
Interpersonal skills	50.9	47.3	0.9	0.9	0
Cost	28.2	41.8	22.7	6.4	0.9
Duration	17.3	52.7	24.5	5.5	0
Contents	36.4	58.2	5.5	0	0

.

From the obtained data, it was observed that high satisfaction percentages (rates) were expressed for each item, and for the preparation of the teacher and his/her availability. Starting from these percentage values, the percentages of response for each parent criterion were calculated by averaging the values of the 3 respective sub-categories, as shown in

Table 6. Percentages of response for each parent criterion.

ITEM	Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
Structure	40.0	50.3	7.3	2.4	0.0
Teacher	56.1	40.6	2.1	1.2	0.0
Organization	27.3	50.9	17.6	3.9	0.3

.

3. Results

3.1. Data Processing

The procedure described in the Methods section was applied for all the pairwise comparison matrices.

3.2. Calculation of Local and Global Priorities (AHP Method)

Regarding the local priorities, the eigenvalue method provided the following ranking concerning the three parent criteria:

• TEACHER weight: 0.443;
• ORGANIZATION weight: 0.387;
• STRUCTURE weight: 0.169.

Both the matrix and the corresponding weights for all parent criteria (teacher, organization and structure) and sub-criteria are shown in

Table 7. Pairwise comparison matrix and AHP priority weights of both parent criteria and sub-criteria.

Parent Criteria
	Teacher	Organization	Structure	Weight 1
Teacher	1	1	3	0.443
Organization	1	1	2	0.387
Structure	1/3	1/2	1	0.169
Sub-criteria belonging to the “teacher” parent criterion
	Interpersonal skills	Preparation	Effectiveness	Weight 1	Overall weight 2
Interpersonal skills	1	1/3	2	0.263	0.116
Preparation	3	1	2	0.547	0.242
Effectiveness	1/2	1/2	1	0.190	0.084
Sub-criteria belonging to the “organization” parent criterion
	Duration	Contents	Cost	Weight 1	Overall weight 2
Duration	1	1	2	0.400	0.155
Contents	1	1	2	0.400	0.155
Cost	1/2	1/2	1	0.200	0.078
Sub-criteria belonging to the “structure” parent criterion
	Equipment	Comfort	Location	Weight 1	Overall weight 2
Equipment	1	2	3	0.540	0.091
Comfort	1/2	1	2	0.297	0.050
Location	1/3	1/2	1	0.163	0.028

¹ Weight: obtained by applying the methodology described in Section 2.4. ² Overall weight: obtained by multiplying the sub-criterion weight by the corresponding parent criterion weight.

.

After calculating these values, which represent, as mentioned, the absolute weights of the sub-criteria, it was possible to define the “hierarchy of needs” of the users (

Table 8. Learners’ needs in decreasing priority order.

Sub-Criterion	Overall Weight
1. Preparation	0.242
2. Duration	0.155
3. Contents	0.155
4. Interpersonal skills	0.116
5. Equipment	0.091
6. Effectiveness	0.084
7. Cost	0.078
8. Comfort	0.050
9. Location	0.028

).

Table 8 shows that the top priority is to meet very competent teachers (preparation). High importance is assigned to all three teacher characteristics: preparation, interpersonal skills and effectiveness, as well as to the duration and contents of the courses. Lower importance is instead assigned to their cost, to the comfort of the rooms and to the location.

3.3. Matrix Consistency

By applying Equations (4) and (5), the following CR values were obtained for each category:

$$C{R}_{teacher}=0.130$$

$$C{R}_{organization}=0$$

$$C{R}_{structure}=0.009$$

The CR equals 0, indicating that the judgments are perfectly consistent [56,57,58]. Moreover, according to Saaty (2008) [3], when the CR exceeds 0.1, the judgments set may result in being inconsistent and unreliable. Nevertheless, in practical applications and in quality assessment studies involving a large number of participants, CR values slightly higher than 0.1 can be considered acceptable. In our case, except for the matrix related to the parent criterion “teacher”, which shows a very slight degree of inconsistency, all the other matrices of the pairwise comparisons are strongly consistent.

3.4. Sensitivity Analysis

In order to widen and improve the AHP process, we also carried out a sensitivity analysis to assess the impact that variations in the parent criteria’s weights, e.g., changes due to a higher or lower importance assigned to a specific criterion, can have on the obtained hierarchy of needs (final priority weights). Twelve dynamic scenarios were simulated by considering a ±10% and ±20% increase/decrease in all the parent criteria’s weights, as briefly summarized in

Table 9. Increase in the percentage of each parent criterion in all the simulated scenarios for the sensitivity analysis.

Scenario/Parent Criterion	1	2	3	4	5	6	7	8	9	10	11	12
C1 (Teacher)	−10%	10%	10%	10%	−10%	−10%	−20%	20%	20%	20%	−20%	−20%
C2 (Organization)	10%	−10%	10%	−10%	10%	−10%	20%	−20%	20%	−20%	20%	−20%
C3 (Structure)	10%	10%	−10%	−10%	−10%	10%	20%	20%	−20%	−20%	−20%	20%

.

The graphs shown in the following Figure 5 report the dynamical changes in the hierarchy of needs (final priority weights) as a function of the changes in the parent criteria’s weights. Each graph represents a specific scenario (as reported in Table 9) where the weights of the parent criteria are represented as gray histograms and refer to the primary y-axis, while dynamical changes in the final priority weights are represented as colored straight lines and refer to the secondary y-axis. For each scenario, five cases are represented on the x-axis: (i) original case (no changes in the parent criteria’s weights); (ii) change in the weight of only the first parent criterion (C1—teacher) without altering the other two parent criteria; (iii) change in the weight of only the second parent criterion (C2—organization) without altering the other two parent criteria; (iv) change in the weight of only the third parent criterion (C3—structure) without altering the other two parent criteria; (v) simultaneous changes in the weights of all the parent criteria (overall).

The analysis of the simulated scenarios showed that the AHP is slightly sensitive to changes in the importance of the parent criteria. Indeed, the ranking of the four main priorities of the original hierarchy of needs (Preparation, Duration, Contents and Interpersonal skills) does not exhibit significant changes, preserving their order in the most dynamical scenario presented. For example, the Preparation item preserves its relative position in the ranking in all the presented scenarios (Figure 5a–l); a small shift from the original ranking occurs in Figure 5h,j, where the overall priority weight associated with the term Interpersonal skills slightly exceeds the overall weight for the Duration and Contents items compared to the original ranking. On the contrary, low or moderate sensitivity is detectable by looking at the variations in the final priority weight of the Equipment item, which was shown to be particularly sensitive to changes in the importance of the organization criteria. For example, when the importance of organization is decreased from 0.39 to 0.31 (e.g., in Figure 5h,l), the overall rank of Equipment changes from 0.09 to 0.06, producing slight deviations from the original ranking. Overall, we can state that the hierarchy of needs is preserved with only low variations in some specific scenarios simulated in the sensitivity analysis.

3.5. Data Aggregation and Comparison

As a final step, the information gathered by means of the employed techniques (AHP and LS) was combined in order to propose a practical tool that can facilitate the process of decision making and then improve the users’ perceived quality.

In Figure 6, a direct, visual (therefore simple to interpret and quick) comparison of the results obtained with the two methods is presented.

As stated, it is easy to understand from Figure 6, in which we summarize users’ satisfaction through an LS (top of Figure 6) and the AHP (bottom of Figure 6), that one of the main features that, according to the AHP, could affect the quality of training courses concerns the teacher’s preparation (bottom of Figure 6), which is first in the AHP ranking. However, according to the LS, the same feature is associated with a high value of learners’ satisfaction (top of Figure 6), thus suggesting that, regarding the teacher’s preparation, the Centre of Biotechnology has already reached its target. Furthermore, the duration of the courses is the second most important feature according to the AHP hierarchy (bottom of Figure 6), thus requiring priority intervention. For the same feature, the LS results (top of Figure 6) show a low satisfaction value associated with the course duration. This indicates that the course duration, while representing an element of great importance, generates a degree of satisfaction that is relatively low; thus, a higher priority should be given to this aspect in order to achieve a greater improvement in the course quality.

Analogous considerations and analysis can be extended to all the items, so it is possible to identify the users’ needs through the application of the AHP and users’ satisfaction through the application of the LS. Furthermore, in view of the sensitivity analysis of the AHP that was carried out, we may also state that, since the hierarchy of needs from the AHP does not exhibit significant changes, especially for the items with the highest priority, when the parent criteria weights are varied, the comparison with the LS results is far more necessary since it can suggest considering as priorities those items that, despite having a lower AHP ranking, show a lower user satisfaction degree and thereby require primary improvements even with respect to the top-ranked AHP items.

4. Discussion

The concept of quality has become increasingly popular in business, a very competitive environment and nowadays substantially without borders. The national and international competition requires adaptation of products offered to the customers’ expectations. In particular, it is very important to continuously improve the quality of goods and services in order to reduce costs and eliminate waste. The quality evaluation implies some difficulties related to their definition, often dependent on the context, and to the different methods that can be employed.

This issue is strongly felt in the healthcare field, in which the quality assessment is particularly complicated due to the characteristics of the structure, which does not produce a good but rather a special service, with large intangible components and enhanced customization. Costs and quality are two key points concerning the healthcare industry worldwide [59,60,61]. Thus, methods developed and used in the industrial and manufacturing sectors have also been extended to the transactional and service fields [62], including methods aiming at assisting and supporting decision makers and health personnel with healthcare issues [63,64,65,66]. The introduction of the continuing medical education (CME) concept in health organizations allowed health personnel to maintain their own competences and to update themselves about new and developing areas and new care modalities [67], ultimately leading to better patient health [1,68,69,70]. However, studies have shown a substantial difference between the real and ideal performance of CME, suggesting a lack of effect of formal CME [67,71,72,73].

In this study, we extended the methodology, results and discussion briefly presented in our previous conference paper [4]. In particular, we proposed the use of a hybrid approach by combining two different methodologies, namely, an LS and the AHP, for quality assessment and multicriteria decision making in the field of medical education.

We are aware that many methodologies are available in the field of multicriteria decision making, as detailed in the Introduction section, such as: the IR-AHP-MABAC model [37], which is mainly used to determine the weight coefficients of the criteria; Level Based Weight Assessment (LBWA), which enables the involvement of experts from different fields, and for this reason, we considered it not to be pertinent to our study that involves a number of learners attending medical training courses; the Best Worst Method (BWM), which is considered the most data- and time-efficient method when there are a large number of criteria, which is, however, not in line with the case study proposed here because it considers a limited number of criteria; Multi-Attributive Ideal-Real Comparative Analysis (MAIRCA) and TOPSIS [74,75], both based on the comparison of theoretical and empirical alternative ratings, which, however, fall outside of our methodological framework; the Characteristic Objects Method (COMET), Stable Preference Ordering Towards Ideal Solution (SPOTIS) and the SIMUS method, which have been developed to avoid the rank reversal phenomenon [76,77,78], which, however, does not occur in the present study. Therefore, we decided to focus on the combination of more generalizable, versatile and traditional approaches to tailor them to our specific case. Of course, other methods and their variations could also be considered; however, we chose an LS and the AHP for two substantial reasons. First, the objective of this paper was not to investigate the mathematical details and the performances of the different possible models of multicriteria decision-making methods, but rather to propose a proof of concept. Secondly, the chosen approaches appear to be widely used in the healthcare field.

After presenting the theoretical aspects that guided us to the design of such a hybrid approach, we applied it to the problem of evaluating the level of quality achieved in the training course provision at the Centre of Biotechnology of the National Hospital A.O.R.N. “A. Cardarelli”, in the service of high medical education. The proposed analysis was performed by taking into account the users’ perspectives (i.e., medical specialists attending medical training courses). The quality in the delivery of medical training courses perceived by the learners, which is presented in Figure 5, represents a driver for continuously improving and re-designing the training process in order to effectively respond to the dynamic user needs.

In addition to the information presented in [4], further methodological detail, procedural steps, results and discussion were included and thoroughly described in view of the opportunity to generalize the proposed approach to more complex scenarios and different decision-making problems in the sector of quality assessment of education programs.

In the presented case study, the combination of the two methodologies allowed us to highlight the most important expectations of the learners, and then to verify if the quality of the service offered fulfils the necessities of the users. Three parent criteria, with three sub-criteria associated with each parent criterion, were defined. Then, a questionnaire to evaluate the user satisfaction degree was designed based on the LS technique, which already proved to be a successfully adopted assessment of user satisfaction in the health sector [4,23,28]. The obtained outcomes underline that an analytical and qualitative evaluation of the service quality of the training process can be accomplished by combining the two methods in order to provide a more complete perspective on what learners perceive. In particular, using the AHP alone, the obtained hierarchy does not provide sufficient information, which could be misleading, to reveal which interventions are necessary to improve the service, because this technique does not provide information about user satisfaction. In turn, combining the two methods improves the understanding of service quality, as is easy to note from the obtained results in this study.

In summary, it is worth highlighting that the goal of this study was to propose a methodology to verify whether the needs expressed by the stakeholders (i.e., the learners in our case) correspond to the actual services needing improvements. In this sense, our contribution lies in the impact of this approach. Indeed, we propose a novel combination of traditional methods, i.e., the AHP and an LS, and adopted and combined these two techniques to show that, despite the fact both the AHP and LSs are widely used and recognized methodologies in several fields, if they are used separately, they can provide misleading conclusions. Instead, if we combine them, we can provide decision makers with a wise trade-off between the voice of the customer (i.e., the needs expressed by the users, learners) and the voice of the process (i.e., those services that need to be improved more than others). This could suggest better decisions regarding the prioritization of the improvement actions and a better organization of resources.

5. Conclusions

This study aimed to provide a proof of concept about the combined use of the AHP and an LS in the quality assessment of medical training courses. Through the synergistic application of both methods, the AHP and an LS, a decision maker could be able to improve those service features which are very important but that do not meet the users’ satisfaction. Obviously, assessing and analyzing users’ satisfaction through a continuous collection of learners’ opinions are important drivers to improve the quality of the training process, as well as designing a monitoring system, whose aim is to reveal deficiency and the related corrective actions. Furthermore, by using this approach, a decision maker can avoid intervening on less relevant aspects, thus saving money and time. The aim of this analysis was to reveal the main areas in which people can invest money and time for continuously improving the training process according to the user satisfaction degree.

Future studies will aim to start from the proposed proof of concept in order to optimize it and compare it with other methodologies and to generalize the approach to multiple applications. Alternative and revised AHP approaches will be evaluated alone or in combination with the LS to address the same or analogous issues in the medical education field. In addition, future research directions will aim at overcoming the major limitations of this study. Indeed, the very specific case study proposed in this work could be generalized in a multi-center study involving multiple courses and education programs. Moreover, a sensitivity analysis will be carried out to assess the performance of the method and identify possible strengths and weaknesses in order to optimize the approach and also test it in different scenarios.