Application of Fuzzy Logic to Evaluate the Economic Impact of COVID-19: Case Study of a Project-Oriented Travel Agency

Abstract

The aim of the study is to create a performance evaluation controlling model to evaluate the performance of tourism enterprises as a function of the economic effects of COVID-19. As a result of the significant change in demand resulting from the economic environment, expectations and cyclicality caused by the pandemic, the assessment of organization performance has become subjective. Under these changed environmental conditions, most of the methods used by tourism companies to evaluate performance are not effective enough. In our research, we illustrated a controlling model based on fuzzy logic through a case study. By applying the model, it becomes possible to evaluate project-oriented tourism organizations according to different standardized norms. Our model considers the subjectivity derived from measurability and goal setting. We point out that the performance of organizations operating in the tourism industry significantly influenced by COVID-19 can be subjectively assessed during the pandemic period and thus depends on the analytical context. By evaluating the performance of tourism organizations along internal organizational goals, more relevant information content and more informed managerial decision support can be achieved.

1. Introduction

The tourism supply chain is a network that spans continents [1]. For businesses in these networks, adaptation to ever-changing global conditions can be described as a fundamental competitive criterion. The most significant negative global macro condition in recent decades has been the COVID-19 pandemic crisis. The pandemic has caused significant devastation in almost every industry worldwide [2]. Among the most severely affected industries are the tourism and hospitality sectors, which have almost completely ceased to function as a result of the epidemic [3]. As a result of the epidemic and defense strategies, international tourism and cross-border tourism have also been drastically reduced or eliminated. As a result, tourism and hospitality organization have had significant losses [4,5].

The pandemic has repeatedly highlighted the fact that the tourism sector is a highly exposed industry to external conditions [6]. This exposure was manifested not only in the almost complete halt in global tourism during the pandemic but also in the growth seen in the summer of 2020 [7]. This exposure is also supported by the studies by Ramelli and Wagner (2020) and Song et al. (2021), which draw attention to the fact that pre-epidemic fundamentals and strategies of tourism enterprises also influence the successful adaptation of organizations to the COVID-19 epidemic period to changed economic conditions [4,8].

As a result of COVID-19, significant changes in the tourism industry will lead to inequalities in both space and time. Various government measures and industry advocacy organizations can make a significant contribution to reconstructing inequalities and successfully recovering. Furthermore, as the recovery process and consumer demands change, there will be a greater emphasis on sustainable and rural tourism, as well as domestic tourism, in the planning of rethinking strategies [9]. Based on Rastegar et al.’s (2021) study, it is important to take into account that in recovering the tourism sector, it is important not to return to the previous system but to create a new system that changes ecological, social and economic processes [10]. At the same time, regardless of whether the recovery is based on government interventions or based on rethought strategies, the tourism industry is undergoing significant changes [11,12]. As a result of these significant changes and adaptation to the environment, organizations in the tourism sector also need to rethink their strategies.

It is inevitable for management to develop an effective planning and controlling system to support the development and implementation of an appropriate strategy. Controlling can be a key area that provides information to help business make the right decisions at every level. This helps one to adapt to a dynamically changing environment [13].

The tourism sector is characterized by external exposure in addition to being a competitive and dynamically expanding industry. Many innovations have contributed to the development of the sector in recent years, and many areas of activity have been transformed. This includes the operating model of travel agencies and the nature of the products they offer [14,15,16,17].

During travels organized by travel agencies, there is a strong trend of declining demand for standard travels. This can be attributed to the proliferation of various innovative accommodation platforms and the rapid and universally accessible content of information on the Internet [18,19]. Due to these negative changes, various tour operators have responded with new business models, reaching new target audiences and creating innovative products [20,21]. One such new product range is the organization of travel that is specifically customer oriented. Both supply and demand for these product categories are growing dynamically [22]. Organizing and selling these new products require a fundamentally different business model. As each travel is unique and customer oriented, they can be considered as different projects. Although unique and customer-oriented travel projects may have common features, they can be considered unique in each case. For companies that offer these unique customer-oriented products, the project-oriented organizational form can be a highly efficient structure.

Evaluating the success of individual projects and evaluating project-oriented organizations raise a number of issues due to their unique nature [23,24]. Performance evaluation is basically a sub-area of controlling that includes a number of methods. However, the controlling systems may not be able to accurately monitor and evaluate non-standardized processes and sub-areas [25,26]. Consequently, project-oriented organizations need a controlling method that is not only suitable for evaluating projects but also for evaluating related sub-areas [27]. Such a controlling model makes it possible to explore the points of intervention at both the functional and strategic levels, and in addition to the unique nature, organizational performance can also be assessed [28]. Nowadays, such a project-oriented controlling system can now be effectively developed and implemented in organization operations by the level of development of IT and mathematical-statistical methods [29]. Using the innovations created by IT development and digitalization, databases and data-processing capabilities are available that will fundamentally change controlling and ERP systems regardless of industry [30]. This innovation development has created the opportunity for the various reports to be able to illustrate the effectiveness of an entire area or organizational unit in a single indicator. The basic criteria for such aggregated indicators are the appropriate and efficient infrastructure and the use of professional and mathematical methods [31]. Different evaluation algorithms and standardization norms are required in each case to interpret the indicators and make the resulting decisions [32].

2. Materials and Methods

Through an extended case study [33] in our research, we develop a controlling aspect-based model for evaluating strategic effectiveness based on plan–fact analysis. The company included in our extended case study is an organization based in Hungary. The main activity of the company is the organization of individual-customer-oriented trips. The main destinations of travel are Central and Eastern European countries. In the present study, we analyze the results of the organization’s 113 unique travel projects in 2019 and 120 in 2020. The organization considers these travels to be separate projects due to their unique nature. Our research purpose is to develop a complex and general performance evaluation model that is suitable for organizations operating in a project organization structure. Using our model, we point out that the performance of organizations operating in the tourism industry significantly influenced by COVID-19 can be subjectively assessed during the pandemic period.

The company evaluates its performance based on plan–fact analysis ratios, which it calculates at several hierarchical levels. These ratios are stored in the corporate governance system for several years. The calculation basis of our model is the ratios calculated from the plan–fact analysis at different hierarchical levels. The unit of measurement for these ratios is in all cases given as percentages. (Data on ratios from plan–fact analysis in

Table 1. Data table of KPIs of project portfolios.

	Project Portfolio 1.						Project Portfolio 2.
KPIs	ST1.A	ST1.B	ST2.	ST3.A	ST3.B	ST3.C	ST1.A	ST1.B	ST2.	ST3.A	ST3.B	ST3.C
1.	4.62	9.77	1.30	4.62	9.77	0.75	4.13	9.51	1.34	4.13	9.51	1.01
2.	5.03	17.31	2.04	3.52	12.12	0.67	4.73	17.25	2.09	3.31	12.08	1.47
3.	−1.87	6.16	−2.21	−1.12	3.70	−0.67	−1.51	5.59	−1.38	−0.91	3.35	−1.05
4.	−2.67	−3.39	−2.92	−2.67	−3.39	−1.15	−5.73	−2.41	−2.64	−5.73	−2.41	−1.34
5.	1.92	11.79	0.92	1.54	9.43	0.05	3.61	12.22	0.45	2.89	9.78	0.46
6.	8.36	11.65	1.08	5.02	6.99	0.30	6.49	11.41	2.27	3.89	6.85	0.24
7.	12.26	8.08	0.20	9.81	6.46	0.13	8.93	9.07	0.75	7.14	7.26	0.23
8.	4.85	5.13	−5.00	2.91	3.08	−2.49	8.03	5.19	−2.36	4.82	3.11	−3.69
9.	−3.58	4.38	0.45	−2.51	3.07	0.24	−1.56	4.80	0.66	−1.09	3.36	0.45
10.	−1.77	−0.95	0.61	−1.77	−0.95	1.37	−0.78	−0.96	0.48	−0.78	−0.96	1.45
11.	6.67	8.50	1.02	4.00	5.10	4.19	5.02	8.41	1.32	3.01	5.05	6.13
12.	1.56	5.52	−1.00	1.09	3.86	−0.12	2.61	5.97	−0.22	1.83	4.18	−0.24
13.	−0.70	−3.68	−10.30	−0.49	−2.58	0.32	−0.26	−4.11	−9.21	−0.18	−2.88	0.33
14.	−0.48	25.07	4.10	4.10	17.55	0.26	−3.76	18.77	6.14	−2.63	13.14	0.89
15.	0.34	20.49	3.49	0.34	20.49	−2.45	−0.57	19.72	8.20	−0.57	19.72	−1.64
16.	−5.78	22.73	8.84	−3.47	13.64	3.00	−3.42	21.88	10.24	−2.05	13.13	4.03
17.	2.64	10.09	−3.59	1.58	6.05	−0.79	3.26	10.52	−3.47	1.96	6.31	−1.18
18.	−1.76	4.91	−3.95	−1.41	3.93	−0.82	−0.13	5.42	−1.42	−0.10	4.34	−1.28
19.	−4.74	−1.94	−3.57	−2.84	−1.16	−0.72	−3.31	−1.22	−1.78	−1.99	−0.73	−1.04
20.	4.93	29.28	2.07	4.93	29.28	2.41	5.66	29.10	1.90	5.66	29.10	3.40
21.	−4.90	−4.21	−1.66	−3.92	−3.37	−0.29	−2.10	−4.05	−3.42	−1.68	−3.24	−0.28
22.	5.47	4.91	0.14	3.56	3.19	0.75	3.37	5.71	0.17	2.19	3.71	1.10
23.	0.19	6.15	−2.74	0.15	4.92	−0.63	2.42	5.74	−2.51	1.94	4.59	−1.18
24.	13.24	7.00	5.98	13.24	7.00	4.20	14.72	12.09	6.31	14.72	12.09	5.19
25.	0.00	14.70	1.21	0.00	9.56	0.83	−1.09	14.63	1.21	−0.71	9.51	1.14
	Project Portfolio 3.
KPIs	ST1.A	ST1.B	ST2.	ST3.A	ST3.B	ST3.C
1.	5.84	9.91	1.08	5.84	9.91	0.98
2.	6.33	18.25	1.10	4.43	12.78	1.04
3.	1.41	5.59	−5.61	0.85	3.35	−0.88
4.	−4.65	−3.26	−1.22	−4.65	−3.26	−1.04
5.	3.66	11.77	1.42	2.93	9.42	0.31
6.	7.62	12.00	2.06	4.57	7.20	0.10
7.	10.75	9.72	1.08	8.60	7.78	0.22	Very underperforming:
8.	3.64	5.75	−6.53	2.18	3.45	−2.43	Not acceptable:
9.	−0.50	5.12	2.36	−0.35	3.58	0.28	Acceptable:
10.	−0.06	−0.98	0.75	−0.06	−0.98	0.96	Appropriate:
11.	5.85	9.17	7.84	3.51	5.50	4.19	Distinguished:
12.	2.24	6.08	−1.00	1.57	4.26	−0.28
13.	−2.51	−3.48	−7.59	−1.76	−2.44	0.12
14.	−1.06	23.72	2.63	−0.74	16.60	−0.12
15.	−3.26	20.31	5.63	−3.26	20.31	−2.79
16.	−4.17	14.47	9.75	−2.50	8.68	4.58
17.	1.95	10.45	−5.32	1.17	6.27	−0.62
18.	−1.54	5.16	−4.75	−1.23	4.13	−0.69	ST1.A = ST1. 2019 (%)
19.	−2.93	−1.45	−3.57	−1.76	−0.87	−0.71	ST1.B = ST1. 2020 (%)
20.	4.57	28.82	4.31	4.57	28.82	2.32	ST2. = ST2. 2019–2020 (%)
21.	−5.87	−4.58	−4.69	−4.70	−3.66	−0.53	ST3.A = ST3. 2019 (%)
22.	6.05	5.76	0.41	3.93	3.74	0.88	ST3.B = ST3. 2020 (%)
23.	1.52	6.01	−6.68	1.22	4.81	−0.69	ST3.C = ST3. 2019–2020 (%)
24.	12.25	11.73	1.96	12.25	11.73	4.65
25.	−1.69	15.59	1.36	−1.10	10.13	0.73

,

Table 2. Data table of KPIs.

	Aggregated Key Performance Indicators
KPIs	ST1.A	ST1.B	ST2.	ST3.A	ST3.B	ST3.C
1.	3.97	8.09	1.07	3.97	8.09	1.07
2.	4.40	14.64	1.52	3.08	10.25	1.07
3.	−0.65	4.80	−2.59	−0.39	2.88	−1.56
4.	−3.78	−2.46	−1.86	−3.78	−2.46	−1.86
5.	2.62	9.97	0.83	2.09	7.98	0.66
6.	6.13	9.72	1.47	3.68	5.83	0.88	Very underperforming:
7.	8.69	7.48	0.58	6.95	5.98	0.47	Not acceptable:
8.	4.88	4.45	−3.85	2.93	2.67	−2.31	Acceptable:
9.	−1.53	3.98	0.97	−1.07	2.78	0.68	Appropriate:
10.	−0.72	−0.80	0.44	−0.72	−0.80	0.44	Distinguished:
11.	4.79	7.21	2.84	2.87	4.33	1.70
12.	1.84	4.90	−0.65	1.29	3.43	−0.45
13.	−0.86	−3.17	−7.57	−0.60	−2.22	−5.30
14.	−1.53	19.08	3.57	−1.07	13.35	2.50
15.	−1.17	17.14	5.01	−1.17	17.14	5.01
16.	−3.71	16.33	8.21	−2.23	9.80	4.93
17.	2.26	8.65	−3.42	1.35	5.19	−2.05	ST1.A = ST1. 2019 (%)
18.	−0.84	4.33	−2.79	−0.67	3.46	−2.24	ST1.B = ST1. 2020 (%)
19.	−3.02	−1.25	−2.48	−1.81	−0.75	−1.49	ST2. = ST2. 2019–2020 (%)
20.	4.28	24.24	2.29	4.28	24.24	2.29	ST3.A = ST3. 2019 (%)
21.	−3.33	−3.55	−2.76	−2.66	−2.84	−2.21	ST3.B = ST3. 2020 (%)
22.	3.96	4.58	0.22	2.57	2.98	0.14	ST3.C = ST3. 2019–2020 (%)
23.	1.26	4.95	−3.33	1.01	3.96	−2.67
24.	11.34	8.97	3.84	11.34	8.97	3.84
25.	−0.79	12.45	1.03	−0.51	8.09	0.67

,

Table 3. Data table of BSC.

	Balanced Scorecard Perspectives
	Weights (points)	ST1.A	ST1.B	ST2.	ST3.A	ST3.B	ST3.C
Financial	1.00	1.44	5.43	0.26	1.44	5.43	0.26
Learning and growth	0.80	2.04	3.06	0.52	1.63	2.45	0.42
Internal business	0.90	−0.73	7.79	2.86	−0.66	7.01	2.58
Customer	0.85	2.03	6.38	0.59	1.73	5.42	0.50
		Very underperforming:
		Not acceptable:
		Acceptable:
		Appropriate:
		Distinguished:
		ST1.A = ST1. 2019 (%)
		ST1.B = ST1. 2020 (%)
		ST2. = ST2. 2019–2020 (%)
		ST3.A = ST3. 2019 (%)
		ST3.B = ST3. 2020 (%)
		ST3.C = ST3. 2019–2020 (%)

and

Table 4. Strategic performance index.

Strategy
	ST1A.	ST1.B	ST2.
Strategy index	1.04	5.08	0.94
	Very underperforming:
	Not acceptable:
	Acceptable:
	Appropriate:
	Distinguished:
	ST1.A = ST1. 2019 (%)
	ST1.B = ST1. 2020 (%)
	ST2. = ST2. 2019–2020 (%)

are also given as a percentage.)

Our model development is based on data collected in the organization’s ERP system. From the KPIs measured by the organization, we selected twenty-five indicators that were included in the analysis. In selecting KPIs, the criterion was contribution to strategic effectiveness. The formulation of this criterion is necessary because the aim of our research is to create a strategic performance index. The weights of the project portfolios, key performance indicators (KPI), and balanced scorecard (BSC) perspectives were determined based on the opinions of the managers of the organization. Weight values were measured using a questionnaire method. The time of the survey was from 21 October 2020 to 4 November 2020. The questionnaire was completed by nine top managers. The nine fillers are all employees in managers positions in the company who are involved in strategy development and decision makers. The nine managers are leaders in various functional and strategic areas. By completing the questionnaire with all managers, determining the weights of KPIs that affect strategic effectiveness has become more relevant. To determine the weights, we added the scores of the managers evaluations. The results were categorized into predefined categories that we created. The reason for this is the treatment of subjectivity and fuzzy nature from the questionnaire method. Based on these, we defined five different fuzzy categories. The categories were defined based on the results of the questionnaires. This is how the following five fuzzy categories were created, which also represent weight values. For KPIs: 1.00 (score: 41–45); 0.80 (score 36–40); 0.70 (score: 31–35); 0.65 (score: 28–30); 0.60 (score 22–27). For BSC perspectives: 1.00 (score: 42–45); 0.90 (score 38–41); 0.85 (score 34–37); 0.80 (score: 30–34). For project portfolios: 1.00 (score: 41–45); 0.85 (score 34–40); 0.70 (score: 27–34). Due to the distribution of the results of the questionnaire, it was not necessary to define additional weights in either case.

The plan values for KPIs are defined by the organization. In addition to the KPIs belonging to the projects, we also include in the analysis indicators that cannot be interpreted in the case of travel projects but in relation to a given project portfolio as a whole. Projects have equal weight in relation to each other, within a given project portfolio. KPIs also have equal weight in different project portfolios. At the level of project portfolios and BSC, different aggregates have different weights. The strategic level consists of four perspectives with different weights defined at the BSC level (Figure 1).

After organizing these data into a logical structure, we build a BSC-based performance evaluation model of the examined organization. In order to incorporate high subjectivity into our controlling model, we use fuzzy methodology. Before applying the fuzzy-logic methodology to measure performance, the following is an overview of the fuzzy-set concept.

2.1. Basic Concepts of Fuzzy Logic

In the field of artificial intelligence research, various expert systems have been developed since the 1950s that draw inferences based on data and knowledge base and Boolean algebraic logic [34]. Boolean algebra is based on binary values: true–false. However, in the natural sciences and social sciences, in many cases, phenomena occur that can be poorly or subjectively defined. Modeling these phenomena with exact methods is not possible at all. In response to this problem, Zadeh developed the fuzzy logic method of the continuum with infinite set of values in 1965 [35].

The meaning of fuzzy is vague; hence, the classification into a given set in these systems is determined by membership functions. Membership functions illustrate the value of a particular language variable [36]; for example, the evaluation of a particular organization can be the values of a linguistic terms: ineffective, moderately effective, and very effective. Belonging to a given set can be determined using the function. This operation is called fuzzification [37]. It is then necessary to develop a set of rules that perform operations and draw conclusions using each linguistic term. As a result of this process, an aggregate of member functions can be created, which is an essential element of defuzzification. Defuzzification results in an actual value that can be considered the end result of fuzzy analysis [35,37].

Within project evaluation, fuzzy-logic-based methods are common among the methodologies used to evaluate project success [38,39,40,41,42]. The success of projects significantly determines the overall organizational effectiveness [43]. The fuzzy concept of project success is since that the word “success” is an indicator that has no sharp boundaries on the basis of which general categories could be created [44]. “Indicator A, B and C did not meet the plan value” and “Indicator A value is unacceptable”, but “Indicator A value is acceptable compared to indicator B value”. These examples illustrate the fuzzy set theory developed by Zadeh in judging the performance of indicators [45].

In our research, we define the classification of the strategic effectiveness index as a fuzzy subset. For a model based on fuzzy logic, it is necessary to define the universe (U), the elements (xi) U, where U = {x1 + x2 + … xn}, and the fuzzy subset A is included U, where

$$A=\left\{\frac{x}{{\mu }_A\left(x\right)|x\in U}\right\}$$

(1)

The membership function of a fuzzy subset A is most commonly expressed by:

μ_A: U → [0,1], which assigns to each element x ∈ U the membership degree of (x) μx in A: μ_A(x) = μx.

The most commonly used fuzzy logic operations are intersection, union, and complement [35]:

-

Intersection of two fuzzy subsets A and B: μ_A∩μ_B = minimum {μ_A(x), μ_B(x)}

-

Merge of two fuzzy subsets A and B: μ_A∪μ_B = maximum {μ_A(x), μ_B(x)}

-

The complement of A: μ_A’(x) = 1 − μ_A(x)

2.2. Steps of Modeling

Step 1: Selection of KPIs measured by the organizational controlling system that are KPIs influencing strategic performance. Determining plan–fact analysis ratios for these KPI. Further analysis is based on the plan–fact analysis ratios of these KPIs.

Step 2: Weighting of aggregates created based on subjective expert opinion. Choice of different standardization norms to evaluate ratios from plan–fact analysis for the KPIs.

Step 3: Evaluate aggregates, created based on subjective expert opinion, based on selected standardized norms.

Step 4: Evaluate the aggregated results of the selected KPIs based on the selected standardized norms.

Step 5: Define and evaluate aggregates for balanced-scorecard perspectives based on different standardization norms.

Step 6: Define and evaluate a strategic performance index based on different standardization norms.

3. Results

The model we have built evaluates the effectiveness of a travel-organizing company operating in a project-oriented organizational structure. The project-oriented structure of the examined organization stems from the unique nature of travel organization. The unique nature of the travels means that they are not standardized; therefore, it is not pre-assembled travel packages that are sold, but small-group packages tailored to individual customer needs. Thus, organizational effectiveness is significantly influenced by the success of projects. The primary internal organizational structure is subordinate to project management. Some project activities are planned and implemented centrally. Centralized background functions provide the basic operating conditions for the execution of individual projects [46]. At the same time, the operational project manager is responsible for the functional processes of the projects [47]. Implementing an organizational structure requires a project-based organizational strategy and the application of management methods [23].

The hierarchical controlling system we built consists of five (Figure 1) levels. The lowest level is made up of the values of the KPIs for the different projects. The values of KPIs are determined by the company using different metrics. For each project in the organization, KPIs created by the same measuring points are evaluated. In order to be able to evaluate projects effectively, it is necessary to aggregate projects according to a certain aspect. The aggregation of travel projects can also be performed along several expert opinions and using different mathematical-statistical methods. The organization uses grouping by price category as an aggregation method. Based on this, three groups were created, which are located on the second level in our model. The third level is the aggregate values of the KPIs included in the analysis. At the fourth level, the values of already-grouped KPIs are aggregated along the perspectives of the balanced scorecard. The controlling system used by the company does not systematize the data in this structure. The need to use the BSC was caused by the more systematic structure of the controlling system and the more efficient analysis of the data arranged in the structure. Furthermore, the application of the BSC provides an opportunity to assess the economic impacts of the pandemic and the resulting planning and delivery effectiveness through different strategic perspectives. The top level is the strategic level, for which we create a strategic performance index. Using this index, we evaluate organizational strategic effectiveness in an aggregate indicator.

Plan–fact analysis ratios are used to evaluate the lowest level travel projects. The KPIs for these projects include plan–fact analysis ratios and plan–fact analysis ratios for KPIs that can only be interpreted at the project portfolio level. The average of these indicators represents the result of the examined project portfolio. Aggregate KPIs at the third level are the weighted average of specific indicators (e.g., ROE, ROA, CSR, etc.) related to the projects in the project portfolios. The value of the four different aspects of the balanced scorecard at the fourth level is the weighted average of the aggregated KPIs for those perspectives. The strategic performance index can be calculated as a weighted average of the values of the BSC perspectives (Figure 1).

3.1. Step 1. Selection of the KPIs that Affect Strategic Effectiveness and Determining Plan–Fact Analysis Ratios for the KPIs

The organization uses a number of KPIs to measure performance and functional areas (Figure 2). From the KPIs measured by the organization, we selected twenty-five different KPIs influencing strategic performance. They can be calculated from the data of the selected KPIs and measuring points. These KPIs are measured during each travel project. This allows the company to have standardized monitoring of unique projects. Furthermore, during monitoring, a plan value is predefined for each KPI to which the actual fact value is compared. The application of plan–fact analysis provides an opportunity to standardize KPIs as a percentage. Plan values are predefined by the company’s strategic decision makers and controllers. In most cases, the plan values formed in this way are determined based on the company’s past period, capacity, internal organizational data, industry forecasts, macro-environmental factors and consumer surveys.

In our analysis, the actual value is the current value at the same time as the plan date. In each case, the ratio resulting from the plan–fact analysis is an indicator of past and current performances for a predefined given planning period. The definition of planning periods and plans for them is thus a key factor. The planning periods are industry specific, with the period from May to September being the most prominent of these planning periods for the organization under study. In our research, we use the exam ratios of the plan–fact analyses for the period from May to September of 2019 and 2020.

The company subjectively evaluates the extent of deviation from the value of the predefined plan in order to process the information content and make effective decisions. The ratio from the plan–fact analysis is crisp in nature, as the plan value appears as a threshold in the analysis, but the subjective definition of the thresholds raises the possibility of applying fuzzy logic. The crisp-like classification does not have enough information content to evaluate KPIs and make effective decisions. Based on the value of the plan–fact analysis ratio, the effectiveness of the KPI is only an indicator that does not have sharp boundaries by which a general classification could be established [44].

3.2. Step 2: Weighting of Aggregates Created Based on Subjective Expert Opinion. Choice of Different Standardization Norms to Evaluate Ratios from Plan–Fact Analysis for the KPIs

The second level of our model consists of aggregates of different unique travel projects. In our research, travel projects were aggregated along KPI and price category. The need to aggregate projects was caused by the quantitative reduction of data and the more efficient analysis of data. Aggregation by the price category used allows for more efficient exploration of intervention points. The company groups the grouping of projects solely according to this subjective aspect. Based on the applied subjective professional aspect, three project portfolios can be created. The first portfolio includes “P1-Basic”, unique travels that can be considered average or below average compared to the average price of all travels organized by the company. The second portfolio, in the case of grouping by price category, is the “P2-Luxury” category. This portfolio includes above average, but not above 20% above average, price categories. The third portfolio is the “P3-Extra Luxury” category, which includes travels with a price of more than 20% above average. These three project portfolios have different weights in achieving the strategic goal of the company ((P1 0.70); (P2 0.85); and (P3 1.00)). We determined the weights of the project portfolios based on a questionnaire method. The results of the questionnaire are in line with the guidelines in the corporate strategy that the “P3-Extra Luxury” travel category takes priority over the other categories.

The performance of the project portfolios and the organization is classified according to the standardized norm defined by the company and two other possible standardized norms chosen by us to support our research goal.

The standardized norm (ST 1) applied by the organization is based on the subjective classification of plan–fact differences for a given planning period defined by the organization. According to the thresholds set by the organization, the ratios can be divided into five classes. The limits of classification are based on subjective choice and can therefore be interpreted as fuzzy logic. The KPI is classified into one of the five predefined classes based on the value of the plan–fact analysis ratio.

The second of the standardized norms (ST 2) was formulated by us. This standardized norm is also classified into five classes based on the value of the plan–fact analysis ratio. In the analysis, the five classes and their thresholds do not change compared to ST 1. In this case, however, the factual data are the results of the period under review, while the plan data are the plan data of the previous period. In the present case, the factual data are the values for the period from 1 May to 30 September 2020. The plan data are the values for the period from 1 May to 30 September 2019.

Using the third standardized norm (ST 3), we point out that indicators classified according to ST 1 can be assigned to a different evaluation class by changing the standardized norms, with the same threshold values and plan–fact analysis ratios. Thus, it can be stated that the assessment of ratios is not necessarily clear.

We chose the ratio to the weighted average as ST 3. At the second level, the aggregate value of the KPI of the given project portfolio is the fact data, which is equal to the weighted average of the given KPIs of the projects in the project portfolio (first level). The plan value is the weighted average of the values of the analyzed KPIs for all project portfolios. Determining the weights of KPIs makes it possible to evaluate the effectiveness of KPIs within the project portfolio and the aggregated KPI. The weightings of KPIs are determined based on the subjective opinion of the strategic and functional managers of the project-oriented organization. We measured the values of KPI weights using a questionnaire method. The KPI weights thus developed are the same for each project. At the third level, similarly to the previous level, the weighted average of the values of the analyzed aggregate indicator and aspects represents the plan value. This plan value is compared to the weighted average per project portfolio for the KPI analyzed, which in this case is the actual value. At the fourth level, the aggregate values for the different BSC perspectives represent the actual value, which is the weighted average of the corresponding KPIs. The plan value is the value of the strategy index at the fourth level, which is the weighted average of the results of the four different BSC perspectives. At the fourth level, due to the structure of the analysis and the model, the results cannot be evaluated according to this standard norm.

The classification of KPIs along different standardization norms indicates that the indicator may be assigned to a different assessment class with the same plan–fact analysis ratio value (ST 1, ST 3) and threshold values (ST 1, ST 2, ST 3). (Table 1).

The function used to classify:

$${\sigma }_j=\frac{\sum \frac{A_{ji}}{{{\rm N}}_j}\times {\xi }_i}{K}$$

(2)

where A: the actual value of the KPI, N: the predefined plan value, ji: the serial number of the examined element, K: the number of examined elements belonging to the KPI/BSC aspect, and ξ_i: derived value of weight.

The organization defines five different classes to evaluate the effectiveness of its indicators.

$${{\rm T}}_j\left\{\begin{array}{c}\mathrm{Very}\mathrm{underperforming}\\ \mathrm{Not}\mathrm{acceptable}\\ \mathrm{Acceptable}\\ \mathrm{Appropriate}\\ \mathrm{Distinguished}\end{array}\right.\begin{array}{c}\mathrm{if}{\sigma }_j<-\alpha \\ \mathrm{if}{\sigma }_j\in \left[-\alpha ;1)\right.\\ \mathrm{if}{\sigma }_j\in \left(1;\alpha \right)\\ \mathrm{if}{\sigma }_j\in \left(\alpha ;\beta \right]\\ \mathrm{if}{\sigma }_j>\beta \end{array}$$

(3)

$${{\rm T}}_j\left\{\begin{array}{c}\mathrm{Very}\mathrm{underperforming}\\ \mathrm{Not}\mathrm{acceptable}\\ \mathrm{Acceptable}\\ \mathrm{Appropriate}\\ \mathrm{Distinguished}\end{array}\right.\begin{array}{c}\mathrm{if}{\sigma }_j<0,95\\ \mathrm{if}{\sigma }_j\in \left[0,95;1,0)\right.\\ \mathrm{if}{\sigma }_j\in \left(1,0;1,05\right)\\ \mathrm{if}{\sigma }_j\in \left(1,05;1,1\right]\\ \mathrm{if}{\sigma }_j>1,1\end{array}$$

(4)

Using the fuzzy function (Figure 3), the firm can evaluate its effectiveness using the five classification categories. The function allows the controlling system to serve as an indicated feedback function for the business.

The function is used as a computational methodology to evaluate and classify different project portfolios, KPIs, BSC perspectives, and the strategic performance index. The classification is performed along linguistics terms; these linguistic terms were defined by the managers of the project-oriented organization; and in the present research, we use these classes and the thresholds belonging to the classes. When applying the linguistics terms of the classes, it is not the value taken on the scale but the threshold values and standardized norms that determine them.

3.3. Step 3: Evaluate Aggregates, Created Based on Subjective Expert Opinion, Based on Selected Standardized Norms

Table 1 lists the projects as aggregates of three different project portfolios. Based on the analysis of the project portfolios, it can be stated that a significant part of the projects belonging to the given project portfolios were assigned to different evaluation classes with the same thresholds and ratios based on different standardization norms. This result demonstrates that the evaluation of results is not clear, subjective, and benchmark dependent. In the case of organizational project portfolios, in 2019 and 2020, they do not show an important difference in the fulfillment of the examined KPIs. Overall, extreme classes (Very underperforming, Distinguished) can also be interpreted as intervention points. The plan values for these KPIs need to be reviewed. During the review, we may receive feedback on the unreality of the plan or on the actual extreme performance of the KPI fact value. During the evaluations according to ST 1 (2019) and ST 3 (2019), it can be stated that the values of the plan–fact analysis ratio of the KPIs within the given project portfolio have almost the same classification. This means that KPIs performed similarly relative to their own plan values (ST 1 2019) and also to average performance (ST 3 2019). In the case of Project Portfolio 3, a significant difference can be observed in the analysis of ST 3 (2019–2020) compared to ST 2 (2019–2020). This means that the majority of the KPI performances of 2020 P3 (KPI 3-CPS, KPI 8-Cost of consulting services, KPI 13-Proportion of missed milestones, KPI 17-Proportion of digitized documents, and KPI 23-Average complaint processing time) show a positive change in the contribution to the average target values of P3 in 2019. Based on this, it can be stated that the results of 2020 would have resulted in a different level of contribution to the 2019 P3 objectives in many cases.

3.4. Step 4: Evaluate the Aggregated Results of the Selected KPIs Based on the Selected Standardized Norms

The KPI results included in the analysis (Table 2) are the average of the project portfolios, which have already been corrected by the weight values of the project portfolios. Table 2 shows the aggregated value of KPIs for all projects. In the analysis of these aggregate values, it can be stated that the values of KPIs are assigned to different valuation classes for different standardized norms, with the same thresholds. In the case of the ST 1 (2019) and ST 1 (2020) analyses, it can be seen that the actual value of a KPI does not have a negative (very underperforming) classification compared to the pre-defined plan values. Thus, the processes and areas measured by the KPIs did not show a markedly negative performance over the period under review based on ST 1 analyses. In the ST 1 (2019) analysis, KPI 24 (Proportion of returning customers) has a positive extreme class. Based on this, it can be concluded that the company significantly exceeded its objectives associated with the KPI. In the case of the ST 1 (2020) analysis, several KPIs (KPI 2-Profitability, KPI 14-Proportion of projects made, KPI 15-Proportion of canceled projects, KPI 16-Suspended projects, KPI 20-Quote conversion, and KPI 25-Customer acquisition cost) falls into this class. Based on these indicators, it can also be assumed that the performance in 2020 would be more effective compared to the results of the previous year. This assumes more efficient performance in 2020 compared to 2019. However, based on the results of the ST 2 (2019–2020) analysis, the actual values for 2020 did not perform outstandingly compared to the plan values for 2019. Thus, these data adequately illustrate that the assessment of 2020 performance was influenced by expectations of the negative economic impact of COVID-19. The values of the KPIs in the ST 3 analyses are the value adjusted for the weightings of the project portfolios and the given KPI. No significant difference is observed in these analyses; therefore, the evaluation according to this standardized norm does not change the assessment of performance compared to other standardized norms.

The analysis confirms that as a result of the pandemic, the organization significantly reduced its targets in its 2020 planning. The 2020 actual data thus performed similarly to the 2019 plan values. Based on this, it can be concluded that the effect of COVID-19 on the functioning of the organization is not clear.

3.5. Step 5: Define and Evaluate Aggregates for Balanced-Scorecard Perspectives Based on Different Standardization Norms

The BSC perspectives (Table 3) used to analyze an organization’s strategic effectiveness have detailed information content to assess organizational performance. After analyzing the four perspectives, the indicators that fall into one of the extreme (Very underperforming, Distinguished) classes based on the classification of each standardized norm can be explored. Furthermore, exploring the reasons for changes in class-changing indicators based on the results of the analysis can also provide relevant information.

In the analysis of the financial perspective, a change of class can be observed in the case of ST 1 (2020) compared to ST 1 (2019). Among the KPIs that make up the finance perspectives, KPI 1 (ROE), KPI 2 (Profitability), KPI 5 (ROS) and KPI 6 (Employee efficiency) should be highlighted in the more detailed analysis. The change in these indicators (Table 2) resulting from a positive plan–fact analysis significantly influenced the change in the financial aspects class. Among the listed indicators, KPI 2 (Profitability) should be highlighted, which is also the most significant (distinguished) contribution to the average performance of the 2020 financial perspectives during the ST 3 (2020) analysis. The improvement in the performance of the financial aspects, however, is not clear as the analysis in (Table 3) ST 2 (2019–2020) shows that the actual value for 2020 met the plan values for 2019 in an “Acceptable” manner. This illustrates that as a result of COVID-19, the company reduced its expectations, which it exceeded despite predestined negative macro conditions. Thus, judging a firm’s financial performance during a 2020 pandemic period depends on the analytical context.

Dependence on the same analytical context can also be observed for customer perspectives. There was a class change similar to the financial aspect for ST 1 (2020) compared to ST 1 (2019). Among the KPIs, KPI 20 (Quote conversion), KPI 24 (Proportion of returning customers) and KPI 25 (Customer acquisition cost) have the most significant influence on the class change of the indicator. These indicators also show a reduction in expectations as a result of the pandemic. This effect is most noticeable in the case of the quote conversion. However, the 2020 “Appropriate” performance can only be categorized as “Acceptable” compared to the 2019 plan values.

In the case of internal business perspective, the change of class is different from the previous two perspectives. In ST 1 (2019), aspects are classified as “Not Acceptable”, while in ST 1 (2020), they are classified as “Appropriate”. This significant class change is due to the influence of the change resulting from the positive plan–fact analysis of KPI 14 (Proportion of projects made), KPI 15 (Proportion of canceled projects), KPI 16 (Suspended projects) and KPI 17 (Proportion of digitized documents). Among the indicators, KPI 14 (Proportion of projects made) and KPI 15 (Proportion of canceled projects) should be highlighted. The values of plan–fact analysis ratios for these indicators were also influenced by the effect of COVID-19. The implementation of unique and customer-oriented travels and the cancellation of travels significantly exceeded the plans. This perspective (Table 3) also changed class compared to 2019, which means that the actual value in 2020 exceeded the plan value in 2019.

Learning and growth perspective are an exception to the other perspectives because there is no class change for any of these standardized norms. Among the KPIs that make up the aspects, KPI 7 (Labor turnover) should be highlighted. The value of this indicator decreased less than expected due to external environmental factors. The company’s workforce retention capacity in 2020 compared to its 2019 plan values is “Acceptable”.

3.6. Step 6: Define and Evaluate a Strategic Performance Index Based on Different Standardization Norms

The strategic performance indicator created (Table 4) expresses how the company meets its strategic goals. The index expresses strategic performance in an indicator. The actual value of the indicator is the sum of the ratio to the average performance of the BSC perspectives (ST 3). We classify the actual value calculated in this way based on ST 1 and ST 2. Classifications based on ST 3 are not covered in our research, as in this case the industry average or several years of internal company data would be required. These analyses are inconsistent with our current research goal.

Based on the analysis of ST 1 (2019), the strategic performance of the company is classified into the “Acceptable” class. In 2020, on the other hand (ST 2 2020), the indicator changed class and placed in the “Appropriate” category. This means that the company will perform well in 2020, depending on the strategic goals. Among the BSC perspectives (Table 3), the indicators Financial, Internal business and Customer contributed significantly to this organizational effectiveness.

The company’s performance in 2020 exceeded its plans, but the assessment of the period can only be considered “Acceptable” based on the analysis of ST 2 (2019–2020) (Table 4). The assessment of how well a company has performed in the period under review in 2020 varies along the standardized norms examined. The results achieved along different norms express the evaluation of organization performance with the same realistic approach. In this model, we do not use fuzzy logic to form an accurate classification with real content. The purpose of fuzzy in the model is not to establish this exact classification either. The aim is to show that judging the effectiveness of a business depends on the context.

Overall, our model points out that the presupposed negative effect of COVID-19 on the performance of the organization appeared during planning. However, during the actual performance evaluation, these expectations were significantly exceeded by the examined company; therefore, depending on this context, the company may use its “Appropriate” indicator as an indicator of its effectiveness. However, compared to the 2019 plans, it is only “Acceptable”. Assessing company performance in 2020, so based on these, is a subjective judgment of company managers.

4. Discussion

The differences in the measurement of organizational performance and their criteria raise the question of what effectiveness does the evaluation by a given performance appraisal system actually assume. This issue is the subject of many studies on measuring the performance of tourism businesses [48]. Although travel agents play an important role in the sale of tourism products, it should be emphasized that only a few studies focus on assessing their effectiveness and performance [49]. In recent years, the efficiency of travel agencies has been most often evaluated using nonparametric DEA techniques [50,51,52]. In their study, Barros and Matias (2006) examine the efficiency of 25 Portuguese travel agencies using a cost-boundary model. In their model, relative efficiency is determined based on several input and output indicators. Based on the results of their analysis, it can be stated that capital, labor, sales and marketing activities are the ones that determine efficiency in the sector [53]. In addition to evaluating the efficiency of travel agents, there is no generally accepted method for analyzing performance evaluation. Sainaghi et al. (2013) in their studies suggest an alternative approach to performance evaluation. This alternative approach focuses primarily on qualitative performance evaluation methods, as opposed to the financial aspect [54]. Another category of organizational performance assessment is analysis based on financial and accounting indicators [55]. These traditional financial and accounting analyses focus only on the analysis of financial indicators. However, this approach does not express either organizational strategic or complex effectiveness. Furthermore, it does not consider social, cultural and sustainability perspectives [56]. The BSC model developed by Kaplan-Norton (1992) provides an opportunity to evaluate the tourism sector from a controlling perspective through different perspectives [57]. The model constructed in our research therefore implements the BSC perspectives. This approach is consistent with Atkinson and Brown’s (2001) study examining performance appraisal in the UK hotel industry. Based on the results of their research, it can be stated that within this sector, most of organizations operating in the industry evaluate performance on the basis of financial aspects. However, in their study, the authors emphasize the importance of other aspects of BSC in performance evaluation [58]. In our research, as well as with their study, the financial point of view is formulated as the most emphasized aspects. However, based on Assaf and Josiassen (2012), it can be stated that the exact definition of the performance of tourism enterprises and the relative importance of the factors influencing performance are not clear [59]. Based on these findings, the determination of the relative importance of KPIs and BSC perspectives used to evaluate performance in our research was determined based on the subjective opinion of the managers of the examined enterprise.

Fuzzy logic is a widely used method in organizational performance evaluation. Its application can be observed in many fields [25,60,61,62]. However, it is not a common method for measuring the performance of tourism organizations. Its application occurs in the selection of different criteria in the sector. This was described and confirmed by Lin et al.’s (2009) study. According to the authors, fuzzy AHP is a suitable method for determining the relative importance of factors involved in evaluating the performance of a travel intermediary service. In their study, the various indicators are weighted and ranked based on the opinions of 36 tour top managers and 56 operational managers [63]. In our study, fuzzy logic appears in several aspects: on the one hand, in defining the weights of project portfolios and KPIs, and on the other hand, in defining classification thresholds. It also appears for setting reference values. The organization included in our case study operates as a project-oriented organization, treating the organization of unique-customer-oriented travels as independent projects. The fuzzy methodology is also widely used in the performance evaluation and controlling activities of projects and project portfolios. Its most common occurrence in the literature can be observed during the risk assessment of constructions projects, the selection of project portfolios and the evaluation of project performance [40,64,65,66]. The logic of the project controlling activity based on this fuzzy logic is the same as the controlling model we created. However, due to the fuzzy logic, the disadvantage of our model is that it does not define exact values but gives the values and classes of indicators as fuzzy numbers. Therefore, in terms of formalizing inferential processes, the model provides only approximate answers in the analyses [34]. In order to overcome the disadvantage of the model, neural network and Bayesian-logic-based models are also used in the analyses, performance evaluation and controlling systems of the tourism sector [67,68,69,70,71].

5. Conclusions

In our research, we illustrate a fuzzy-logic-based model through an extended case study that enables project-oriented organizations to effectively monitor their performance. Our case study analyzes the operation of a company that organizes unique and customer-oriented travels. Using the data of the examined company in 2019–2020, we illustrated the operation of the controlling model created by us. In our research, we structured the KPI already used by the company along the BSC perspectives. Based on these findings, we created a hierarchical model, with the help of which it became possible to detect the evaluation of the operation of the organization along different standardized norms.

Unlike performance evaluation models, this controlling model is based on non-benchmark analyses and takes into account the subjectivity derived from measurability and goal setting. Subjectivity from measurability and goal formulation is treated by the model by evaluating along different internal standardized norms. This allows for analysis that takes into account different contexts relative to organizational goals. Using the model, it may be possible to evaluate unique projects according to different standardized norms. The thresholds of the three standardized norms we use do not change, but the reference values are different in each case. By applying the plan–fact analysis ratios used in the research, it becomes possible to standardize the results of the KPIs related to the projects.

Our results indicate the impact of COVID-19 on performance for the organization under study. Our research revealed that the pandemic fundamentally changed the company’s planning for the period under review compared to its plan values for the same period in 2019. Using KPI groupings according to different BSC perspectives, we illustrate that the financial perspective, which is the most emphatic for the company, had its results ranked better in 2020 compared to 2019. Based on the ST 3 analysis, however, the classification of the same 2020 fact data highlights that although the company has an “Appropriate” classification in the pandemic year (2020), this can only be assessed with an “Acceptable” classification compared to the 2019 plan data. This means that the company under investigation rates its performance better during the 2020 pandemic situation, along similar financial results. From an internal business perspective, it can be seen that the company performed better in 2020, regardless of planning. Based on these findings, the question arises whether the more efficient operation of the organization was caused by the company’s internal decisions or the positive macro-environmental changes. Examining the effectiveness of the organizational performance of the strategy, the outstanding “Appropriate” result in 2020 did not achieve an outstanding result compared to the 2019 plan data. Further research would be needed to explore which standardized norm classification is more relevant for managers. As a result of this research, the performance classification ability of the model can be made more accurate.

Based on our results, it can be stated that in a deconjunctural economic environment, the assessment of the performance of companies always depends on the research context. COVID-19 has had a significant negative impact on the tourism industry, which has also significantly affected the real performance appraisal of companies. Therefore, by evaluating the performance of tourism organizations along internal organization goals, more relevant information content and more informed management decision support can be achieved. In our model, negative expectations are incorporated into the goals when evaluating as a function of internal companies goals. Thus, the result of the model evaluates the real performance across different contexts compared to the expectations of the macro conditions resulting from COVID-19.

One of the most significant limitations of the model is that, due to fuzzy logic, the model cannot be applied with extreme values. Based on fuzzy logic, no clear answers can be given, only approximate results can be achieved. Due to internal standardized norms, an additional limitation is that the model is not suitable for industry comparative analysis. The lack of exploration of causal relationships between the applied KPIs and functional areas can be mentioned as a limiting factor. The disadvantage of setting target and weight values in a subjective way appears in the model. These values have a significant impact on the assessment of an organization’s performance in the model. Thus, reducing subjectivity increases the accuracy of model evaluation.

The model does not provide a clear answer to the fact that the changes in organization profitability during the pandemic period under study occurred only due to the effect of COVID-19. On the other hand, in our model, the appearance of the effect of COVID-19 can be clearly observed during the formulation of goals and expectations related to organization performance. By incorporating expectations into the performance assessment, the expected economic changes influenced by COVID-19 are also reflected in the model. However, a number of other influencing factors may have had an impact on the changes examined. The model can be further developed with performance indicators for other strategic and functional areas. This is supported by a study by Yeon et al. (2021), which draws attention to the need to pay attention to the influencing effect of other factors in the performance evaluation of a company during the COVID-19 period [5]. These influencing factors and different company characteristics need to be considered in order to effectively explore the effect of COVID-19. By examining the indicators for these functional areas and their causal analysis, the influencing factors that influence the studied changes in addition to the effect of COVID-19 can be explored. Among these factors, it may be worthwhile to include in a more detailed analysis the changes in communication, marketing channels and organizational management strategies. Another research opportunity is the implementation of the model we have developed for the operational processes of other project-oriented organizations and to expand the model by taking into account the specifications of regional and enterprise development projects. We also recommend the use of data from the tourism sector and unique travel companies as a benchmark of a new standardized norm. It can also be an appropriate way to monitor the sustainability of tourism organizations by involving different sustainability indicators.

6. Implications of the Study

Our study can serve as a standard model for tourism organizations operating in a project-oriented structure that can be used to evaluate effective organizational performance. The created controlling model is able to reveal the intervention points at different hierarchical levels. Classifications according to different standardized norms may indicate different interpretations of the results of the same indicators. This difference in interpretation is also illustrated by the results of our study, in that the assessment of the economic impact of COVID-19 on the performance of the studied organization is not clear.

The model also makes it possible to compare the results of different periods. It supports more accurate, appropriate decision making for the managers of the organization at all strategic, operational and functional levels. In our research, we recommend the use of three generally applicable standardized norms. Along these norms, the results of the examined indicators can be evaluated in different contexts. This provides a significant amount of information for travel agencies to make their decisions. Not only short-term but also long-term strategic decisions are supported by the model, as it standardizes project performance using plan–fact analysis. In this way, it is able to express organizational strategic performance in one indicator.