1. Introduction
Due to the COVID-19 pandemic, quarantine has forced institutions to reorient and reorganize the entire education system in a short period of time. The biggest part of the Military Academy of Lithuania (MAL)’s education process (subject, social, even physical) was transferred from the direct from into the form of distance education. This sudden change caused by unforeseen conditions affected the interactions, involvement, and roles of the participants in the educational process, both teachers and cadets, in the educational process, the educational environment, the content, and the nature. This situation forced the MAL to remember and apply the already well-known experience of distance learning and to look for criteria for improving the quality of distance learning courses.
As is well known, distance learning is generally perceived as a way of learning where the learner’s geographical location differs from that of the geographical location of the education institution [
1]. The development of distance learning has witnessed gradual progress. Typically, new technologies for knowledge transfer have penetrated different distance learning systems and their development has promoted the emergence of modern distance learning courses. Many researchers classify the history of distance learning modules (DLMs) in terms of so-called generations of distance learning [
2,
3].
First-generation distance learning courses mainly refer to handwritten or printed handouts that were sent to learners via postal mail. This form of DLM has been expanding in all developed countries since the mid-19th century. With the advent of radio in the 1920s and television in the 1950s, new information and communication technologies came into use, including DLMs in the form of developing and broadcasting radio and television training programmes. The year when the British Open University was established, i.e., 1969, is considered to be the beginning of the second generation of the DLM. It was then that multimedia entered into service. The British Open University was famous for producing high-quality learning materials specially designed for distance learning. The third generation is characterised by the use of interactive, electronic and other computer-related technologies. Computer systems provide a two-way communication channel, both synchronous (simultaneous communication via video/audio conferencing) or asynchronous (communication at different times using e-mail, discussion forums). Despite the fact that computerised distance learning was implemented in several educational institutions in 1980, it gained popularity only with the advent of the World Wide Web. The spread of the Internet contributed to the popularity of DLM due to the ability to communicate quickly and flexibly and to present learning materials interactively using Web technologies.
The Dictionary of Contemporary Lithuanian defines “a course” as “a one-year study unit in a higher education institution or special secondary school or the delivery of a subject in a higher education institution” [
4]. The form of distance learning is applied in both higher education and non-formal adult education. This paper analyses distance courses delivered in higher education institutions. Thus, a distance course can be defined as a course unit delivered remotely using information technologies. According to [
2], “studies” mean personal learning, when someone who has finished at least his education in the secondary school continues his studies in a higher-level education institution in a particular study programme.
Analysis of e-learning and distance learning has a prominent place in Lithuanian research. Many Lithuanian researchers have devoted a considerable amount of work to the area of learning community and second-generation web (Web 2.0) technologies [
1,
5,
6]. Subsequently, analysis of Web 3.0-based individualisation of learning objects in simulated learning–teaching environments has been carried out [
7]. Lithuanian researchers have proposed and used in their works quality control methods for training software [
1] and quality evaluation for learning scenarios [
7]. The issue of software localisation has also been addressed in a number of research studies [
5]. In addition, digital learning objects have been analysed, extending their metadata model [
8]. The use of reusable learning objects has been explored, seeking methods to separate e-learning materials from display rules, thus improving the use of reusable learning objects [
9]. A number of researchers have published their findings on the importance of virtual learning environments in e-learning [
10,
11,
12,
13].
Advantages and disadvantages of e-learning and blended learning related to the use of the virtual learning environment Moodle for blended learning have been analysed by Stonkienė [
8]. A number of research studies have also been carried out to explore the importance of e-learning and visualisation of user actions as well as learners’ activity and analysis of their preferences [
14,
15]. Researchers have further explored the architecture of the virtual learning environment to increase options for individualisation and investigated actions taken by software agents in a virtual learning architecture based on Q-learning algorithms, with the virtual learning environment playing the curating role. A number of studies have been carried out on the integration of smart modules into the capabilities of the virtual learning environment Moodle and the implementation of personalised learning depending on certain learning styles [
16,
17]. In addition, many research studies have been conducted to explore the choice of learning scenarios and to analyse the components of e-services in virtual learning environments [
18]. Researchers have developed and integrated into the Moodle platform a model of grades transfer from one scale to another [
19].
The aim of this research is to identify the cause-and-effect relationship between the quality of the content of a distance learning module and its technical feasibility using the expert group’s aggregated decision. Group decision making (GDM) is more multifaceted compared with individual decision making, since it includes a number of challenging aspects such as incompatible personal purposes, ineffective learning, validity of communication, individual incentive, and private opinions. The GDM problem lies in combining the opinions of multiple experts into a decision of a certain single group. In this research, each group’s decisions were pooled in the group’s aggregated decision. Given that expert evaluation was independent, each expert produced his/her individual set of evaluation criteria selected for the research independently of other experts. These individual judgements were subsequently combined and eventually presented as a set of evaluation parameters of a separate expert group. On the basis of this set, a multi-attribute decision-making approach was applied to express the opinion of and decision made by a certain group [
20]. In order to facilitate the systematisation of the methodology used for distance learning quality evaluation, the research employed a scheme presented in
Figure 1. The research scheme illustrates in detail the structure of the research process, which was followed to analyse the quality of three distance learning modules and to identify key criteria.
The conducted study’s purpose and results are represented in seven sections. The relevance of the problem and a short overview of the history of distance learning according to our researches investigations is presented in the introduction. The representation of the problem of distance study quality assessment in detail is discussed by a literature review in
Section 2.
Section 3 describes the methodology that was chosen to solve the multi-criteria problem for decision making by Fuzzy Technique used to Order Preference by Similarities to Ideal Solution (FTOPSIS) procedure when the Triangular Fuzzy Numbers are applied to give proper weightage to criteria by the Fuzzy Analytic Hierarchy Process (FAHP). The problem of qualitative data in the TOPSIS model is discussed in
Section 4. The numerical examples showing how the applications of the use of fuzzy set theory with the MCDM technique help decision-making to deal with indistinctness and ambiguity are explained in
Section 5. Lastly, the discussions and some concluding explanations are made in the last two sections,
Section 6 and
Section 7.
2. Materials and Methods
The quality of e-learning can be linked to all learning and educational processes, outcomes, and services provided through information and communication systems [
18,
19,
20,
21]. The quality of distance learning can be improved during subject design through the purposive selection of IT tools for the implementation of the distance learning process and during subject delivery through efforts to increase the efficiency of the virtual learning environment. A number of significant research studies have been conducted in the area of quality assurance in distance learning, focusing on various aspects of such education. Many researchers have analysed the impact of technological change on the quality of distance courses, as the main focus in developing a distance course module and providing training is on the development of learning material. This is an important moment in the whole learning process [
22,
23]. Therefore, researchers have analysed in detail and presented a specific model for designing distance learning curricula based on quality assessment factors of distance learning content [
24]. One of the factors for assessing the quality of distance learning curricula design identified by the authors is the development of a student assessment strategy. In addition, peculiarities of quality assessment of distance learning content were analysed with the use of expert interview data [
25].
With regard to the regularity analysis of planning and organising distance (further vocational) learning content, researchers have emphasised that adequate training of teachers/lecturers, supervisors, administrators, managers, and consultants at pre-learning, learning and post-learning stages are factors that play an important role. In this context, research suggests that achievement of learning content quality requires a regular analysis of learners’ preferences [
26], setting learning objectives, application of appropriate learning organisation methods, proper planning of the assessment of learning outcomes, and selection of appropriate curriculum-specific technological means [
27].
To assess the quality of distance learning, researchers also used Quality Function Deployment (QFD), consisting of interactions between customer preferences and quality criteria of alternatives [
28]. The Quality Function Deployment was proposed by a Japanese scholar in 1996 [
29]. Another method explored and proposed by researchers is a method for selecting and evaluating e-learning products, which involves evaluation and ranking of learning products and alternatives with respect to target product characteristic values [
30] in evaluating the quality of services in e-learning [
31].
Evaluation of teaching employs various models. Some researchers applied a neural network model based on the optimisation approach [
32], while others conducted research based on the Fuzzy Analytic Hierarchy Process (FAHP) [
33] to evaluate teaching quality in the classroom. In evaluations of distance learning quality, researchers emphasised the learner’s perspective as the most important factor, since the quality of the delivered module directly depends on the learner’s expectations [
34]. This approach generated several tools for Student’s Evaluation of Teaching Effectiveness (SETE), including Student Instructional Rating System (SIRS), Instructor and Course Evaluation System (ICES), Student Description of Teaching (SDT), Students’ Evaluations of Educational Quality (SEEQ), and Instructional Development and Effectiveness Assessment (IDEA) [
35].
The literature review shows that effective teaching using web-based remote systems depends on many factors, also known as criteria [
36,
37,
38,
39,
40]. Therefore, it is not a simple task to develop an easily perceived hierarchy of criteria—a model that reflects the main purpose of the task. After research, certain criteria selection and grouping methods and model-making principles are proposed that could facilitate modelling. Drawing on the research [
41] analysing the principles of the structure of models (criteria) for evaluating the quality of multi-criteria decision-making analysis, evaluation criteria can be selected taking into account the following important aspects:
Linking concepts with their goals (value relevance): decision-makers should be able to link the key concepts of expert evaluation in the appropriate area with the purpose of the evaluation.
Equal understandability: all decision-makers should have a shared understanding of the evaluation criteria.
Measurability: the criteria need to be realistically measurable in practice; they should be explained by sub-criteria and quality compliance criterion levels.
Non-redundancy: the same aspect (factor) should not be measured by several different criteria.
Judgement independence: decision-makers should evaluate alternatives separately with respect to each criterion but keep in mind that there are links between the criteria; i.e., criteria form a framework.
Balancing completeness and conciseness: there should be neither too many nor too few criteria, and they should not be too large or too detailed; i.e., they should capture all the main aspects of the alternative without going into too much detail. In other words, the model should not force decision-makers to evaluate the quality of alternatives blindly according to formal sub-criteria. The decision-makers should understand the requirements of the criterion as a whole and judge accordingly.
Operationality: the model should be applicable in practice without taking an unreasonable amount of time.
Balancing simplicity and complexity: notwithstanding the complexity of the problem of quality assessment, the modeller should provide decision-makers with a simple and clear criteria tree [
41].
The Consistent Fuzzy Preference Relations (CFPR) method could be used for ranking and selecting the most important criteria for study quality assessment model (i.e., establishing consistent fuzzy preference relations). The CFPR method evaluates the effectiveness of the criteria for effective distance learning and reduces the number of questions in the questionnaire and the number of similar criteria and prevents inconsistencies [
42].
Drawing on the researchers’ insights emphasising that all researchers focus on quality assessment methods while not giving due attention to the quality assessment model, the validity and reliability of research findings should be therefore sought with a focus on constructing the Multi-Criteria Decision Making (MCDM) model itself. For this reason, researchers began to construct integrated models using the aspect investigation and Decision Making Trial and Evaluation Laboratory (DEMATEL) method. The DEMATEL technique has received reasonable interest, as it allows identifying the relationships between the criteria and determining the main criterion, which is the principal one to represent the effectiveness of the group of criteria [
43,
44].
To justify the choice of the MCDA method, we would like to turn the reader’s attention to, e.g., the effectiveness of methods for determining the significance of standards in supportable transport difficulties; see e.g., [
44,
45].
Highlighting criteria sensitivity analysis, Thomas L. Saaty for different criteria evaluation proposed the Fuzzy Analytical Hierarchy Process (AHP) [
46], which is one of the more popular MCDM methods, because a set of factors can be shaped and logically assessed taking into account their importance. Alternatives can be evaluated based on these factors and obtain a final mark that replicates this importance rank.
There is no agreement on how to regulate the sensitivity investigations, i.e., the precision of a decision technique and the consistency of the outcomes. The sensitivity investigation can be defined as stability or performance of the solution to minor variations in preferences, which happen throughout the determination process or to minor variations in the values taken for measures; this is what some research studies call effectiveness multicriteria decision analysis, e.g., Pamučar and Ćirović [
45]. Notwithstanding a large number of established and new methods, the challenge of multicriteria choice is still not trivial. Subsequent to the outcomes achieved, the assessment of alternatives according to the criteria and the choice of the criterion for ranking alternatives using dissimilar ranks have a reflective effect on the concluding choice.
The Fuzzy Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) is the other analysis that is similarly widespread. In TOPSIS analysis, criteria of the alternatives are associated with that of an ideal. TOPSIS replicates a normal human thought process where people judge things not by certain criteria but by an ideal example of the same type [
47,
48,
49].
Because the quality of the remote module depends on many factors such as clearly presented and interestingly arranged material, well-organized teaching process, appropriate IT tools used, relevance of module material, student motivation and teacher/lecturer qualification and professionalism, all of these criteria were included in the study and all chosen criteria were evaluated by experts in the relevant field. Moreover, the Fuzzy Analytic Hierarchy Process (FAHP) and Fuzzy Technique for Order of Preference by Similarity to Ideal Solution (FTOPSIS) is used to identify the best distance learning module (DLM). The AHP was chosen to identify the crispified value of factors [
46]. Inaccurate linguistic terms used by the experts were transformed to Triangular Fuzzy Numbers. These investigations’ idea was to rank the distance learning modules using fuzzy numbers for criteria weights and by using FAHP to FTOPSIS ranking.
It is common knowledge that the AHP method has a problem with the rank reversal phenomenon. To prevent, it the authors used the methodology presented by Wang and Elhag in [
39]. They explained the origin of rank problem phenomenon and how that rank exchange is caused by variation of local significances before and after an alternative is added or deleted. It is recommended to consider the local priorities unchanged to obviate the rank exchange phenomenon.
A wide range of multicriteria decision methods have been proposed by researchers, but all these methods primarily use the same evaluation table, and only then their analysis background can vary according to the additional information they are looking for. The PROMETHEE analysis methods necessitate very clear supplementary data that are found simply and understood by both decision-makers and consultants.
In the PROMETHEE computer software PROMCALC and DECISION LAB, the analyzer can easily express the relative importance of the criteria, because it allows one to present agreed-upon values for the criteria weights. These values are then divided by their quantity so that the weights are normed routinely. Measuring weights to the criteria is not simple. This process involves the priorities and observations of the decision-maker. PROMCALC and DECISION LAB take in numerous sensitivity tools, which can be involved in a different set of weights in order to help to fix them. Additional information can be found in Goumas and Lygerou’s paper [
40].
When solving the optimization problem while evaluating the distance learning courses, the values of the criteria weights are determined by experts and are not changed. Therefore, it became necessary to use multi-criteria methods such as SAW, TOPSIS, COPRAS, MOORA, and PROMETHEE, which use constant expert estimates in the calculations.
The use of this priority function does not highlight the advantages of one alternative, as the values from q to s increase linearly. PROMETHEE, like supplementary multi-criteria analysis, supports the principles of the SAW method, uses the values of specially selected priority functions instead of the normalized values of the criteria [
20,
33,
50,
51].
The study methodology and methods are presented in detail in
Section 3.
3. Research Methodology
It is a characteristic of many tasks that there are insufficient numerical data to solve them or that the research object is impossible to measure. In such cases, experts are relied upon to complement data with their judgements. Valid decisions must be based on experience, knowledge, and intuition, since the degree of quality should be determined on the basis of expert judgements by professionals qualified in the field at issue. Such people are required to have specific knowledge and skills in particular fields. In the process of research, the problem of selecting experts is one of the most difficult, both theoretically and practically, because persons evaluating one or another factor may differ in their competences and values. Therefore, for the purpose of this research, experts were selected on the basis of their characteristics related to their professional competence: work experience, length of service, academic degree and research activity, and the ability to solve specific problems in the respective field [
21]. Interviews with selected experts employed different methods of interviewing: implicit expert method in terms of interrelationships between experts, one-off interviews in terms of judgement synchronisation procedure, and one-to-one interviews in terms of the number of experts. Research-based principles were applied to define groups of criteria for the quality of distance learning courses [
41].
3.1. The Construction of a Decision-Making Matrix
Responding to the individual or group decision problematic with a multi-criteria background, the analysis starts with the creation of a decision-making matrix or a few matrices. These matrices represent the values of the criteria for assessment alternatives chosen. Criteria values, according to researchers’ choice, can be real, interval numbers, fuzzy numbers, or qualitative labels. Accordingly, criteria values can designate a decision-maker’s expert judgement set of values D = {1, 2,…, K}. The multi-criteria values can be transformed into a
k matrix:
| C1 | C2 | | Cn |
A1 | | | | |
A2 | | | | |
| | | | |
Am | | | | |
where:
- -
are feasible alternatives that decision-makers have to select from,
- -
are the criteria for which the alternative feature is judged,
- -
is the decision-maker ’s score of alternatives with connection to the criterion ( is a numerical, interval data, or fuzzy number). According to this rule for alternatives and criteria, the matrix can be drawn, where is the value of alternative taking into account the criterion for the decision-maker k, .
The comparative rank of each criterion can be calculated by a set of weights that are normalized to sum to one. The weight vector can be represented by for decision-maker k, where is the decision-maker k’s weight of criterion and .
Finally, it can be pointed out that multi-criteria methods principally emphasise three types of decision difficulties: to make a choice when the most appropriate (best) alternative must be selected, to make a ranking to explain the order of the alternatives from the best to the worst, and to make the categorization when the best k alternatives must be from a list.
3.2. The Construction of Fuzzy AHP
Researchers advise a wide range of different methods for criteria weights assessment [
52], but the most used is the AHP (Analytic Hierarchy Process) developed by Saaty [
46]. The fuzzy theory Buckley [
53] combined with AHP creates the fuzzy AHP (FAHP). The method for finding fuzzy weights, based on a direct fuzzification, was designed by Saaty and is described in his work [
46]. The fuzzification procedure of FAHP [
46] can be conducted by few steps.
Step 1: To present fuzziness, the pair-wise numeric has to be changed into a matrix using Triangular Fuzzy Number (TFN).
To determine the importance of each criteria preference, natural language or numeric values can be used. In this case, each of the two elements first have to be compared using a scale of importance. For these investigations, first, a linguistic scale from one to nine was determined (see
Table 1), and the TFN was defined by the membership function presented by Chen [
47].
Step 2: Next, fuzzy pair-wise comparison matrices have to be designed.
According to analysis methodology, the decision-makers individually make the pair-wise comparison among all criteria and give the linguistic term represented by a triangular fuzzy number.
Mathematically, this can be represented as
being an
n × n matrix, where
is the rank of criterion
in account to criterion
, according to the fuzzy preference scale shown in the table:
Step 3: This step is about how to compute the normalization of the fuzzy weights.
The
criterion fuzzy weight represented as
can be found by the equation:
where
.
3.3. The Fuzzy TOPSIS Method
As was mentioned above, for this study, the Fuzzy TOPSIS was chosen. Overall, the procedure for the TOPSIS can be started by creating the decision matrix illustrating the determined cost of each criterion according to each alternative. Subsequently, the direct matrix is normalized by a chosen method, and the criteria values are multiplied by the identified criteria weights. Next, the positive-ideal and negative-ideal outcomes are calculated, and the distance of each alternative to these outcomes is calculated with a distance value. Lastly, all alternatives are ordered based on their comparative closeness to the ideal measure.
In this section, the detailed TOPSIS procedure for group decision-making based on Shih, Shyur, and Lee’s suggestion [
49] is presented as a procedure.
Step 1: Create a decision matrix.
To assess the distance learning courses, fourteen criteria and three alternatives (distance learning courses) were chosen that were ranked based on the Fuzzy TOPSIS method. The table below presents the type of criterion and weight assigned to each criterion as fuzzy numbers, linguistic terms and triangular scale (see
Table 2).
Step 2: Create the normalized decision matrix.
Based on the positive and negative ideal solutions, a normalized decision matrix can be calculated by the following relation:
The normalized decision matrixes were calculated for future step 3 analysis.
Step 3: Design the weighted normalized decision matrix.
Considering the dissimilar weights of separate criteria, the weighted normalized decision matrix can be calculated by multiplying the weight of individual criteria in the normalized fuzzy decision matrix, according to the following formula:
where
represents the weight of criterion
The weighted normalized decision matrixes are presented in the results section below (see in
Section 5).
Step 4: Determine the fuzzy positive ideal solution (FPIS, A*) and the fuzzy nega-tive ideal solution ().
As mathematical formulas, the FPIS and FNIS of the alternatives can be defined by equations:
where
represents the max measure of
of all assessed alternatives and
is the min measure of
for all the alternatives. B and C explain the positive and negative ideal explanations, respectively. The positive and negative ideal solutions are presented below in the results
Section 5.
Step 5: Calculate the distance between each alternative and the fuzzy positive ideal outcome and the distance between each alternative and the fuzzy negative ideal outcome .
The gap between each alternative and FPIS and the gap between each alternative and FNIS are calculated, respectively, as follows:
where
d is the gap between two fuzzy numbers, when given two triangular fuzzy numbers
and
and a measure between the two criteria can be calculated by the equation:
It can be noted that and are crisp numbers.
The calculated distances from positive and negative ideal solutions are presented below in results
Section 5.
Step 6: Identify the closeness coefficient and rank the alternatives.
The closeness coefficient of each alternative can be identified by the equation:
The higher evaluated alternative is always closest to the FPIS, and the lower-ranked to the FNIS. The closeness coefficient of individual alternatives and the ranking are presented below in the results
Section 5.
5. Study Results
In this study, fuzzy AHP was used because the fuzzy analysis takes into account the uncertainty and vagueness of the experts’ conclusion. The FAHP methodology, like AHP, was based on the pairwise comparison method, applied to recognize the relative importance of criteria (see
Table A1,
Appendix A). Before calculating the final ranking of distance learning courses, the weights for each criterion were first defined. Accordingly, the linguistic terms used by the experts were converted into Triangular Fuzzy Numbers (TFNs), and then the combined decision matrix was calculated. This research used criteria fuzzy weights that connected the experts answers with a membership function characteristic presented in
Table 1 and additionally in
Table A2,
Appendix A. The calculated FAHP weights for the main criteria are presented in the
Table 3.
The calculated fuzzy AHP criteria weights then were used for distance learning courses ranking using fuzzy TOPSIS. In this study, the criterion A1 “Distance learning course structure” for the content of the distance learning course part assessment, B3 “Possibility of personalization of the learning process“ for the use of information tools part evaluation, and C3 “Self-learning organization for the participants of the distance module” for the military students’ opinion of the quality of the courses part were chosen as non-beneficial criteria, while the other criteria in this study were used as beneficial criteria. The steps of numerical FTOPSIS have been executed below for three expert groups.
Step 1: The distance learning courses in terms of fourteen criteria were evaluated. The multiple experts (34 experts) participated in the evaluation; accordingly the matrix below represents the arithmetic mean of all expert groups (see in
Table 4,
Table 5 and
Table 6).
Step 2: The fuzzy decision matrixes were normalized using Equations (3) and (4).
Step 3: Equation (5) was used to compute the weighted normalized matrixes for three experts’ groups are presented in three tables (see
Table 7,
Table 8 and
Table 9).
Step 4: The fuzzy positive ideal solution (FPIS,
A*) and the fuzzy negative ideal solution (
) were determined. The FPIS represents the maximum value of
for all the distance learning courses, and
is the min value (FNIS). The Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) are calculated using Equations (6) and (7) and are presented by three tables according to the investigation stages by expert groups’ assessment (see
Table 10,
Table 11 and
Table 12).
The table below shows the distance from positive and negative ideal solutions for distance learning courses (see
Table 13).
Notably, the best alternative is nearby to the FPIS and furthest to the FNIS. The distance learning modules (DLM1, DLM2, and DLM3) were evaluated by closeness coefficient, which helped to establish the ranking order for modules by three assessment stages (A—course structure; B—information tools usage; C—military students’ opinion) (see
Table 13). Additionally, the study result is presented by a diagram where the closeness coefficient for each distance learning module is used to clarify how experts’ groups rated each of modules by fourteen criteria and three evaluation stages: first, course structure (
A); second, information tools usage (
B); third, military students’ opinion (
C) (see
Figure 1). Notable, the larger value for closeness coefficient (
) indicates the most preferred alternatives; accordingly, the distance learning modules ranking can be represented by (
). Therefore, from
Figure 2, it can be identified that the largest value of course structure (
A) in this study appeared for DLM1 and the lowest value for DLM3. Furthermore, when experts ranked the information tools usage (
B), a different situation than they evaluated the course structure (
A) was identified, because the DLM3 was evaluated as best and the lowest value was identified for DLM2. Moreover, when military students’ opinions were used for three distance learning modules evaluating, the best was identified as the DLM3 and the lowest value was given for DLM1. The presented distance learning modules’ assessment was identified according to the completed data analysis.
Additionally, the investigation analysis was conducted to assess the impact of criteria weights for different distance learning modules. The students’ opinions on the quality of the three courses are presented in
Figure 3.
After evaluating the quality of three distance learning modules DLM1, DLM2, and DLM3 by four criteria C1, C2, C3, and C4, according to expressed military students’ opinion, it was identified that the distance learning module quality ranking highest was due to to the professionalism of the lecturer (C1) and lecturer assistance to students in the learning process (C2).
After analyzing three distance learning modules evaluation by experts’ ranking according to course structure (
A), it was realized that the query of importance by five criteria appears as
A2 >
A5 >
A3 >
A4 >
A1 for each distance learning module (see
Figure A1,
Appendix A). Therefore, the distance learning module quality (evaluation stage A) was highest due to the content of lecturer’s study material offered, which has to be in high compliance with the purposes of the study program (
A2) and novelty and clarity of the material presented (
A5).
Additionally, the study results of the second evaluation stage when the information tools usage (
B) was judged must be discussed. From the calculation, it can be concluded that the ranking results of the criteria of the B stage can be ordered as
B1 >
B2 >
B4 >
B5 >
B3 (see
Figure A2,
Appendix A). Moreover, the most important criteria were identified as knowledge testing assessment calculation tools (
B1) and material scanning and availability (
B2). This result led us to conclude that the three distance learning modules’ quality was correspondingly ranked in comparison with the evaluation processes of stage A (five criteria), B (five criteria), and stage C (four criteria).
This distance learning quality sensitivity analysis shows that the three stages of different expert groups’ weightings for different criteria fit the differences in the module’s grades. The points found through comprehensive analysis evidently show the sensitive background of the ideal ranking constructed on distance learning modules design. A change in the study quality criteria and quality of experts’ groups leads to changes in the distance learning course’s possible rankings.
6. Discussion
Taking into account the COVID-19 pandemic situation, the Lithuanian government has adopted an order that higher education institutions move to a virtual learning environment. Due to the pandemic situation, the higher study institutions have been forced to reorient and reorganize the entire education system process to distance learning courses in a short period of time. Therefore, like most higher education institutions, MALs are intensively developing and improving distance learning modules in order to improve the quality of studies. However, this process is not straightforward, as all universities face complex challenges that need to be addressed urgently in order to develop more student-focused studies.
Newly emerging information and communication tools make it possible to improve traditional studies, make them more acceptable, but change the main principle of study organization. Notable, the development of information technology affects all areas of human activity, including science and studies. Most of the advantages of information and communication technologies are likely associated with distance learning, which are rapidly gaining popularity due to flexibility and the ability to study at a convenient time and in a convenient location. However, the diversity of information and communication technologies and their application to distance learning does not in itself determine the efficiency of the study process. Therefore, special attention is paid to the quality of distance learning. As a result, there are increasing requirements for teachers’ knowledge of IT tools. The study results of the judged information tools’ usage (B) showed that three modules were ranked by quality in the order DLM3> DLM1> DLM2 by criteria that were ranked as B1 > B2 > B4 > B5 > B3. The results showed that all distance learning modules have must improve the quality of the possibility of the personalization of the learning process (B3).
This result is linked to other research [
18] that emphasized that the learning environment should be tailored to individual needs. Applying a specific environment to teaching provides an opportunity to recommend relevant material to a particular learner according to his or her specific needs, such as learning style, modality, cognitive style, and competencies. The survey helped to identify the need for further improvements in synchronous and asynchronous communication capabilities (
B5), which highlights the relevance of the need for improvement as it can be an opportunity to improve the quality of distance learning and teaching.
The same investigation result was proven by other researchers [
54,
55] who showed that, to improve learning conditions and quality, appropriate technological tools must be selected for the learning content, as well as IT procedures such as software agents that support students’ self-directed learning and facilitate the work of teachers. This means that each distance learning module lecturer requires the ability to select the most appropriate resources for organizing distance learning, to evaluate the possibilities of their use, to know alternative resources, and to have a clear plan for distance learning. Moreover, lecturers have to be ready to solve a number of additional difficulties related to the quality of distance learning.
Consequently, the third stage of the conducted survey analysis, which was done in order to achieve the quality of learning content and the needs of students [
56,
57], showed that the criteria by their importance according to students’ opinion were ranked as
C1 >
C2 >
C4 >
C3 (see
Figure 2). Notably, an experienced lecturer (
C1) presents learning material in an interesting way, strives to provide assistance to students in the process of organizing teaching, uses the appropriate IT tools (
C2), and raises students’ motivation by choosing relevant course material to maximize the practical benefits of the course, because this develops students’ competencies and skills (
C4). Strong student willingness was also expressed about the importance of the impact of feedback on the distance study results, but according to the results of this study, the students were extremely lacking (
C3) (see
Figure 2). The study results according to the opinion expressed by military students are similar to what was clarified in other surveys [
56,
58].
Moreover, after the evaluation of three distance learning courses by the FTOPSIS method by virtual learning environment and the use of IT tools, the DLM3 was rated as the best and the DLM2 as the worst. However, based on the results of the third assessment stage, when the quality of the three distance learning modules was compared by the students’ opinion, the DLM3 was rated as the best and the DLM1 as the worst. We believe that the dissimilarity of modules ranking is due to the qualification and experience of experts who participated in this survey. As was mentioned above in
Section 4.2.1, this survey evaluating distance learning courses included experts who were highly interested in the quality of distance studies, but in fact were from various fields of study-related activities. These were the lecturers, cadets, IT professionals, and decision-makers from the administration department of the MAL. This multilayered approach with the replication of various interests of the participants of the evaluated distance learning courses helped to investigate different viewpoints reflecting the diverse judges’ interests. The obtained distance modules evaluations allowed us to identify the directions of the improvement of the distance courses, taking into account the evaluators’ assessments and comments. Correspondingly, it can be pointed out that this survey contributes to the development of an easily understandable hierarchy of criteria model that reflects the main goal of the study quality assessment. This research used FAHP criteria selection, grouping methods, and evaluation principles that facilitated the quality assessment of distance learning modules [
59,
60,
61,
62]. This survey had few limitations. First, the thirty-four judges were chosen with specific knowledge and skills in the specific field and with very different competencies. Second, for the distance learning module quality measurement, a different number of criteria were chosen. Third, when the composition of expert groups changed, for example, with the decreasing or increasing the number of experts, the assessment results could be different. Fourth, the source of uncertainty is the subjectivity of expert opinions. Taking into account all these limitations, the presented conclusions must be interpreted carefully. For better clarification of the quality assessment process, the additional analysis of experts’ qualification has to be done, and the additional criteria could be included.
The survey conducted was subject to a comprehensive assessment that took into account the uncertainty of the expert data; in future investigations, a comparable comprehensive course quality assessment methodology could be used to assess the quality of other similar services or courses provided. This evaluation methodology could be successfully applied for the assessment of distance learning courses that are provided by any company or institution, because the evaluation of the quality of distance learning became important for the secondary schools due to the COVID-19 pandemic situation, such that the distance education process has to be organized. Such studies could provide further information about student complaints.
7. Conclusions
This used Fuzzy-TOPSIS-based analysis to identify the value for distance learning courses’ quality. The basis of the analysis used was fuzzy logic and TOPSIS multi-criteria technique. The used analysis’ mixture was demonstrated to be a good alternative with excellent interaction, which can make a comprehensive assessment of three courses by fourteen criteria.
The DLM evaluation was done on three levels: the first stage was the evaluation of distance learning course structure; second, the evaluation of the quality of information tools usage; and third, the military students’ opinion of the quality of distance learning courses regarding the problem of adaptability to distance course user requirements. The survey results showed the fuzzy TOPSIS technique to be a practical method that provides a very satisfactory value analysis and ranking. The conducted survey showed that the backgrounds and ways of assessing the distance learning modules’ quality are appropriate for future usage.
The MCDM techniques FAHP and FTOPSIS were used to select the best distance learning course and to identify the criteria ranks. The study used fourteen criteria for the ranking of the three DLMs. Moreover, the results of the evaluation of the three courses and the application of the AHP Fuzzy method, which allowed us to determine the importance of the criteria weights, helped identify the shortcomings of the distance course modules. However, in summarizing the results of the three-stage assessment in this study, it was not taken into account that the importance of the assessment stages differed due to this approach: the scope of studies differs, as does the time required to prepare and evaluate the DLM.
The result of FTOPSIS and sensitivity analysis of criteria let us identify the best ways to increase distance learning quality; accordingly, the lower-ranked criteria were the distance learning structure (A1), the possibility of the personalization of the learning process (B3), and the strategy of self-learning for participants of the distance module. It can also be concluded that the possible explanation for the best ranking of DLM3 belongs to the enclosed survey results, where the majority of the criteria assessed by three expert groups in this study module were evaluated to be high.
The modular approach resolves the problem of the importance of distance course assessment in determining the specific weights of criteria for which the military academy administration department is responsible. Moreover, it was identified that the individualization of the teaching process can be an important lecturer decision to improve the quality of education, which requires that many factors be taken into account, including learner profiles, learning materials, and learning strategies.
For future investigations, a group of experts can be included for the assessment by other MCDM techniques such as Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE). In addition, the PROMETHEE method can be applied with fuzzy logic, interval number, or hesitant fuzzy sets.