Exploring Influential Factors of Industry–University Collaboration Courses in Logistics Management: An Interval-Valued Pythagorean Fuzzy WASPAS Approach

Abstract

The development of E-commerce and digitalization drives the rapid change in logistics management practices and poses challenges to traditional talent training modes in logistics field. Nowadays, companies expect university graduates equipped with more practical logistics skills to connect tighter with the industry. This motivates universities to establish more practically relevant curriculums to enhance students’ career competitiveness. Under such background, industry–university collaboration courses are increasingly adopted in higher education institutes in logistics discipline. Due to the difference between this type of course and the traditionally taught courses, the learning outcome of it can be difficult to guarantee. Therefore, it is necessary to identify the influential factors of the learning outcomes of industry–university collaboration courses and establish the actionable strategies to enhance course quality. However, the current literature in logistics management education has little focus on this topic, resulting in gaps on clarifying the influential factors of learning outcomes of industry–university collaboration courses in this discipline. Applying a mixed method, this study conducted a case study for an industry–university collaboration course of a logistics discipline in a Chinese university. The interval-valued Pythagorean fuzzy (IVPF) numbers and the Weighted Aggregated Sum Product Assessment (WASPAS) methods were used. The results showed that there are 15 factors which can influence the outcomes of industry–university collaboration courses in logistics discipline. Among them, the most important factor is the working environment, followed by the students’ own ability. Also, the results indicated that students’ optimistic attitudes towards the course, whether students take the course seriously, and course evaluations can be influential factors for good learning outcomes. The sensitivity analysis was then conducted, showing that the results were robust. This study can contribute to the existing literature by providing a theoretical framework to understand and assess the quality of industry–university collaboration courses in logistics and relevant subjects, as well as offering new analytical tools for management educational studies. Moreover, this study can provide practical implications for educators to develop and maintain good industry–university collaboration courses and trainings. Specifically, a practical life-cycle view was suggested to put pertinent efforts in all periods before/during/after the course to achieve high course outcomes.

1. Introduction

Logistics has become a pivotal support for economic growth [1]. Nowadays, the development of advanced technologies largely changed the processes of logistics, making logistics systems more complicated in practice than ever [2]. This poses the challenges of effectively operating logistics under the current digitalization era [3], which makes it more difficult for practitioners to manage logistics activities [4]. To meet the market requirements, students majoring in logistics management need not only to grasp the disciplinary knowledge, but also to properly apply them into the practices of logistics operations [5]. This drives the changes in higher education and promotes the industry–university collaboration education.

In China, the logistics talent training and education have transformed to be more practice-oriented. For example, in 2023, the national smart logistics industry–education integration community was established to link logistics education more tightly to the industrial practices (see: https://zhijiao.eol.cn/detail/2023/10/16/1697446523_9342.html, accessed on 2 July 2025). Also, some universities developed industry–university collaboration courses and projects. For example, Hubei University of Economics collaborated with the Kingdee company to develop a business operations course to teach students digital management systems (see: https://zcb.hbue.edu.cn/dc/ac/c8152a318636/page.htm, accessed on 2 July 2025), which covers manufacturing, operations, and overall management practices. It can be seen that industry–university collaboration education has become a popular mode for higher education in logistics and relevant disciplines.

Although this education mode attracts the attention of logistics educators, the learning outcomes of such mode may not be easily guaranteed. Compared to the traditionally taught courses in higher education, the industry–university collaboration courses can differ in many aspects, such as curriculums, delivery approaches, instructors, and evaluation approaches [6,7]. These differences could not only amplify the difficulty in course design, but also make it difficult for students to adapt, thereby reducing the learning outcomes. Therefore, it is important to investigate the influential factors that enhance the learning outcomes of industry–university collaboration courses.

However, although the current literature covers different aspects of industry–university collaboration (e.g., [8,9]), there are few publications in logistics management discipline investigating the influential factors of learning outcomes of industry–university collaboration courses, with even fewer studies providing the actionable advice for learning outcome enhancement from a logistics educator perspective. As the discipline of logistics management has its own uniqueness, and as its knowledge and practices change rapidly due to the market environments, lacking in deep analysis on how good learning outcomes can be achieved in industry–university collaboration courses will slow down the quality improvement of logistics higher education.

Therefore, to fill the gap of the literature and to pave the way to enhance course quality, this study aims to solve the following research questions: What are the factors that can influence the learning outcomes of the industry–university collaboration courses and what are their relative importance? To answer the questions, a case study based on a logistics industry–university collaboration course in a Chinese university was conducted, and mixed methods including qualitative exploration and quantitative modeling using IVPF numbers as well as WASPAS were applied.

This study can have the following contributions:

• From the academic perspective, this study identified the 15 factors that can influence the learning outcomes of logistics industry–university collaboration courses, providing a theoretical framework for future research in education quality assessment. Also, this study linked IVPN-WASPAS to rank the relative importance of different factors, confirming its methodological values in logistics educational studies and providing new analytical tools for relevant fields. Compared with the previous literature related to industry–university collaboration courses in logistics management, this study can contribute to the theoretical basis as well as methodological support for studies in this topic.
• From the practical perspective, this study can inform the course designers, lecturers and tutors, as well as students who take this type of course about how the courses can be better prepared and implemented. The “life-cycle” approach of the course delivery that this study advocates can give educators information about how good outcomes of industry–university collaboration courses in logistics discipline can be achieved. This can therefore contribute to the fruitful practice-related teaching and learning in higher education and eventually enable logistics management graduates greater career competitiveness.

This study has six sections. After the introduction, the second section thoroughly reviews the literature relevant to this study. The third section introduces the methods used to explore the research questions, followed by the fourth section presenting the results. The fifth section provides the discussion of the results, and the study is concluded by implications, limitations, and future directions.

2. Literature Review

This study concerns three streams of literature, namely, education in logistics management discipline, industry–university collaboration education, and MCDM in higher education field.

2.1. Education in Logistics Management Discipline

The importance of logistics management in economic growth and sustainable development has been widely recognized, leading to higher needs to train logistics experts. In the existing literature, the logistics management education is investigated from multiple perspectives. For example, Huang et al. [10] explored the factors that can influence the student performance of contest-based teaching in logistics management. Using interpretive structural modeling, the authors identified the hierarchical structures of factors and developed a contest-based teaching framework. Malka and Austin [11] used a logistics program as a case to study how learning modes can be linked to the students’ success after graduation in the supply chain field. Using the necessary condition analysis, the authors investigated how different learning modes can impact students’ success and how such an impact can be influenced by different variables. Zheng et al. [12] applied topic modeling and sentiment analysis to the online reviews of logistics MOOCs (i.e., Massive Open Online Coursework). Using the push–pull–mooring theory, the authors identified the current gaps in courses and opportunities for improvement. Similarly, Huang et al. [13] explored the negative online reviews in supply chain management MOOCs. By using a mixed method with TOPSIS (i.e., Techniques for Order Preference by Similarity to Ideal Solution) and CoCoSo (i.e., Combined Compromise Solution), the authors found the key factors reducing the outcomes of MOOC learning, and provided the suggestions for improvement. Emre et al. [14] investigated the conceptual awareness of digital logistics among students. By conducting a survey method among Turkish universities, the authors identified the awareness levels of university students and offered the suggestions for course design and improvements for higher education. Schinckus and Nguyen [15] studied a new logistics teaching approach called the real-life-based teaching. Using a case study in Vietnam, the authors presented how this method can be applied to the logistics education and confirmed the students’ benefit from this method compared with the traditional teaching approaches.

2.2. Industry–University Collaboration Education

Industry–university collaboration education has become increasingly popular in higher education. To increase the outcomes of this education approach, multiple investigations have been made. For example, Yu et al. [16] focused on the evaluation of the industry–university collaboration course quality. They developed a modified fuzzy Analytic Hierarchy Process (i.e., AHP) method and created an evaluation index system. By applying the newly proposed method to an industry–university collaboration course, the authors examined the course quality level. Yan [17] focused on the higher vocational educational context and used the decision-tree algorithms to assess the industry–school collaboration education, and the improvement opportunities for the collaboration education were identified. Hu and Panyadee [18] proposed a mixed methodological framework including TODIM (a term in Portuguese for Interactive and Multicriteria Decision Making), PROMETHEE (i.e., Preference Ranking Organization Method for Enrichment Evaluation), and probabilistic linguistics methods to evaluate the industry–university collaboration education. By applying this framework to different cooperation schemes from the collaboration education perspective, the authors evaluated their development levels. Bian and Wang [19] proposed a school–enterprise collaboration and cooperation mechanism based on a modified decision tree method. The authors showed that, by applying it, the information sharing and communication between companies and universities can be improved, and the training and education effectiveness can be enhanced by using the school–enterprise training platforms. Li [20] focused on the digital platform of the industry–university collaboration courses. Using big data and internet-of-thing technology, the author designed a course platform which significantly increased the effectiveness of the industry–university collaboration courses.

2.3. MCDM in Higher Education

To properly assess the higher education quality, MCDM methods have been widely used in recent years. Ersoy [21] applied data envelopment analysis (i.e., DEA) and TOPSIS methods to measure the performance of distance education in Turkish public universities. The author ranked the performance of universities by the efficiency of distance education and found that seven universities were efficient. Liu et al. [22] proposed a mixed method combining MCDM, neural network, and Cobweb model to assess the development level of higher education. Using an 11-indicator measurement system, the authors conducted a multi-national evaluation for the higher education system. Gul and Yucesan [23] proposed a method combining Bayesian-BWM (i.e., best-worst method) and TOPSIS to evaluate the higher education in Turkish universities. By using the method, the authors ranked the universities based on the values calculated from the MCDM approach. Geng and Zhao [24] adopted a sustainable perspective and evaluated the development levels of sustainable higher education in different provinces in China. Based on the entropy-TOPSIS and coupling coordination degree methods, they found that the majority of the examined regions’ sustainable higher education development levels are relatively low, although the coastal regions have higher levels. Chen et al. [25] conducted a case study of higher education evaluation by using MCMD methods. They proposed a picture fuzzy MACONT (i.e., Mixed Aggregation by Comprehensive Normalization Technique) method and applied it to the teaching quality evaluation in higher education contexts. Do et al. [26] proposed a method integrating fuzzy AHP and TOPSIS to assess the lecturer performance in Vietnam. The performance indicators for the evaluation were identified and used in the assessment, providing information and improving the efficiency of human resource management in higher education institutes.

2.4. Research Gaps

Based on the above literature review, the comparisons of different literature streams can be developed, with this paper positioned in

Table 1. Literature review summary and comparison.

	Education in Logistics Management Discipline	Industry–University Collaboration Education	MCDM in Higher Education
Huang et al. [10]	✓		✓
Malka and Austin [11]	✓
Zheng et al. [12]	✓
Huang et al. [13]	✓		✓
Emre et al. [14]	✓
Schinckus and Nguyen [15]	✓
Yu et al. [16]	✓		✓
Yan [17]	✓		✓
Hu and Panyadee [18]	✓		✓
Bian and Wang [19]	✓
Li [20]	✓
Ersoy [21]			✓
Liu et al. [22]			✓
Gul and Yucesan [23]			✓
Geng and Zhao [24]			✓
Chen et al. [25]			✓
Do et al. [26]			✓
This study	✓	✓	✓

.

From the above table, the following research gaps could be identified. On the one hand, it can be seen that although multiple studies have been conducted in logistics education and industry–university collaboration education, there are few studies covering both directions and investigating the influential factors of the outcomes of industry–university collaboration courses in logistics discipline. This can pose challenges to the academician and practitioners who tend to design and implement such courses in their logistics educational system. On the other hand, the methodological tools are relatively lacking in exploring the influential factors of industry–university collaboration course outcomes in logistics discipline. This leads to the gap that the relative importance of influential factors is not fully clarified, hindering the way for course designers and instructors to prioritize and improve the course implementation processes. As the IVPF-WASPAS method is flexible in capturing rich information and robust in examining the relative importance of factors [27,28], the method is promising to address the current gaps in literature in the factor exploration and enhance the effectiveness of logistics management educational studies. Therefore, to fill these gaps, this study attempts to adopt the IVPF-WASPAS method to thoroughly explore the influential factors of industry–university collaboration courses in logistics management and rank factors by their relative importance.

3. Materials and Methods

This study aims to explore the influential factors of the learning outcomes of industry–university collaboration courses in logistics management. To achieve this, this study conducted a case study using mixed research methods. A practice-oriented logistics management course is selected as the case course of this study. In the rest of this section, the case background is presented first, followed by an explanation of the specific methods adopted.

3.1. Case Background

The case course deployed in this study is a course collaborated between a university and logistics companies, including a port company and an express company. Students taking this course are in their second year of university study, and they need to participate in different daily operations of companies as interns for several weeks. Companies will appoint experienced employees as students’ tutors to teach them basic knowledge about how logistics companies operate. After the completion of the internship, students are asked to identify the operational process and possible problems of the logistics companies and write course reports to solve them by applying their discipline knowledge. The academic performance will be evaluated by simultaneously considering their performance in the company as well as in the reports. Through this, students can understand the basic operations and policies of logistics companies, enhancing their ability of applying discipline knowledge in practice.

The reason why this course is selected as the case is justified by its representativeness. In contrast to other courses which are taken in the classroom, this course gives the students a chance to work in companies, and the students can not only receive the teaching from university lecturers, but also obtain guidance from the industrial tutors. Also, as the course requires students to stay in the company for several weeks rather than one or two days, the students can have enough time to gain thorough experience of logistics practices. This makes them have a deep understanding about how learning outcomes of industry–university collaboration courses in logistics management discipline can be improved, which can give our study rich information to investigate.

3.2. Methodology

To thoroughly identify the relevant factors and investigate their relative importance, the methodological flows in Figure 1 were deployed in our case study. Figure 1 indicated that this study consists of two main components, namely, qualitative exploration and quantitative modeling. For the qualitative exploration, the study focused on identifying relevant influential factors from students’ feedback. To achieve this, students were invited to provide course feedback after course completion, and three authors conducted qualitative coding to analyze the content and generate the relevant factors. After that, the quantitative modeling was applied to compare the relative importance of the coded factors. The IVPF-WASPAS method was adopted due to its advantage in capturing rich information from decision makers and in result robustness [28]. The questionnaire was circulated to students for their opinions of the relative importance of different factors, which was then analyzed by IVPF-WASPAS. The factor rank was derived and the result robustness was examined by sensitivity analysis. The following two sections provide the details of the methodology.

3.2.1. Qualitative Exploration

The study started with the qualitative exploration, and the first step was to collect the students’ feedback. After the completion of the industry–university collaboration course, we asked the students to share their course feedback. To more effectively collect the influential factors of the course outcomes, we asked the students to provide the feedback for the following open question: What kind of factor(s) do you think will influence the outcomes of your industry–university collaboration course before/during/after the course? The question considered the temporal changes in the courses to help students better organize their ideas and clarify different factors. The reason why the three stages (i.e., before/during/after the course) were differentiated in qualitative feedback is largely inspired by Gagné’s theory on the conditions of learning [29]. The well-known Nine Events of Instruction model provided by Gagné [30] can enable good cognitive processing and essentially can be justifiable to suggest the importance of factors in pre-class (e.g., gaining attention, informing the learner of the objective, or stimulating recall of prerequisite learning), during-class (e.g., presenting the stimulus material, providing learning guidance, or eliciting the performance), and post-class (e.g., providing feedback, assessing the performance, enhancing retention and transfer). Therefore, underpinned by the well-developed theory, we asked students to provide feedback on learning outcome factors by considering different stages of the courses to help them build a more organized logic to express their experience.

After collecting the feedback, three authors conducted qualitative analysis to code the relevant influential factors of the course outcomes. A rigorous content analysis process was adopted as follows. First, three authors coded the feedback separately, generating potential factor codes. Second, three authors met to unify their codes and identify the ones that they cannot achieve an agreement. Finally, the authors thoroughly discussed the identified codes that had not reached an agreement, and eventually achieved an agreement by modifying the codes. The rigorous coding process ensures the validity of the factor generation steps.

3.2.2. Quantitative Modeling

After the completion of the qualitative exploration, we developed quantitative models to quantify the relative importance of the factors. Specifically, IVPF-WASPAS was adopted for this study. Compared to other traditional MCDM methods like AHP and TOPSIS, IVPF-WASPAS can bring the following superior advantages. First, the IVPF number can capture richer information from the participants’ linguistic articulation, where the degree of membership, non-membership, and indeterminacy for describing a specific object can be all expressed using IVPF numbers [27,31]. This therefore enables a more inclusive quantification of the participants’ response. Also, the analysis can benefit from the robustness of WASPAS compared to other methods, as the result is a weighted combination of the weighted sum and the weighted product which are two types of alternative evaluation [32]. The following explains how this method is used.

To better introduce the method, the key definitions of the IVPF set are presented.

Definition 1.

Suppose $\tilde{A}$ is an interval-valued fuzzy Pythagorean number (IVPFN) on a fixed set $X$. It can be quantified as follows [33]:

$$\tilde{A}=\{<x,u_{\tilde{A}}\left(x\right),v_{\tilde{A}}\left(x\right);x\in X>\}$$

(1)

where $u_{\tilde{A}}\left(x\right)=\left[u_{\tilde{A}}^L\left(x\right),u_{\tilde{A}}^U\left(x\right)\right]\subset [0, 1]$; $v_{\tilde{A}}\left(x\right)=\left[v_{\tilde{A}}^L\left(x\right),v_{\tilde{A}}^U\left(x\right)\right]\subset [0, 1]$. Based on the property of IVPFN, the following relationship holds:

$$0\le u_{\tilde{A}}^U{\left(x\right)}^2+v_{\tilde{A}}^U{\left(x\right)}^2\le 1$$

(2)

The degree of indeterminacy, ${\pi }_{\tilde{A}}=\left[{\pi }_{\tilde{A}}^L\left(x\right),{\pi }_{\tilde{A}}^U\left(x\right)\right],$ can be determined directly:

$${\pi }_{\tilde{A}}^L\left(x\right)=\sqrt{1-u_{\tilde{A}}^U{\left(x\right)}^2-v_{\tilde{A}}^U{\left(x\right)}^2}$$

(3)

$${\pi }_{\tilde{A}}^U\left(x\right)=\sqrt{1-u_{\tilde{A}}^L{\left(x\right)}^2-v_{\tilde{A}}^L{\left(x\right)}^2}$$

(4)

For simplicity, we denote IVPFN as $\tilde{A}=\left(\left[u_{\tilde{A}}^L,u_{\tilde{A}}^U\right],\left[v_{\tilde{A}}^L,v_{\tilde{A}}^U\right]\right)$.

Definition 2.

For an IVPFN, its product and power operations with a real number are as follows [31]:

$$\lambda \tilde{A}=\left(\left[\sqrt{1-{\left({1-\left(u_{\tilde{A}}^L\right)}^2\right)}^{\lambda }},\sqrt{1-{\left({1-\left(u_{\tilde{A}}^U\right)}^2\right)}^{\lambda }}\right],\left[{\left(v_{\tilde{A}}^L\right)}^{\lambda },{\left(v_{\tilde{A}}^U\right)}^{\lambda }\right]\right)$$

(5)

$${\tilde{A}}^{\lambda }=\left(\left[{\left(u_{\tilde{A}}^L\right)}^{\lambda },{\left(u_{\tilde{A}}^U\right)}^{\lambda }\right],\left[\sqrt{1-{\left({1-\left(v_{\tilde{A}}^L\right)}^2\right)}^{\lambda }},\sqrt{1-{\left({1-\left(v_{\tilde{A}}^U\right)}^2\right)}^{\lambda }}\right],\right)$$

(6)

Definition 3.

For two IVPFNs, ${\tilde{A}}_1=\left(\left[u_{{\tilde{A}}_1}^L,u_{{\tilde{A}}_1}^U\right],\left[v_{{\tilde{A}}_1}^L,v_{{\tilde{A}}_1}^U\right]\right)$ and ${\tilde{A}}_2=\left(\left[u_{{\tilde{A}}_2}^L,u_{{\tilde{A}}_2}^U\right],\left[v_{{\tilde{A}}_2}^L,v_{{\tilde{A}}_2}^U\right]\right)$, the summation and the product of them are as follows [28]:

$${\tilde{A}}_1⨁{\tilde{A}}_2=\left(\left[\sqrt{{\left(u_{{\tilde{A}}_1}^L\right)}^2+{\left(u_{{\tilde{A}}_2}^L\right)}^2-{\left(u_{{\tilde{A}}_1}^L{u}_{{\tilde{A}}_2}^L\right)}^2},\sqrt{{\left(u_{{\tilde{A}}_1}^U\right)}^2+{\left(u_{{\tilde{A}}_2}^U\right)}^2-{\left(u_{{\tilde{A}}_1}^U{u}_{{\tilde{A}}_2}^U\right)}^2}\right],\left[v_{{\tilde{A}}_1}^L{v}_{{\tilde{A}}_2}^L,v_{{\tilde{A}}_1}^U{v}_{{\tilde{A}}_2}^U\right]\right)$$

(7)

$${\tilde{A}}_1⨂{\tilde{A}}_2=\left(\left[u_{{\tilde{A}}_1}^L{u}_{{\tilde{A}}_2}^L,u_{{\tilde{A}}_1}^U{u}_{{\tilde{A}}_2}^U\right],\left[\sqrt{{\left(v_{{\tilde{A}}_1}^L\right)}^2+{\left(v_{{\tilde{A}}_2}^L\right)}^2-{\left(v_{{\tilde{A}}_1}^L{v}_{{\tilde{A}}_2}^L\right)}^2},\sqrt{{\left(v_{{\tilde{A}}_1}^U\right)}^2+{\left(v_{{\tilde{A}}_2}^U\right)}^2-{\left(v_{{\tilde{A}}_1}^U{v}_{{\tilde{A}}_2}^U\right)}^2}\right]\right)$$

(8)

Definition 4.

$S\left(\tilde{A}\right)$ is the defuzzied value for $\tilde{A}$, and it can be calculated as follows [27]:

$$S\left(\tilde{A}\right)={\displaystyle \frac{u_{\tilde{A}}^L+u_{\tilde{A}}^U+\sqrt{1-{\left(v_{\tilde{A}}^L\right)}^2}+\sqrt{1-{\left(v_{\tilde{A}}^U\right)}^2}+u_{\tilde{A}}^L{u}_{\tilde{A}}^U-\sqrt{\sqrt{1-{\left(v_{\tilde{A}}^L\right)}^2}\sqrt{1-{\left(v_{\tilde{A}}^U\right)}^2}}}{4}}$$

(9)

After defining IVPFN, the questionnaire for evaluating the importance of each factor using IVPFN was developed and circulated to the students. Specifically, the questionnaire listed the coded factors generated from the qualitative exploration steps. For each specific factor, students were asked the following question: How significantly do you think this factor will influence the outcome of the course you attended? For the answer of this question, based on the previous literature [31], a seven-scale choice was adopted, ranging from certainly low significance to certainly high significance.

After collecting the questionnaire from the students, the judgments obtained from each student in the linguistic terms were then transformed into IVPFNs based on [31], with the transformation relationship listed in

Table 2. Linguistic terms and their transformation into IVPFNs.

Linguistic Terms	IVPF Numbers
Certainly low significance (CL)	([0.1, 0.3], [0.7, 0.9])
Very low significance (VL)	([0.2, 0.4], [0.6, 0.8])
Low significance (L)	([0.3, 0.5], [0.5, 0.7])
Medium level significance (M)	([0.4, 0.6], [0.4, 0.6])
High significance (H)	([0.5, 0.7], [0.3, 0.5])
Very high significance (VH)	([0.6, 0.8], [0.2, 0.4])
Certainly high significance (CH)	([0.7, 0.9], [0.1, 0.3])

.

Based on the above transformation, the decision matrix, ${\mathit{D}}_{\mathit{m}\times \mathit{n}}$, can be formed as follows:

$${\mathit{D}}_{\mathit{m}\times \mathit{n}}=\left[\begin{array}{ccc}\begin{array}{cc}\begin{array}{c}{\tilde{D}}_{11}\\ {\tilde{D}}_{21}\end{array}& \begin{array}{c}{\tilde{D}}_{12}\\ {\tilde{D}}_{22}\end{array}\end{array}& \dots & \begin{array}{c}{\tilde{D}}_{1n}\\ {\tilde{D}}_{2n}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{\tilde{D}}_{m1}& {\tilde{D}}_{m2}\end{array}& \dots & {\tilde{D}}_{mn}\end{array}\right]$$

(10)

For the decision matrix, $m$ is the number of factors, while $n$ is the number of valid responses collected from questionnaires. The ${\tilde{D}}_{ij}$ is the element of ${\mathit{D}}_{\mathit{m}\times \mathit{n}}$ with $1\le i\le m$ and $1\le j\le n$. According to the previous literature, to conduct WASPAS, the decision matrix needs to be normalized. The normalized matrix, ${\mathit{S}}_{\mathit{m}\times \mathit{n}}$, is formed as follows:

$${\mathit{R}}_{\mathit{m}\times \mathit{n}}=\left[\begin{array}{ccc}\begin{array}{cc}\begin{array}{c}{\tilde{R}}_{11}\\ {\tilde{R}}_{21}\end{array}& \begin{array}{c}{\tilde{R}}_{12}\\ {\tilde{R}}_{22}\end{array}\end{array}& \dots & \begin{array}{c}{\tilde{R}}_{1n}\\ {\tilde{R}}_{2n}\end{array}\\ ⋮& \ddots & ⋮\\ \begin{array}{cc}{\tilde{R}}_{m1}& {\tilde{R}}_{m2}\end{array}& \dots & {\tilde{R}}_{mn}\end{array}\right]$$

(11)

where ${\tilde{R}}_{ij}={\displaystyle \frac{{\tilde{D}}_{ij}}{\underset{i}{\max }S\left({\tilde{D}}_{ij}\right)}}$, with $1\le i\le m$ and $1\le j\le n$.

Based on the normalized matrix, the arithmetic weighted matrix ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$ and power weighted matrix ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$ are calculated separately. Here, this study assumes that each student filling the questionnaire has the same importance; therefore, an equal weight method is used in the calculation of ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$ and ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$. As there are $n$ valid results from students, the weight is essentially ${\displaystyle \frac{1}{n}}$. Suppose the element of ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$ is ${\tilde{T}}_{ij}$ with $1\le i\le m$ and $1\le j\le n$, and the element of ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$ is ${\tilde{P}}_{ij}$ with $1\le i\le m$ and $1\le j\le n$. Based on Equations (5) and (6), the relationships are as follows:

$${\tilde{T}}_{ij}={\displaystyle \frac{1}{n}}{\tilde{R}}_{ij}=\left(\left[\sqrt{1-{\left(1-{\left(u_{{\tilde{R}}_{ij}}^L\right)}^2\right)}^{{\displaystyle \frac{1}{n}}}},\sqrt{1-{\left(1-{\left(u_{{\tilde{R}}_{ij}}^U\right)}^2\right)}^{{\displaystyle \frac{1}{n}}}}\right],\left[{\left(v_{{\tilde{R}}_{ij}}^L\right)}^{{\displaystyle \frac{1}{n}}},{\left(v_{{\tilde{R}}_{ij}}^U\right)}^{{\displaystyle \frac{1}{n}}}\right]\right)$$

(12)

$${\tilde{P}}_{ij}={\tilde{R}}_{ij}^{{\displaystyle \frac{1}{n}}}=\left(\left[{\left(u_{{\tilde{R}}_{ij}}^L\right)}^{{\displaystyle \frac{1}{n}}},{\left(u_{{\tilde{R}}_{ij}}^U\right)}^{{\displaystyle \frac{1}{n}}}\right],\left[\sqrt{1-{\left(1-{\left(v_{{\tilde{R}}_{ij}}^L\right)}^2\right)}^{{\displaystyle \frac{1}{n}}}},\sqrt{1-{\left(1-{\left(v_{{\tilde{R}}_{ij}}^U\right)}^2\right)}^{{\displaystyle \frac{1}{n}}}}\right]\right)$$

(13)

Based on the above relationship, the WASPAS can be derived by using the following steps [32]. The weighted sum value of each factor is defined as $K_i$ with $1\le i\le m$, while the weighted product value of each factor is defined as $H_i$ with $1\le i\le m$. The following relationships are derived from ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$ and ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$, respectively, as follows:

$$K_i={\sum }_{j=1}^n{\tilde{T}}_{ij}$$

(14)

$$H_i={\prod }_{j=1}^n{\tilde{P}}_{ij}$$

(15)

Based on the weighted sum and product values of each factor, the WASPAS value of each factor is notated as $B_i$ with $1\le i\le m$ and can be calculated as follows:

$$B_i=\alpha K_i+\left(1-\alpha \right)H_i$$

(16)

The factors are then ranked by their importance measured by the defuzzied value of $B_i$ using Equation (9). The $\alpha$ here is defined as the weight of $K_i$ and is the WASPAS parameter, indicating the relative importance of $K_i$. The α is defined in the range between 0 and 1 [32]. In the initial calculation, the $\alpha$ is set equal to 0.5 [32,34]. After that, a sensitivity analysis is performed to examine the result robustness to the different values of $\alpha$.

4. Results

Applying the methods in Section 3, the results of this study were obtained. The results are presented first, followed by the sensitivity analysis of the parameter of IVPF-WASPAS (i.e., $\alpha$).

4.1. Qualitative Results

First, for the qualitative exploration, 21 students were willing to provide their feedback of the influential factors of the course outcomes, with information spanning the periods before/during/after the course. This resulted in the collection of 63 pieces of qualitative and descriptive feedback. Based on the qualitative feedback, three authors rigorously analyzed the content to code the relevant factors. Specifically, a summative table was created in the spreadsheet to record every piece of feedback. Each feedback was given a number, and each author read the feedback piece by piece carefully. After fully familiarizing the content, a three-round analysis was conducted. In the first round, the authors individually coded the factors by extracting the information from the raw feedback, identifying all possible factors. In the second round, three authors discussed together to compare and contrast their results, further refining the factors. Whenever there were disagreements, the third round analysis was conducted to achieve the final agreements. In the first round of coding, three authors individually coded the feedback, leading to 25 factors, 12 factors, and 25 factors, respectively. In the second round, three authors discussed together, modifying the factors and resulting in 15 factors. However, they did not reach an agreement on two factors. In the third round, the authors re-read the feedback and modified these two factors, and finally achieved an agreement. Eventually, 15 factors were identified, with their codes (i.e., C1 to C15) and definition provided in

Table 3. Factors influencing the outcomes of the industry–university collaboration course.

Before the Course	C1	The degree that students know about the company and their jobs allocated by the company before the course.
	C2	The level of preparation and training before the students start the course.
	C3	Whether the students have positive/optimistic attitudes towards the incoming course and set clear objectives for the course learning.
	C4	Whether the specialized knowledge and skills of students can fit the requirements of the jobs allocated by the company.
	C5	The promised level of salary/allowance for the students before the course.
During the Course	C6	Whether the work and rest scheduling during the course is appropriate.
	C7	The degree of students’ own ability, including the ability to work under pressure, time management ability, problem-solving ability, communication ability, and physical fitness.
	C8	Whether the working environments during the course, including physical environments, interpersonal relationships and team atmospheres, are good.
	C9	Whether the guidance given by the tutors and leaders in the company is appropriate during the course.
	C10	Whether the students take their work seriously during the course.
	C11	The level of sense of gap between the student’s expectation and the reality of the course.
After the Course	C12	Whether the students properly review their learning experience after the course.
	C13	Whether the students receive the effective feedback on their course performances after the course.
	C14	Whether the course performance evaluation is fair and transparent after the course.
	C15	Whether the students continuously learn and apply the practice-related knowledge and keep receiving the occupational planning and guidance from the company and the university after the course.

.

The factors summarized from the feedback are largely consistent with the previous literature. For example, for factors before the course, the previous literature indicated that prior knowledge related to the course can influence the course outcome [35]. This directly supports C2 and C4, and suggests the importance of C1 as the knowledge of the company and job can also be prior knowledge, which possibly gives students more information to adapt to the industry–university collaboration course, thereby enhancing course outcome. Also, ref. [36] demonstrated that motivational factors are important to contribute to the good course outcomes, meaning that C3 and C5 can pose impacts on learning results. Specifically, C3 is the students’ self-motivation, while C5 is the monetary motivation provided in the course.

For factors during the course, ref. [37] suggested that the good learning environments can directly influence the learning outcomes of students, confirming the importance of C8. Ref. [38] revealed that the difficulty and complexity of the course can influence the student learning outcome, supporting the importance of C6. Ref. [10] found that the learner’s amount of time and efforts put in the course as well as the students’ own ability such as time management ability and the ability of working under pressure can contribute to the student performance. This essentially supports the importance of C7 and C10. Also, ref. [39] indicated that the effective guidance provided by course instructors with appropriate methods can improve the learner performance, indicating the necessity of C9. Finally, as study [36] suggested that motivations are important for students’ learning outcome, it can be reasonably inferred that high levels of sense of gap between the student’s expectation and the reality of the course can demotivate the students during the course, suggesting the importance of C11.

Finally, for factors after the course, the previous literature suggested that the feedback and support of the courses can contribute to good learning outcomes of a course [40], which is in line with C13 and C15. Also, ref. [41] suggested that reviewing and recapping after the course will help the learners consolidate the contents of courses, suggesting the importance of C12 in influencing the learning outcome. Finally, ref. [42] supported the importance of C14, and a fair, standard, and transparent course evaluation can better help students accurately locate what they need to improve, enabling students to clarify their learning directions after the course, thereby enhancing learning outcomes and achievement.

4.2. Quantitative Results

Based on the factors identified, the questionnaire based on linguistic terms was developed and circulated to the students. A total of 87 students provided their evaluation during the time of survey. Based on the previous literature [43], the responses giving all items the same rating were considered as invalid and thus 22 questionnaires were removed. As a result, there were 65 questionnaires left, leading to a validity rate of around 75%. This means that our survey is highly valid for investigating our research questions. The Cronbach’s alpha for the response was around 0.852, indicating a high reliability of the results. The full results for 65 valid responses are presented in Appendix A 

Table A1. Valid results of students’ questionnaire.

Questionnaire	Factors
	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15
1	H	H	H	H	H	H	VH	H	H	H	H	H	H	H	H
2	L	CL	CH	CH	CH	M	H	CH	M	CH	H	M	H	CH	VH
3	M	M	VH	H	VH	H	VH	VH	M	VH	M	H	VH	CL	H
4	L	H	H	H	H	H	H	H	H	H	M	H	H	H	H
5	H	H	H	VH	VH	H	CH	H	H	VH	H	VH	H	VH	H
6	H	H	H	H	L	H	H	H	H	H	H	H	H	H	H
7	M	M	H	VH	H	H	VH	VH	VH	H	M	H	H	H	VH
8	VH	CH	VH	VH	H	H	VH	VH	H	H	M	H	H	H	VH
9	H	M	VH	H	M	H	H	H	H	VH	H	H	H	H	VH
10	M	VH	H	M	H	H	H	H	H	H	H	H	H	H	H
11	M	H	M	L	H	H	H	VH	H	H	H	H	H	H	H
12	CH	CH	M	M	M	M	M	M	M	M	M	M	M	M	M
13	H	M	VH	M	H	M	L	M	H	H	VH	L	L	M	H
14	M	H	VH	VH	H	VH	CH	VH	H	H	VH	H	H	VH	H
15	M	L	H	VH	H	H	H	H	H	H	M	H	H	H	H
16	H	H	H	H	M	H	H	H	H	H	H	H	H	H	M
17	H	H	M	M	H	M	H	M	H	H	H	H	H	H	H
18	H	M	H	VH	CH	CH	H	VH	CH	M	H	M	M	M	H
19	M	H	VH	H	M	H	VH	CH	H	VH	H	H	M	VH	H
20	M	H	H	M	M	M	H	H	H	H	H	H	H	H	M
21	M	L	H	M	CH	VH	H	H	H	VH	H	H	M	H	H
22	L	L	M	M	M	M	M	L	H	H	H	H	H	L	H
23	VH	H	VH	CH	VH	CH	CH	VH	M	H	VH	M	M	H	H
24	H	H	H	VH	M	M	H	H	CH	H	VH	M	M	VH	VH
25	M	M	H	M	H	M	L	M	H	H	H	M	M	M	M
26	M	M	M	L	H	M	H	M	M	H	H	M	M	M	M
27	H	H	H	H	M	H	H	H	H	H	H	H	H	H	H
28	CH	L	H	VH	CL	CH	CH	VH	H	H	CH	CH	CH	CH	VH
29	H	VH	H	VH	CH	H	CH	H	VH	VH	H	H	H	H	H
30	M	M	H	M	M	H	H	H	H	H	H	H	H	H	H
31	M	M	M	M	H	H	H	H	M	H	M	M	M	M	M
32	L	H	VH	M	H	H	H	VH	M	M	H	M	L	H	H
33	M	M	H	M	M	H	M	H	M	M	M	H	M	H	L
34	H	H	VH	VH	H	H	H	H	H	H	H	H	H	H	H
35	H	H	H	H	M	M	H	VH	H	H	M	M	M	M	M
36	H	H	H	H	M	H	H	VH	VH	VH	M	H	H	VH	H
37	M	H	H	H	H	H	H	H	H	H	H	H	H	H	H
38	L	L	VH	M	H	H	H	H	H	H	M	M	H	H	H
39	H	H	H	H	H	VH	VH	CH	CH	CH	CH	CH	CH	CH	CH
40	H	M	H	H	VH	M	M	CH	H	H	M	M	L	M	L
41	H	H	H	M	H	L	M	H	M	H	M	H	VH	M	VH
42	H	M	L	VL	M	M	M	H	H	H	L	H	M	M	H
43	H	VH	H	H	H	H	H	H	H	H	H	H	H	H	H
44	CH	L	CH	CH	M	CH	CH	CH	CH	CH	CH	VH	VH	VH	VH
45	CH	CH	CH	VH	CH	H	M	VH	CH	VH	CH	H	VH	VH	CH
46	H	VH	VH	M	H	H	VH	M	CL	H	M	M	L	VH	L
47	H	M	M	H	VH	H	H	VH	VH	H	H	H	H	H	H
48	VH	VH	CH	H	L	VH	VH	VH	VH	VH	VH	VH	VH	VH	H
49	H	M	M	H	H	H	H	H	M	M	M	M	M	H	M
50	M	M	M	M	M	M	H	H	M	M	H	M	M	M	M
51	M	VL	VH	M	H	M	H	VH	M	M	VH	CH	H	CH	H
52	L	CH	VH	M	CH	VH	CH	H	L	H	VH	H	H	VH	CH
53	H	H	H	H	VH	M	H	H	VH	H	H	M	H	M	M
54	H	H	H	H	L	H	H	H	H	H	H	H	H	H	H
55	M	H	M	VH	H	H	M	M	M	VH	VH	H	VH	M	M
56	VH	VH	CH	CH	VH	VH	VH	VH	VH	VH	VH	VH	VH	VH	VH
57	H	H	H	M	VH	VH	VH	H	VH	M	H	M	M	VH	VH
58	L	L	M	H	CH	H	H	M	H	M	H	M	M	H	M
59	H	H	M	M	M	H	VH	M	M	M	M	M	M	M	M
60	CH	CH	CH	CH	M	CH	VH	H	VH	VH	M	CH	CH	CH	CH
61	H	H	VH	H	M	H	H	VH	H	VH	H	H	H	H	H
62	M	M	M	M	M	H	M	M	M	M	M	M	M	M	M
63	L	M	M	M	M	L	M	M	M	M	M	M	M	H	M
64	M	H	M	H	VH	CH	VH	H	M	H	CH	M	M	CH	H
65	CH	CH	M	M	M	M	M	M	M	M	M	M	M	M	M

.

After collecting the questionnaire answers, the decision matrix can be constructed as

Table 4. Linguistic responses from students.

	Participant 1	Participant 2	……	Participant 65
C1	H	L		CH
C2	H	CL		CH
C3	H	CH		M
C4	H	CH		M
C5	H	CH		M
C6	H	M		M
C7	VH	H		M
C8	H	CH		M
C9	H	M		M
C10	H	CH		M
C11	H	H		M
C12	H	M		M
C13	H	H		M
C14	H	CH		M
C15	H	VH		M

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

. After that, the linguistic terms were transformed into IVPFNs in

Table 5. Decision matrix based on IVPFN transformation.

	Participant 1				Participant 2				……	Participant 65
IVPFN	${\mathit{u}}_1^{\mathit{L}}$	${\mathit{u}}_1^{\mathit{U}}$	${\mathit{v}}_1^{\mathit{L}}$	${\mathit{v}}_1^{\mathit{U}}$	${\mathit{u}}_2^{\mathit{L}}$	${\mathit{u}}_2^{\mathit{U}}$	${\mathit{v}}_2^{\mathit{L}}$	${\mathit{v}}_2^{\mathit{U}}$		${\mathit{u}}_{65}^{\mathit{L}}$	${\mathit{u}}_{65}^{\mathit{U}}$	${\mathit{v}}_{65}^{\mathit{L}}$	${\mathit{v}}_{65}^{\mathit{U}}$
C1	0.5	0.7	0.3	0.5	0.3	0.5	0.5	0.7		0.7	0.9	0.1	0.3
C2	0.5	0.7	0.3	0.5	0.1	0.3	0.7	0.9		0.7	0.9	0.1	0.3
C3	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C4	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C5	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C6	0.5	0.7	0.3	0.5	0.4	0.6	0.4	0.6		0.4	0.6	0.4	0.6
C7	0.6	0.8	0.2	0.4	0.5	0.7	0.3	0.5		0.4	0.6	0.4	0.6
C8	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C9	0.5	0.7	0.3	0.5	0.4	0.6	0.4	0.6		0.4	0.6	0.4	0.6
C10	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C11	0.5	0.7	0.3	0.5	0.5	0.7	0.3	0.5		0.4	0.6	0.4	0.6
C12	0.5	0.7	0.3	0.5	0.4	0.6	0.4	0.6		0.4	0.6	0.4	0.6
C13	0.5	0.7	0.3	0.5	0.5	0.7	0.3	0.5		0.4	0.6	0.4	0.6
C14	0.5	0.7	0.3	0.5	0.7	0.9	0.1	0.3		0.4	0.6	0.4	0.6
C15	0.5	0.7	0.3	0.5	0.6	0.8	0.2	0.4		0.4	0.6	0.4	0.6

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

based on Table 2. The normalized IVPFN matrix was then developed in

Table 6. Normalized decision matrix.

	Participant 1				Participant 2				……	Participant 65
IVPFN	${\mathit{u}}_1^{\mathit{L}}$	${\mathit{u}}_1^{\mathit{U}}$	${\mathit{v}}_1^{\mathit{L}}$	${\mathit{v}}_1^{\mathit{U}}$	${\mathit{u}}_2^{\mathit{L}}$	${\mathit{u}}_2^{\mathit{U}}$	${\mathit{v}}_2^{\mathit{L}}$	${\mathit{v}}_2^{\mathit{U}}$		${\mathit{u}}_{65}^{\mathit{L}}$	${\mathit{u}}_{65}^{\mathit{U}}$	${\mathit{v}}_{65}^{\mathit{L}}$	${\mathit{v}}_{65}^{\mathit{U}}$
C1	0.58	0.78	0.18	0.38	0.33	0.55	0.42	0.64		0.75	0.93	0.06	0.22
C2	0.58	0.78	0.18	0.38	0.11	0.33	0.64	0.88		0.75	0.93	0.06	0.22
C3	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C4	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C5	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C6	0.58	0.78	0.18	0.38	0.44	0.65	0.32	0.53		0.44	0.65	0.32	0.53
C7	0.68	0.87	0.10	0.27	0.55	0.75	0.22	0.42		0.44	0.65	0.32	0.53
C8	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C9	0.58	0.78	0.18	0.38	0.44	0.65	0.32	0.53		0.44	0.65	0.32	0.53
C10	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C11	0.58	0.78	0.18	0.38	0.55	0.75	0.22	0.42		0.44	0.65	0.32	0.53
C12	0.58	0.78	0.18	0.38	0.44	0.65	0.32	0.53		0.44	0.65	0.32	0.53
C13	0.58	0.78	0.18	0.38	0.55	0.75	0.22	0.42		0.44	0.65	0.32	0.53
C14	0.58	0.78	0.18	0.38	0.75	0.93	0.06	0.22		0.44	0.65	0.32	0.53
C15	0.58	0.78	0.18	0.38	0.65	0.85	0.13	0.32		0.44	0.65	0.32	0.53

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

.

Based on the normalized decision matrix, the ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$ and ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$ can be calculated in

Table 7. IVPFN matrix for ${\mathit{T}}_{\mathit{m}\times \mathit{n}}$.

	Participant 1				Participant 2				……	Participant 65
IVPFN	${\mathit{u}}_1^{\mathit{L}}$	${\mathit{u}}_1^{\mathit{U}}$	${\mathit{v}}_1^{\mathit{L}}$	${\mathit{v}}_1^{\mathit{U}}$	${\mathit{u}}_2^{\mathit{L}}$	${\mathit{u}}_2^{\mathit{U}}$	${\mathit{v}}_2^{\mathit{L}}$	${\mathit{v}}_2^{\mathit{U}}$		${\mathit{u}}_{65}^{\mathit{L}}$	${\mathit{u}}_{65}^{\mathit{U}}$	${\mathit{v}}_{65}^{\mathit{L}}$	${\mathit{v}}_{65}^{\mathit{U}}$
C1	0.08	0.12	0.97	0.99	0.04	0.07	0.99	0.99		0.11	0.18	0.96	0.98
C2	0.08	0.12	0.97	0.99	0.01	0.04	0.99	1.00		0.11	0.18	0.96	0.98
C3	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C4	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C5	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C6	0.08	0.12	0.97	0.99	0.06	0.09	0.98	0.99		0.06	0.09	0.98	0.99
C7	0.10	0.15	0.97	0.98	0.07	0.11	0.98	0.99		0.06	0.09	0.98	0.99
C8	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C9	0.08	0.12	0.97	0.99	0.06	0.09	0.98	0.99		0.06	0.09	0.98	0.99
C10	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C11	0.08	0.12	0.97	0.99	0.07	0.11	0.98	0.99		0.06	0.09	0.98	0.99
C12	0.08	0.12	0.97	0.99	0.06	0.09	0.98	0.99		0.06	0.09	0.98	0.99
C13	0.08	0.12	0.97	0.99	0.07	0.11	0.98	0.99		0.06	0.09	0.98	0.99
C14	0.08	0.12	0.97	0.99	0.11	0.18	0.96	0.98		0.06	0.09	0.98	0.99
C15	0.08	0.12	0.97	0.99	0.09	0.14	0.97	0.98		0.06	0.09	0.98	0.99

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

and

Table 8. IVPFN matrix for ${\mathit{P}}_{\mathit{m}\times \mathit{n}}$.

	Participant 1				Participant 2				……	Participant 65
IVPFN	${\mathit{u}}_1^{\mathit{L}}$	${\mathit{u}}_1^{\mathit{U}}$	${\mathit{v}}_1^{\mathit{L}}$	${\mathit{v}}_1^{\mathit{U}}$	${\mathit{u}}_2^{\mathit{L}}$	${\mathit{u}}_2^{\mathit{U}}$	${\mathit{v}}_2^{\mathit{L}}$	${\mathit{v}}_2^{\mathit{U}}$		${\mathit{u}}_{65}^{\mathit{L}}$	${\mathit{u}}_{65}^{\mathit{U}}$	${\mathit{v}}_{65}^{\mathit{L}}$	${\mathit{v}}_{65}^{\mathit{U}}$
C1	0.99	1.00	0.02	0.05	0.98	0.99	0.05	0.09		1.00	1.00	0.01	0.03
C2	0.99	1.00	0.02	0.05	0.97	0.98	0.09	0.15		1.00	1.00	0.01	0.03
C3	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C4	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C5	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C6	0.99	1.00	0.02	0.05	0.99	0.99	0.04	0.07		0.99	0.99	0.04	0.07
C7	0.99	1.00	0.01	0.03	0.99	1.00	0.03	0.05		0.99	0.99	0.04	0.07
C8	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C9	0.99	1.00	0.02	0.05	0.99	0.99	0.04	0.07		0.99	0.99	0.04	0.07
C10	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C11	0.99	1.00	0.02	0.05	0.99	1.00	0.03	0.05		0.99	0.99	0.04	0.07
C12	0.99	1.00	0.02	0.05	0.99	0.99	0.04	0.07		0.99	0.99	0.04	0.07
C13	0.99	1.00	0.02	0.05	0.99	1.00	0.03	0.05		0.99	0.99	0.04	0.07
C14	0.99	1.00	0.02	0.05	1.00	1.00	0.01	0.03		0.99	0.99	0.04	0.07
C15	0.99	1.00	0.02	0.05	0.99	1.00	0.02	0.04		0.99	0.99	0.04	0.07

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

, respectively.

Applying the values of Table 7 and Table 8, the IVPF-WASPAS value of each factor can be derived based on Equations (14)–(16).

Table 9. WASPAS values and ranks of factors.

	${\mathit{K}}_{\mathit{i}}$				${\mathit{H}}_{\mathit{i}}$				${\mathit{B}}_{\mathit{i}}$				$\mathit{S}\left({\mathit{B}}_{\mathit{i}}\right)$	Rank
IVPFN	${\mathit{u}}_{{\mathit{K}}_{\mathit{i}}}^{\mathit{L}}$	${\mathit{u}}_{{\mathit{K}}_{\mathit{i}}}^{\mathit{U}}$	${\mathit{v}}_{{\mathit{K}}_{\mathit{i}}}^{\mathit{L}}$	${\mathit{v}}_{{\mathit{K}}_{\mathit{i}}}^{\mathit{U}}$	${\mathit{u}}_{{\mathit{H}}_{\mathit{i}}}^{\mathit{L}}$	${\mathit{u}}_{{\mathit{H}}_{\mathit{i}}}^{\mathit{U}}$	${\mathit{v}}_{{\mathit{H}}_{\mathit{i}}}^{\mathit{L}}$	${\mathit{v}}_{{\mathit{H}}_{\mathit{i}}}^{\mathit{U}}$	${\mathit{u}}_{{\mathit{B}}_{\mathit{i}}}^{\mathit{L}}$	${\mathit{u}}_{{\mathit{B}}_{\mathit{i}}}^{\mathit{U}}$	${\mathit{v}}_{\mathit{B}}^{\mathit{L}}$	${\mathit{v}}_{{\mathit{B}}_{\mathit{i}}}^{\mathit{U}}$
C1	0.42	0.61	0.44	0.63	0.39	0.57	0.49	0.66	0.55	0.76	0.21	0.42	0.6660	14
C2	0.42	0.61	0.44	0.63	0.38	0.57	0.50	0.67	0.54	0.76	0.22	0.42	0.6653	15
C3	0.45	0.64	0.40	0.60	0.43	0.62	0.44	0.62	0.59	0.80	0.18	0.37	0.7055	3
C4	0.43	0.62	0.42	0.62	0.41	0.59	0.46	0.64	0.56	0.78	0.20	0.40	0.6825	10
C5	0.43	0.63	0.42	0.62	0.40	0.59	0.48	0.65	0.56	0.78	0.20	0.40	0.6824	11
C6	0.44	0.63	0.41	0.61	0.42	0.61	0.45	0.63	0.58	0.79	0.19	0.38	0.6927	6
C7	0.46	0.65	0.39	0.59	0.44	0.62	0.43	0.61	0.60	0.81	0.17	0.36	0.7112	2
C8	0.45	0.65	0.39	0.59	0.44	0.63	0.43	0.61	0.60	0.81	0.17	0.36	0.7121	1
C9	0.43	0.63	0.42	0.61	0.41	0.60	0.47	0.64	0.57	0.78	0.20	0.39	0.6860	9
C10	0.44	0.64	0.41	0.60	0.43	0.62	0.44	0.62	0.59	0.79	0.18	0.37	0.7018	4
C11	0.43	0.63	0.42	0.61	0.42	0.60	0.45	0.63	0.57	0.78	0.19	0.39	0.6897	7
C12	0.42	0.61	0.44	0.63	0.40	0.59	0.47	0.64	0.56	0.77	0.20	0.40	0.6753	12
C13	0.42	0.61	0.44	0.63	0.40	0.58	0.48	0.65	0.55	0.76	0.21	0.41	0.6709	13
C14	0.44	0.64	0.41	0.60	0.42	0.61	0.46	0.64	0.58	0.79	0.19	0.38	0.6957	5
C15	0.43	0.62	0.42	0.61	0.41	0.60	0.46	0.63	0.57	0.78	0.19	0.39	0.6878	8

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

presented the IVPFN as well as its defuzzied values (i.e., $S\left(B_i\right)$) of each factor. Based on the defuzzied values, the ranks of each factor can be obtained.

4.3. Sensitivity Analysis

Due to the property of IVPF-WASPAS, the ranks of each factor could vary under different values of $\alpha$ [34]. Therefore, the sensitivity analysis was conducted by varying $\alpha$ from 0.1 to 0.9 to test the result robustness. The defuzzied WASPAS values and ranks of each factor are presented in

Table 10. Defuzzied WASPAS values for sensitivity analysis.

$\mathit{\alpha }$	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
C1	0.6537	0.6568	0.6599	0.6630	0.6660	0.6690	0.6720	0.6749	0.6777
C2	0.6480	0.6525	0.6568	0.6611	0.6653	0.6695	0.6736	0.6775	0.6815
C3	0.6967	0.6990	0.7012	0.7034	0.7055	0.7077	0.7098	0.7119	0.7140
C4	0.6715	0.6743	0.6770	0.6798	0.6825	0.6851	0.6878	0.6904	0.6929
C5	0.6677	0.6715	0.6752	0.6789	0.6824	0.6860	0.6894	0.6929	0.6962
C6	0.6839	0.6861	0.6883	0.6905	0.6927	0.6948	0.6969	0.6990	0.7011
C7	0.7022	0.7045	0.7068	0.7090	0.7112	0.7134	0.7155	0.7177	0.7198
C8	0.7045	0.7064	0.7083	0.7102	0.7121	0.7140	0.7158	0.7177	0.7195
C9	0.6738	0.6769	0.6800	0.6830	0.6860	0.6889	0.6918	0.6947	0.6975
C10	0.6957	0.6973	0.6988	0.7003	0.7018	0.7033	0.7048	0.7063	0.7077
C11	0.6813	0.6834	0.6856	0.6877	0.6897	0.6918	0.6939	0.6959	0.6979
C12	0.6674	0.6694	0.6714	0.6733	0.6753	0.6772	0.6791	0.6810	0.6829
C13	0.6615	0.6639	0.6662	0.6686	0.6709	0.6731	0.6754	0.6776	0.6798
C14	0.6837	0.6868	0.6898	0.6928	0.6957	0.6986	0.7014	0.7043	0.7070
C15	0.6794	0.6815	0.6836	0.6857	0.6878	0.6899	0.6919	0.6940	0.6960

Remark: Please refer to Table 2 for the definitions and details of C1 to C15.

and Figure 2, respectively. The sensitivity analysis indicated that our results are robust. Specifically, factors 7 and 8 always rank first and second under different scenarios, indicating that both factors should be prioritized to improve the learning outcomes of the industry–university collaboration course in logistics management. Similarly, factor 3 ranks in the third place and factor 4 ranks in the fourth place for all scenarios, indicating their importance for course developers and lecturers.

5. Discussion

Based on the results of the IVPF-WASPAS model as well as the sensitivity analysis, it can be identified that the most important factors for the outcomes of the logistics industry–university collaboration course are factors 8 and 7, followed by factors 3, 10, and 14. Therefore, this study will focus on these factors and discuss the approaches to enhance the learning outcomes.

First, factor 8 ranks as the most important element based on the students’ questionnaire. This means that the working environment of the course is important for the learning outcome. This is consistent with the previous literature that suggested the environment comforts can directly influence the learning outcomes, and appropriate learning and teaching environments should be created for the learners [37]. From the literature, the results as to why this factor is valued most by the students can be reasonably inferred as that it is probably a good working environment that can help students quickly adapt to the new norms of their study. Therefore, the course developers need to prioritize the development of good working environments for the students. To achieve this, staff from the university need to work with the managers from the companies to create a good physical environment, such as good dormitory and workplace, and a good team atmosphere, such as friendly tutoring and good team allocations.

Second, factor 7 which concerns the students’ own ability is also an important element for successful course learning. This is not surprising and has been witnessed in the previous literature for logistics management education (e.g., [10]). For this industry–university collaboration course, it simulates how logistics companies really operate in practice; students are in essence asked to behave like the normal employees of logistics companies so that they can finish the tasks allocated. To achieve this, multiple abilities, not just the learning ability in the classroom, should be equipped. As suggested by [10], these important abilities for students in logistics management major can possibly be grasped by the efforts of the instructors and the students. This may give the course developers the hint that, to make students benefit more from the course, they should purposefully cultivate the students’ different abilities to work under pressure, manage time, solve problems, communicate with team members, etc. before the course by giving students mixed practices, instead of expecting students to grasp the important abilities by themselves.

In addition, factor 3 ranks in the third place, meaning that students should be self-motivated to obtain better learning outcomes in this course. Such experience from students is suggested by the previous literature (e.g., [36]). This may be due to the fact that this course is practice-oriented, in contrast to the traditionally taught courses in classrooms in the logistics discipline. The taught courses are usually delivered by a lecturer who will set the clear objectives for the students. Therefore, even though the students have no specific aims in the classroom, they can simply follow the lecturer. This may inform course developers that to make students have better outcomes of learning this course, they need to provide students with the experience and skills to set and make appropriate course objectives, even before the beginning of the course.

Factor 10 also reveals that students are the centers of obtaining the good learning outcomes. They can benefit more if they take the work seriously, instead of overlooking or ignoring it. This is consistent with the previous literature (e.g., [10]). However, ignorance of the work can possibly happen, due to the fact that the approach in which this course is delivered is largely different from the traditionally taught courses. The students could have the sense that the course is not formal enough, and thus put fewer efforts in it. Therefore, to achieve a good learning outcome, the tutors from the companies and the lecturers from the university should both emphasize the importance of the course for students.

Finally, factor 14 indicated that to enhance the learning outcomes, the after-course performance evaluation should be fair and transparent. On the one hand, this may be due to the fact that if the evaluations are not fully fair or inclusive, it can demotivate the students and make them lose interest in the course. Therefore, they may not put an effort in summarizing the experiences and knowledge learnt from the course. On the other hand, as suggested by the previous literature (e.g., [42]), if the standards and rules of marking and performance evaluation are not clear and transparent enough, it is difficult for the students to identify their weakness, and thus the direction to work on, hindering their way of continuous improvement.

To sum up, it can be seen that in all stages of the course (i.e., before/during/after the course), there are important factors that need to be considered by the course developers, lecturers, and tutors. This means the course should be delivered by a “life-cycle” approach to cover all periods of the study. As suggested above, it can be reasonably argued that this insight is essentially consistent with the ideas of Gagné’s theory on conditions of learning [29], informing course designers and instructors to collaborate with industrial practitioners to develop activities in all stages of the courses to improve the learning outcomes, instead of just focusing on a single stage. Also, under such a thought, practitioners for industry–university collaboration courses in logistics management can benefit from using Gagné’s Nine Events of Instruction model [29,30] in course design.

6. Conclusions

In this study, the factors influencing the learning outcomes of industry–university collaboration courses in logistics management discipline were investigated. The study adopted a mixed method. First, the qualitative explorations were conducted, identifying 15 factors that can determine the course learning outcomes across the periods before/during/after the course. After that, the novel modeling approach was built by IVPF-WASPAS to rank the factors by their relative importance. The study found that the most important factors are appropriate work environments as well as the students’ own ability, followed by the students’ attitudes and learning objectives, students’ seriousness to work, and students’ continuous learning and application after the course. The study revealed that all periods of the course (i.e., before/during/after the course) are important for outcome improvement. Therefore, this study advocates that the course designers and lecturers should follow a life-cycle style to prepare and implement the course, instead of only focusing on the teaching during the course.

This study can provide an academic contribution from the following perspectives. First, this study identified the influential factors for industry–university collaboration courses in logistics management discipline, providing a theoretical basis for future studies to develop the course evaluation index system in logistics management and other relevant disciplines. Also, this work creatively applied the IVPF-WASPAS method in the logistics education field, confirming the method’s potential in management educational studies. This can largely enrich the methodological tools for educational analysis in logistics and other management fields. Finally, this study adopted a temporal view to analyze the influencing factors of course outcomes, covering the pre-, during, and post-course stages. The case study in this paper confirmed the value of such analytical view, and it can support the current logistics educational studies to better design assessments for education quality.

Practically speaking, this study can inform different practitioners and improve the learning and studying quality. For course designers, this study may shed light on the importance of life-cycle course design and preparation, as well as inspire the designers about opportunities to better arrange course activities and scheduling. For lecturers and tutors of the courses, this study can inform them about how students in this type of courses should be taught, evaluated, and motivated to enhance their learning outcomes. The findings of this study can call for different teaching modes compared with the traditional classroom courses. Also, the students can benefit from the findings by developing the important abilities identified in this study. Apart from the educators and students, our results can also inform the different stakeholders in logistics industry sector. For example, the managers in logistics companies or manufacturing companies can possibly benefit from our findings to adjust their training programs in the companies, make human resource management policies, and develop instruction courses for early career logistician. Also, our paper sheds light on how the industry–university collaboration projects can be conducted, informing managers to more appropriately collaborate with educational institutes to achieve fruitful outcomes.

Although this study has academic and practical contributions, we acknowledge that there are still limitations. First, this study was conducted in China. Therefore, the results may be different from other countries due to different structures and policies in higher education systems [6]. In addition, the market differences across countries may lead to the change in our results, as logistics needs across markets can differ. This can lead to the difference in logistics student training and the design of industry–university collaboration courses in higher education institutes. From a disciplinary perspective, as this study only considered the logistics management field, the generalization of the results may be only limited to the relevant field, such as business administration or operations management, but not necessarily valid in the engineering discipline. Methodologically speaking, as the qualitative nature of the IVPN-WASPAS method [27], there could be subjectivity in the fuzzy modeling, causing potential judgmental bias in the results. In addition, we only considered the most widely used operations of IVPN-WASPAS without comparing advanced model extensions. As both IVPN and WASPAS can have multiple extensions, such as the modifications on aggregation operations [44] and defuzzification and scoring [44,45], it can be argued that the extension might bring potential changes to the results.

Therefore, there could be several interesting future directions to explore. First, the future studies can consider extending the research contexts by considering different countries, multiple disciplines, and various industry–university collaboration modes when studying course outcome factors. Also, as we only applied subjective data (i.e., student feedback and questionnaire responses), future research can supplement our study by using some objective data, such as students’ test scores or report performance to validate our findings. Empirical and statistical methods may need to be considered for use, such as regression and structural equation modeling. By applying them, future studies can be expected to triangulate the findings in this study. Also, future research can extend the IVPN-WASPAS method by developing new analytical tools for logistics education, such as using other IVPN operations [44,45] or altering IVPN by other types of fuzzy numbers like intuitionistic fuzzy numbers [46]. To achieve this, different types of fuzzy numbers can be integrated into the linguistic transformation process, and different types of operations can be applied for aggregation. Future research can compare how different types of fuzzy numbers and operations can support the identification of the factors and examine the robustness of the results under these modifications.