1. Introduction
Assessment processes of the perceived quality are nowadays assuming a crucial role to support the organization success. They can suitably provide for the definition of improvement policies and even represent a fundamental step in the continuous improvement cycle [
1]. Furthermore, they allow us to evaluate the customer satisfaction level vs. operational-managerial aspects. Finally, the performance assessment processes deeply support the effectiveness of the internal and external communication system concerning the achieved performance levels and the organization capability to reach the promised performance levels [
2]. In the service context, the precise and accurate assessment of the provided quality level is characterized by a certain complexity. In the first instance, the service quality is not an absolute and exclusive service feature, independent from customer’s perceptions, as it is in the typical tangible production. Actually, the service quality is an unphysical and latent entity that fully involves the customer cognitive sphere. Particularly, it cannot be directly measured, namely its assessment is indirectly performed by taking into account suitable service manifest aspects (i.e., items) with regard to the fundamental service performance aspects (i.e., dimensions). The relationship between items and related dimensions can be formalized by means of appropriate service quality conceptual models [
3].
Based on the previous considerations, the aim of the present work is to accurately and precisely measure the quality level perceived by students of an academic context with relation to the e-service provided through the academic web portal. The literature in this area appears to be lacking despite the great relevance of such a context [
4]. Actually, the academic e-service involves all the students at the university, and through it, the academic organization is introduced, tasks and related services for students are described and delivered, and information deemed useful are maximally disseminated. Furthermore, the academic e-service quality affects the students’ academic efficiency, perceived atmosphere of comfort and order, as well their overall satisfaction [
5]. The latter impacts on the students’ behavioral intention and aptitude to spread word-of-mouth, which affect both the university reputation and students’ retention [
6,
7]. These aspects appear to be crucial in the current academic competitive context [
8]. As a result, the academic e-service quality needs to be at the highest possible level, and consequently it appears to be necessary to measure it continuously via a reliable measurement tool [
9]. In the light of previous considerations, in this paper a conceptual model for measuring the academic e-service quality is developed and validated, thus overcoming the lack of literature in this relevant context. For this purpose, an effective and easy-to-use methodological approach that combines engineering and statistical approaches to support the structuring of a parsimonious and robust measurement model of the service quality is herein proposed. Such a further aspect represents an additional contribution of the present work. Finally, a survey is conducted taking into account viewpoints and perspectives of students at the University of Palermo, and highlighted academic e-service quality shortcomings and criticalities are stressed and discussed.
The remainder of the present paper is organized as follows. The literature analysis is reported in
Section 2, whereas materials and methods are supplied in
Section 3.
Section 4 synthesizes obtained results with reference to the empirical analysis conducted. Discussions and conclusions are finally reported in
Section 5.
2. Literature Analysis
The service quality is an entity closely related to the customer satisfaction [
10]. According to Petrick [
11], the service quality represents the user cognitive assessment as regards a given service, and the customer satisfaction refers to the pleasurable way by which the perceived service performance makes them feel. Lehtinen and Lehtinen [
12] defined the service quality in terms of physical, interactive and business quality attributes. Physical quality refers to the service tangible aspects, interactive quality to the customer-provider interaction, while the business quality refers to the provider image and reputation. A further conceptualization of the service quality was proposed by Rust and Oliver [
13], who suggested a three-component quality model, namely the customer–provider interaction, service environment and results. Grönroos [
14] stated that the service quality includes the functional quality, the technical quality and the provider image. In particular, the functional quality focuses on how the service is delivered, the technical one focus on what it is provided, while the provider image represents a mediator between the functional and technical quality. The ServQual instrument [
15] represents the first conceptual model specifically developed for measuring the service functional quality. It theoretically represents the service quality as a multidimensional entity on two levels. At the upper level the dimensions are stated, while at the basic one the items are considered. In particular, ServQual consists of 5 dimensions and 22 items, and its implementation involves the conducting of a double survey with the aim of capturing both the customer perceived quality and expectations. Cronin and Taylor [
16] disapproved the double ServQual questionnaire. Accordingly, the authors proposed a new conceptual model based only on the service performance assessment. Such a model hypothesizes the service quality as a one-dimensional construct in which related items are directly considered for evaluating the service quality.
Considering the higher education sector, the creation of a more suitable and pleasing learning environment along with the providing of high-quality education-related services nowadays represent the fundamental driver for facing the highly competitive pressure related to student recruitment, retention and loyalty [
17,
18]. In particular, the student satisfaction that arises from the providing of highly performing education-related services significantly affects the results of the so-called “ranking war” [
19]. In Italy, the idea of evaluating the service quality in the academic context is quite recent. The Legislative Decree n. 19/2012 [
20] introduced the implementation of an initial and periodical accreditation system, the periodical assessment of the education-related services and the employment of an effective internal and external communication system as mandatory. In particular, the National Agency for the Evaluation of Universities and Research Institutes (ANVUR) provides the procedures to be used for the self-assessment, periodical assessment and accreditation centered on requirements specified by the ISO 9001:2015—
Quality management systems [
21]. The fundamental aim is to promote the providing of education-related services deeply centered on the student needs/necessities.
E-Service Quality
With the great use of the e-service and e-commerce, in recent years many researchers developed conceptual models and approaches for measuring the e-service quality. Compared to the measurement tools considered for evaluating the quality level of typical services, the consideration of the hedonistic and utilitarian value takes over in these models. For example, Loiacono et al. [
22] developed the WebQual tool composed by 12 dimensions and 36 items on the basis of the theory of reasoned action [
23]. Yoo and Donthu [
24] developed the SiteQual model consisting of 4 dimensions and 9 items. This model is able to measure the e-service quality as regards e-commerce activities. More recently, the WebQual 4.0 model was developed on the basis of the ServQual scale, and it is composed by 23 items within 3 dimensions [
25]. Further service quality conceptual models recently developed in literature taking into account different e-service settings are shown in [
26,
27,
28,
29,
30]. Regarding the academic e-service quality, to the best of the authors knowledge, the literature appears to be lacking despite the great relevance assumed by this issue. For that reason, a parsimonious and robust measurement model of the academic e-service quality was developed and validated, as subsequently detailed.
4. Empirical Analyses Results
4.1. Context Analysis and Evaluation of Existing Literature
As aforementioned, in the present work, a conceptual model for measuring the academic e-service quality was developed and validated. This e-service involves groups of users, namely students, graduates, academics, and employees, among others. These users can be characterized by distinct needs/necessities to be satisfied and even expectation levels. Thus, perspectives and viewpoints about the academic e-service quality can present significant differentiation elements with reference to the considered users group. For example, academics may be greatly concerned by academic e-service aspects marginally taken into account by employees or students. For such a reason, the present work takes into account the students’ viewpoints about the academic e-service quality, as key stakeholder of the considered e-service. As regards existing literature in this field, to the best of the authors’ knowledge, it appears to be lacking. Hence, reference was made to the relevant context of the software and the website quality. First of all, the ISO/IEC 25010: 2011 [
38,
39], which introduces a software quality model including several relevant quality attributes. In addition, the following references were also considered. Baharuddin et al. [
40] analyzed 25 quality dimensions for mobile applications which were synthesized and prioritized obtaining the 10 most important usability dimensions. Coursaris and Kim [
41] developed an adapted an evaluation framework for the context of the mobile computing environment. Han et al. [
42] developed an empirical quality model able to point out the functional relationships among usability criteria and user interface aspects. Lupo and Bellomo [
43] developed a methodological framework based on a DANP model for evaluating the software quality in terms of usability. Orehovački et al. [
44] proposed an articulated framework including six dimensions comprising 33 attributes, for evaluating the quality-in-use of Web 2.0 applications. Seffah [
45] unified existing standards and models into a consolidated hierarchical quality model.
4.2. Items Generation and Revision Development of the Preliminary Questionnaire
This phase was carried out on the basis of the literature analysis and via the support of brainstorming activities involving highly experienced experts in the field. The brainstorming is a technique widely used in the problem-solving context [
46], which has found effective applications in different industrial settings, such as teaching social studies [
47], software development [
48], strategies for tourism development [
49], etc. The focus was initially on aspects regarding the website usability, since the latter represents a crucial feature as regards the website attractiveness and perceived quality. Thus, related items were generated. Subsequently, focusing on activities that the student carries out through the website employment appeared to be evident as the website has to assure safety, effectiveness and efficiency. Therefore, also in this case, the generation of related items was done. Finally, aspects related to the website availability and reliability of fundamental contents were considered, and related items were generated. From this analysis, 27 items were obtained as shown in
Table 1.
After this phase, the revising of the items generated was done in order to select the most effective ones. For this purpose, the quantitative criterion developed by Lawshe was considered in view of its easy-of-use and high reliability level [
50]. In particular, an experts panel composed by eight members was specifically selected to carry out this task, and as regards validity of both the formulation and content of the generated items, each involved expert provided the following judgment: “
essential” or “
useful, but not essential” or “
not essential”. Then, collected evaluations were considered to assess the agreement levels among experts through the so-called Content Validity Ratio (CVR) index, as regard validity of the items formulation (CVR
F) and content (CVR
C), by using the relationship reported below:
where:
Wilson et al. [
51] provided the validity acceptance/rejection criterion by suggesting reliable critical values for the CVR index. In particular, for a significance level equal to α = 0.95, the CVR critical value is equal to 0.69, as shown in
Table 2.
Rejected items were excluded or revised according to their detailed evaluation results, while those deemed valid were directly considered. For example, as can be seen from
Table 1, the items “
The renewal of the password every 120 days is safety” and “
Access to the website (login/logout) is safe” were merged into a single item since both aim at measuring the security service construct, thus obtaining such a new item: “
The password management guarantees the access security”. On the contrary, those items that simultaneously focus on several service aspects were splinted. For example, the item “
The appearance of the website is pleasant, and the contents are clear”, which aims to simultaneously investigate the “
pleasantness of appearance” and “
clarity of the content”, was split into two items: “
The appearance of the website is pleasant” and “
The contents of the website are clear”. Finally, items which were considered unclear were reformulated. At the end of this revising process, 20 final items were obtained as shown in
Table 1. Then, these items were transposed into the preliminary questionnaire, which was considered to collect data needed for the subsequently detailed statistical analyses.
4.3. EFA/CFA and Development of the Final Questionnaire
On the basis of the developed preliminary questionnaire, a web-based investigation in the period from December 2019 to February 2020 was conducted involving a sample of 285 students selected from those of engineering, architecture and education science degrees, and data collected were considered to perform EFA and CFA. In particular, for each questionnaire the missing answers percentage was assessed, and were excluded those questionnaires characterized by a value of such a percentage greater than 5%, i.e., four questionnaires. Moreover, questionnaires that presented the same level of response on all considered items were also excluded, as such situations suggested a low level of interest of related respondents (i.e., acquiescent respondents), and 21 questionnaires were excluded. The next phase was related to the testing of the data univariate and multivariate normality. To verify the data univariate normality, the Kolmorov–Smirnov and Shapiro–Wilk tests were conducted, while the data multivariate normality was tested through the multivariate Curtosi coefficient. These analyses highlighted five anomalous questionnaires, which were excluded. The final number of valid questionnaires was equal to 255. To support EFA and CFA below detailed, suggestions reported in Hair et al. [
52] were considered.
4.4. EFA
The feasibility of EFA needs to be preliminarily verified via suitable tests. In particular, it is necessary to conduct both the Bartlett’s test for verifying the items correlation matrix significance, as wells the Kaiser–Meyer–Olkin (KMO) test required for confirming the sample size adequacy. In the herein treated case, the Bartlett’s test was significant and the KMO test presented a value of 0.87, greater than the threshold of 0.70. Moreover, three factors were extracted considering the maximum likelihood method and according to the Catter criterion [
53].
Table 3 shows the obtained factor loadings, the total variance explained and the items communalities, namely the fraction of the items’ information shared with the factorial model.
As can be seen from
Table 3, the factorial model is characterized by some extremely low factor loadings values, particularly as regards to items 3, 10, 11, 19, 13, 17 and 20. In addition, these items are even characterized by extremely low communality values, highlighting a poor factorial model fitting. Thus, a new EFA was carried out by excluding these items, and the related obtained results are shown in
Table 4.
The obtained new factorial model was considered satisfactory since it presents acceptable levels of total variance explained, factor loadings and items communalities. Factor 1, which is characterized by items 6 (U1), 16 (U2), 1 (U3), 5 (U4), 7 (U5) and 12 (U6), respectively, includes those service aspects related to the Website usability (D1). Factor 2, which is characterized by items 9 (S1), 4 (S2), 14 (S3) and 8 (S4), respectively, includes those service aspects related with the Website security (D2). Factor 3, which is characterized by items 2 (C1), 15 (C2), 17 (C3), respectively, includes those service aspects related to the Website fundamental contents (D3).
Finally, the internal consistency of extracted factors was verified, and in such a regard, the Cronbach’s Alpha was considered:
where
is the item number within the factor;
is the total score variance;
is the item variance, with i = 1, …, k.
A Cronbach’s Alpha value greater than the threshold of 0.70 reveals an adequate internal consistency [
54] and, thus, the obtained results were considered satisfactory, as shown in
Table 4. In the light of previous considerations, EFA results were deemed suitable to perform CFA subsequently in detail.
4.5. CFA
The factorial model fitting was verified by the Comparative Fit Index (CFI) and the Root Means Square Error of Approximation index (RMSEA). CFI values greater than 0.90 indicate acceptable model fit. On the contrary, RMSEA ranges from 0 to 1, with smaller values indicating a better model fit. A value of 0.06 or less is indicative of acceptable model fit.
Table 5 shows obtained results.
Finally, validity as measurement tool of the pointed out factorial model was assessed with reference to effectiveness of its extracted factors. The convergent validity was tested through the Composite Reliability (CR) and the Average Variance Extracted (AVE). In particular, CR and AVE values greater or equal to 0.7 and 0.5, respectively, are indicative of a good convergent validity of the factorial model. Instead, the discriminant validity was tested considering the Maximum Shared Variance (MSV) and the square root of AVE. In particular, the factorial model can be confirmed in terms of discriminant validity if MSV values ore less than AVE values, and values of the AVE square root are greater than the correlation coefficients between factors [
55].
Table 6, in which values of the AVE square root are shown in bold, while those of the correlation coefficients between factors are shown in italics, shows obtained results.
On the basis of these results, the framework composed by 13 items within three dimensions shown in
Table 4 represents a valid, reliable and parsimonious ACademic E-service QUALity (ACEQUAL) measurement model. Finally, such a conceptual model was transposed into the final customer satisfaction questionnaire shown in
Appendix A.
4.6. Survey and Results Analysis
Based on the developed final customer satisfaction questionnaire, a web-based survey was conducted to assess the academic e-service quality provided at the University of Palermo. In particular, 218 students were involved in the period between May/June 2020 by using the Google forms application, in consideration of the pandemic period. In detail, students involved, mainly of engineering degrees but, in a limited manner, also those of architecture and education sciences were selected in relation to opportunity reasons related to the necessity to carry out such a study, within a reasonable time, during the first wave of COVID-19. Thus, the respondents’ sample was not purely random and extended to all the disciplinary areas within the University of Palermo. Actually, the latter represents the main limitation of the present study.
Table 7 shows the profile of respondents involved.
Most of the respondents are aged between 19 and 21 years, 25% of them belong to the 22–24-years class, and about 20% of them to the 25–27-years class. Regarding the website frequency of use, 72% of respondents declared at least a weekly use of the academic e-service, confirming that most of them are characterized by a high knowledge level of the e-service features. Collected questionnaires were examined for verifying their validity, and 22 questionnaires were excluded as incomplete or related to acquiescent respondents. Thus, in total, 196 questionnaires were considered as valid to carry out such an analysis.
Figure 2 shows the average scores as well as related standard deviations with relation to the investigated ACEQUAL items and overall satisfaction aspects.
As can be seen, the overall satisfaction levels of involved students as regards the academic website (G1), student web portal (G2) and their perceived quality level (G3) are quite high, namely 7.74/10, 8.08/10 and 7.98/10, respectively. Thus, the quality level of delivered academic e-service, taken as a whole, is seen as moderately high. As regards the items evaluation, considering the dimension D1 “Website usability”, except for the item U4 “It is easy to interact with the website”, which is scored equal to 3.51/5, all other items are characterized by average scores less than 3, representing the neutrality level of the considered evaluation scale. In particular, the item U3 “The appearance of the website is in harmony with the academic context” is the one with the lowest score equal to 2.79/5. Taking into account the dimension D2 “Website security”, it should be highlighted that all its items are characterized by scores higher than the neutrality level of the evaluation scale. Particularly, the item S2 “The users’ privacy is protected (personal data protection)” is the one with the highest score equal to 3.92/5. Finally, the dimension D3 “Website fundamental contents” the item C2 “The availability of pre-filled forms is useful” is scored equal to 3.00/5, while the other two items are evaluated with higher scores.
Obtained results are particularly relevant in the light of their high capability to measure in precise and accurate way the latent constructs of the academic e-service quality. The latter arises from the high validity and reliability of ACEQUAL. Therefore, these results can reliably generate targeted and focused improvements actions aimed at the improvement of the academic e-service quality.
5. Discussion and Conclusions
With reference to the academic e-service, the fundamental purpose of the present work was to show how important the service quality is, and how relevant can be its accurate and precise measurement process. Actually, the competitive pressure related to the student recruitment, retention and loyalty which is characterizing the higher education sector, encourages towards the delivering of highly performing education-related services, such as the academic e-service. In particular, the latter involves all the students at the university and allows them to take advantage of the delivering of several highly functional and useful services concerning both the education context and the organizational context. Moreover, the quality level perceived by students as regards the academic e-service, as education-related service, represents an antecedent of their overall satisfaction, which represents a crucial aspect for dealing the current academic competitive context.
For these reasons, and even considering the lack of literature in this relevant field, a parsimonious and robust measurement model of the ACademic E-service QUALity (ACEQUAL) was developed and validated, on the basis of the experts’ suggestions and recommendations and even taking into account viewpoints and perspectives of students at the University of Palermo. Furthermore, the relevant aspects regarding both the generation of the fundamental items, as well the development and validation process of the service quality measurement model, were based on an in-depth literature analysis. The suitability of ACEQUAL for measuring the academic e-service quality was established via its convergent and discriminant validities, namely the suitability of ACEQUAL items for measuring the academic e-service quality constructs, and the satisfactory discrimination level among meanings covered by the ACEQUAL dimensions, respectively. Thus, the ACEQUAL capability to measure in an accurate and precise way the academic e-service quality was quantitatively supported. On the contrary, to the best of the authors’ knowledge, methodologies developed in literature in this field of investigation are typically based on Multi-Criteria Decision-Making (MCDM) approaches. In particular, the academic e-service quality is typically scored on the basis of suitable criteria, and eventually underlying items, directly generated considering experts and decision makers proposals, literature analyses and in some cases survey results. In other words, quantitative analyses able to show the model capability to effectively measure the service quality are not considered. In this respect, for more details, the reader can refer to [
56,
57,
58,
59]. Thus, the present work represents a first attempt at evaluating the quality of the academic e-service via a measurement model in which validity and reliability are quantitatively supported. This aspect represents the main contribution of the present paper. Moreover, in the present work, it was deemed necessary to propose an effective methodological approach to support the structuring process of a service quality measurement model. Such an approach unifies and consolidates the use of techniques and methods whose effectiveness and practicality have been widely shown as regards numerous and diversified industrial settings. For example, the implementation of brainstorming activities and even quantitative approaches such as that developed by Lawshe [
50] for carrying out the items generation and revision phases, represent for the authors an innovation in the field of inquiry. Finally, a survey was conducted, and highlighted service quality shortcomings and criticalities were stressed and discussed.
Future research developments in this area may concern two distinct traits. In the first instance, the students sample required to carry out the development and validation of the measurement model could be defined in a fully random manner and even extended to all disciplinary areas of the University of Palermo, so as to take into account more diversified viewpoints of perspectives about the academic e-service quality. In this way, the main limitation of the present study related to the representativeness level of the considered sample, which was defined with some limitations due to the pandemic period, could be overcome. Additionally, such a study could involve further academic settings. Actually, different performance levels of the provided academic e-services in terms of website usability, security and contents, but also with reference to its interactivity and user interfaces, which were not relevant in the considered study, could generate different expectation/need levels in the related stakeholders. Due to the latter, it may not be likely to confirm the herein obtained service quality structure in terms of dimensions and related items. However, the latter would allow a better generalization as well greater robustness of the ACEQUAL model.