An Experience-Based Framework for Evaluating Tourism Mobile Commerce Platforms

This research presents and studies an evaluation framework for tourism mobile commerce platforms based on tourists’ experience. Synthesizing from prior literature, relevant theories, and the results of online questionnaires, we select 24 evaluation indices for preliminary evaluation. Using exploratory factor analysis method, we then extract from these indices the following five principal factors: interactive experience, infrastructure experience, personalization experience, product or service quality experience, and product operation experience. We further employ the confirmatory factor analysis to test the construction of the evaluation framework and demonstrate that the evaluation framework is both robust and effective. Finally, based on our proposed evaluation framework, we empirically evaluate the most popular mobile commerce platforms (Ctrip and Qunaer) in China by using fuzzy comprehensive evaluation method.


Introduction
The proliferating wireless technologies have enabled consumers to increasingly interface and interact with mobile commerce (m-commerce) systems for transactions.In China, mobile devices have become ubiquitous in people's daily activities, resulting in about 50% of e-commerce transactions completed through mobile platforms in comparison to about 20% in the United States and 33% in the United Kingdom.
Among all the m-commerce transactions, online bookings through mobile devices have become increasingly popular.For instance, 25% of total online bookings were made from mobile terminals in 2016 in the United Kingdom, up from 12% three years ago [1].In the United States, the digital travel sales through mobile platforms were expected to exceed 50 billion in 2016 and reach 70 billion by 2018 with 35% of online bookings being mobile [2].Among all the markets, China is the leader in mobile bookings with a projected 60% of online bookings being made on a mobile device by 2017 [3].
Since mobile terminals are driving the increase in the overall traffic of online travels, it is important for tourism companies to ensure the appropriate functioning of their m-commerce platforms so as to optimize the allocation of tourism resources and present their products and services to tourists in a meaningful way to attract more online transactions.Therefore, evaluating different m-commerce platforms used in the tourism industry and analyzing the appraisal results will help m-commerce platform providers to ensure the quality of tourists' experience and the improvement of their customer loyalty.An effective evaluation system will allow tourism service providers to reduce unnecessary costs, improve their efficiencies, and better understand the inherent needs of their tourists, so that they can remain innovative and competitive on the market by providing more convenient, personalized, and meaningful products and services to their customers [4].Therefore, how to evaluate the performance of m-commerce tourism platforms so as to enhance tourists' satisfaction has recently become the focus of academic and business communities of the tourism industry.
Prior studies have proposed some metrics to measure the efficacy of online traveling services (e.g., [5][6][7]).Nevertheless, very few studies have developed a comprehensive evaluation system that can be effectively used to quantitatively evaluate the performance of existing m-commerce tourism platforms.Our research attempts to bridge this gap by making the following contribution to the literature.First, the factors impacting the service quality are identified from related literature, and then used to construct the framework to evaluate m-commerce tourism platforms.Second, a survey is conducted with our purposely designed questionnaire to test the reliability of the proposed evaluation framework.Finally, our framework is applied to evaluate some most popular mobile tourism platforms in China.
The rest of the paper proceeds as follows.Next section reviews prior literature related to our research.Section 3 presents our evaluation framework.Section 4 demonstrates an application of our model with actual examples.The last section concludes the entire paper with insights.

Prior Literature
This section reviews prior literature with a focus on the use and efficacy of m-commerce systems in the tourism industry as well as the factors that influence online tourism services.In addition, the emphasis and contribution of our study are also highlighted in this section.
The growing prevalence of smart phones, tables, and other types of mobile devices has enabled them to be increasingly used in tourism, requiring the platforms or systems to be designed with a user-centered approach [8].Recent studies have further explored the issues and performance of mobile tourism systems and services from users' perspectives.For instance, Wang and Liao [9] assess the effective design of an m-commerce system through conceptualizing and measuring m-commerce user satisfaction construct.Kenteris, Gavalas, and Economou [10] empirically evaluate the user experience of their proposed mobile tourism prototype.Based on a case study of online ticketing services, Mallat et al. [11] suggest evaluating the needs derived from a user's context in order to assess the benefits of mobile systems.Using a factor analysis approach, Goh et al. [12] identify important types of mobile services from tourists' perspectives including transportation, accommodation, and food.Douglas and Lubbe [13] verify mobile devices as useful tools for booking services and indicate the satisfaction level of visitors' experience with their mobile applications.However, prior studies have not explicitly constructed any evaluation for mobile tourism platforms based on user experience.
User experience is defined as "a person's perceptions and responses that result from the use and/or anticipated use of a product, system or service" by International Standardization Organization [14].Recent studies have constructed and investigated user experience in different contexts.For instance, Park et al. [15] classify user experience with mobile phones into three categories (present, brand, and product/service experience) and identify specific elements in each category by using survey, interview, and observation methods.Pu et al. [16] evaluate the perceived quality of recommendations from a recommendation system by using their proposed evaluation framework consisting of four basic constructs: user perceived qualities, user beliefs, user attributes, and behavioral intensions.Xiong et al. [17] construct an evaluation framework based on user experience in the future 5G systems from a technical perspective.Analyzing the results from interviews and workshops, Vermeeren et al. [18] identify the needs for user experience evaluation methods such as those for early phases of development, for social and collaborative user experience evaluation, and for practicability.Nevertheless, very few prior studies have incorporated user experience in the context of mobile tourism and its platforms.
In order to identify and synthesize the elements in our evaluation framework based on user experience and apply the framework in mobile tourism platforms, we further review prior research that has studied the important factors impacting the performance of online services in a broad tourism context.Kaynama and Black [19] develop seven dimensions of online travel agency service quality: content, access, navigation, design, response, background information, and personalized.Zeithaml, Parasuraman, and Malhotra [20] categorize website features into reliability, access, response, effectiveness, easy navigation, flexible, trust, security, price, website design, and personalization, and explore the indicators of e-commerce services including reliability, accessibility, responsiveness, effectiveness, flexibility, price, trust, beauty, security, and personalization.Extending their model, Parasuraman, Zeithaml, and Malhotra [21] constructs a 22-item scale in four dimensions: efficiency, fulfillment, system availability, and privacy, and establishes a second scale that contains 11 items in three dimensions: responsiveness, compensation, and contact.Kim, Kim, and Lennon [22] evaluate online traveling websites with the following nine indicators: security, ease of use, low cost, website design and appearance, speed and useful information, booking service ability, pre-booking flexibility and classification.Based on fuzzy theory, Hu [23] evaluates service quality by using dimensioned criteria such as effectiveness, availability, compensatory, reactivity, integrity, contact, security, benefit, and personalized service.Kim and Lee [24] find that online travel agencies and suppliers share similar commonalities with regard to information content, reputation and security, structure and ease of use, and usefulness.Ho and Lee [25] investigate online tourism by grouping e-service quality constructs into five core components: information quality, security, website functionality, customer relationships, and responsiveness.Ghose and Han [26] investigate users' behavior on mobile devices and identify some influential factors to users' mobile Internet usage, such as social network, extend of geographical mobility, and user mobility.Bernardo, Marimon, and del Mar Alonso-Almeida [27] confirm that both functional and hedonic quality are two important dimensions significant influencing the perceived value with respect to the performance of e-services in online traveling agencies.
In summary, most of the prior research on m-commerce for the tourism industry is restricted to the development of technical models and prototypes.Although some studies attempt to use quantitative methods to construct system models in the tourism industry, very few of them have applied quantitative methods to conduct comprehensive analysis.Furthermore, most of the prior research related to user experience is based on website design, recommendation systems, and technology products; the effects of the tourists' experience have not been formally incorporated into mobile travel services.Our study addresses this gap by formally proposing an evaluation framework based on tourists' experience and using the framework to empirically evaluate two most popular m-commerce tourism platforms in China.

Evaluation Framework
Identifying appropriate evaluation indices is essential for constructing the evaluation framework.Selecting and incorporating different evaluation indices in the framework will have different influences on its accuracy and practicability.Although there lacks a common standard for choosing the evaluation indices for m-commerce tourism application platforms, prior studies show that they all follow some similar principles.Following upon these principles, we collect user experience-based influential factors of m-commerce and online travel services used by many researchers, extract online travel service quality influence indices according to the empirical factors, and then continue to summarize these collected indicators for evaluating m-commerce platforms and websites.We summarize the specific procedure as follows.

Selecting Preliminary Evaluation Indices
Based on prior literature, we categorize all the relevant experience-based factors into the following five preliminary first-level indices: user interface experience, product content experience, software security experience, service quality experience, and personalization experience.The second-level indicators are then listed in each category accordingly.
(1) The user interface experience describes how visitors feel when they browse a mobile application platform (e.g., [28,29]).A visitor's first good impression to the mobile application can improve the visitor's stickiness to the application.The preliminary second level indicators include six indices: interface layout, interface navigation, interaction, APP loading/login time cost, efficiency of operations, smooth guidance of the purchase process, and evaluation feedback.
(2) The product content experience can directly influence a tourist's decision to purchase products and services (e.g., [30][31][32]).Good contents can improve customer loyalty to m-commerce platforms.Many important functions are offered by various m-commerce tourism platforms.For instance, visitors can use the query searching function from mobile service providers to search for the information about tourism products and services, and then continue to booking and payment.They can also share their experiences of offline consumptions after their purchases with other tourists in the community of the m-commerce platforms.All of these behaviors are based on product contents.Therefore, tourists' experience and product contents are closely related.Here, we choose the following seven aspects as the second level indices: product price, product timeliness, product coverage, product content authenticity, product diversity, product booking availability, and membership rebate.
(3) The experience of software security is a crucial concern to users regardless of the PC or mobile terminals they use (e.g., [33,34]).Tourists' willingness to fulfill their m-commerce tourism transactions are contingent on the software security affiliated with the m-commerce platforms as their bank accounts and other personal information must be under good protection.Therefore, the security issues are fundamental to mobile e-commerce operators before they can provide other services.We choose the following three second-level indices for software-security experience: the security and convenience of payment, the authenticity of transaction, and the confidentiality of data information.
(4) A good service-quality experience can improve transaction rate, attract potential offline users, and promote customer loyalty (e.g., [35][36][37]).One of the important reasons to attract visitors to download mobile software applications and further to purchase tourism products is an m-commerce provider's popularity and reputation.Tourists' good offline consumer experience will further contribute to the provider's reputation, which is the best way to further publicize its products and services with the word-of-mouth effect.The second level indices we choose for service-quality experience are follows: visibility and reputation, service friendliness, offline service quality, emergency remedial capacity, advisory hotline, and complaint channel.
(5) Personalization experience is referred as the needs and expectations for different individuals in terms of tourism products and services (e.g., [38][39][40]).Therefore, m-commerce tourism providers should take into account the differences among users' demands and preferences for products and services.In order to meet the needs of different tourists, they will have to continuously improve their mobile traveling service functions.We identify the second-level indices for personalization experience as personalized service, timeliness of information update, and users' expectations.

Determining the Index System
In order to make the identified indices more scientifically rigorous so they can be applied in generic situations, we design a questionnaire to survey and verify the indicators, and then use SPSS software to further analyze the data.

Questionnaire (1) Design of the Questionnaire
The questionnaire consists of two sections.The first section is the main part, including a five point Likert Scale, which is used to measure the importance of the evaluation indicators of the selected indices in the process of their experience.The second section is the basic personal information.It helps analyze the different education, income, occupation of different proportions of the population and their impact on the evaluation indices (See details of the questionnaire in Appendix A).
(2) Distribution of the Questionnaire The targeting group of our questionnaire includes the tourists who have used m-commerce tourism platforms to query information or book traveling products.In order to get the sufficient number of responses in a certain period of time, we adopt the format of e-questionnaire by using the specific tool called "Questionnaire Star".Unlike traditional online questionnaires that can be easily distributed but are not effective, "Questionnaire Star" can improve the effectiveness of questionnaires by inhibiting the repetition of the same IP addresses and sources of information.
In order to obtain effective responses to the questions in the questionnaire, we piloted the survey in a small scale.After adjusting some of the choices based on the results, we then distributed the survey through QQ, WeChat, and some other popular social media apps in China to ensure that the questionnaire can be widely disseminated.
(3) Collecting Questionnaire Results The survey was distributed through "Questionnaire Star" for five days between 7 December 2015 and 15 December 2015 with a total of 310 responses.After discarding those responses with a completion time less than one minute and repeated IP addresses, we finally obtained 184 valid questionnaire responses.Descriptive statistics of the effective responses is summarized in Appendix F.

Reliability Analysis
We use the SPSS20.0 to test the reliability of the questionnaire based on the 184 valid responses.The statistical results show that the Cronbach's Alpha values of the questionnaire are almost all greater than 0.8, inferring that the questionnaire is highly reliable.See Table 1.

Exploratory Factor Analysis
Applying the exploratory factor analysis method, we analyze the 24 indices in the questionnaire to investigate the effect of m-commerce tourism platforms on visitors' experience.In our analysis, we use the principal component analysis approach to extract five immobilization factors and then use the maximum variance method to rotate the factors' load matrix.
(1) Descriptive statistics We summarize the details of the descriptive statistics in Appendix G.
(2) KMO and Bartlett testing Table 2 shows the results of the KMO and Bartlett testing, in which KMO value is 0.955, Bartlett's test of sphericity approximate Chi-Square value is 3871, Degree of freedom is 276, and Significance is 0.000.The significant probability is less than 0.001, indicating that there is a correlation among the variables, so they are suitable for factor analysis.(3) Explanation of factor analysis Table A1 (shown in Appendix B) displays the total-variance of the extracted factors to the original variables.The first factor contributes 28.223%, second factor 13.192%, third factors 13.163%, fourth factors 11.928%, and fifth factors 9.669% to the original variables.The cumulative variance contribution rate of the five factors is 76.175%.From the sixth factor to the last one, its characteristic value becomes smaller, which means its contribution rate to the original variance is less important.Therefore, the extraction of these five factors is sufficient for factor analysis.
(4) Factors' load matrix Table A2 (in Appendix C) shows the load on each of the five factors in the factors' load matrix.Before rotation, although there exists orthogonality between the factors, it is still difficult to explain them.After rotation, the load matrix structure can be simplified, making it easier to explain the practical significance of the common factors.

Reconstruction of m-Commerce Tourism Evaluation Framework
Five principal component factors (first level indices) and their influencing factors (second level indices) can be obtained from the rotated component matrix (in Table A2 of Appendix C).For example, the influencing factors of the first principal component factor include those from emergency recovery capability (A20) to payment security and convenience (A14).Although the results are a little bit different between the expected and evaluation indicators, the overall indicators are able to evaluate m-commerce platforms in a good extent.After adjusting the second level indicators, we obtain the final evaluation framework in Table A3 (see Appendix D).
(1) The first first-level indicator interprets product or service quality experience which includes the following 11 secondary indices: emergency recovery capability, transaction authenticity, data privacy, consultation hotline, visibility and credibility, complaining methods, product content authenticity, service friendliness, product reservation possibility, product price, payment safety and convenience.These indicators are related to the product and service quality for mobile e-commerce platforms, as well as their security issues.These are the primary factors affecting the application software. ( The second first-level indicator explains product operation experience by including these four secondary indices: product timeliness, product diversity, product coverage, and membership rebate.These indices evaluate the effective factors that can attract tourists to purchase and improve customer loyalty. (3) The third first-level indicator deals with personalization experience that includes the following three secondary indices: personalization service, timeliness of upgrade/update, and user preferences and expectations.These indicators reflect the needs and expectations of providing different service information for different users.
(4) The fourth first-level indicator focuses on infrastructure experience with three secondary indices: APP load/login time, evaluation feedback, and convenience of processing operations.These indicators assess the quality of mobile traveling e-commerce application software, not that for products and services.
(5) The fifth first-level indicator describes interactive experience by incorporating three secondary indices: interface layout, interface navigation, humanized interaction.These indices can be utilized by users to develop a self perception for mobile application software.Good interactive design can enhance the browsing and reading experience, highlighting the characteristics of a brand and its public image.

Application of the Evaluation Framework
Having established the formal evaluation framework, we next apply this framework to investigate some of the most popular m-commerce platforms so as to demonstrate the applicability of our proposed evaluation framework and further test its robustness.

Selection of M-Commerce Platforms
According to the 184 effective responses to our questionnaire, the most popular tourism M-Commerce platforms are Ctrip and Qunaer in China (See Table A4 in Appendix E).They account for 38.0% and 38.6% of the total, respectively, followed by Tongcheng 9.8%, Tuniu 4.9%, Mafengwo 1.1%, lvmama 0.5%, and other 7.1%.Therefore, we select Qunaer and Ctrip as our empirical research target because of their popularity.Using our proposed evaluation framework and Fuzzy Comprehensive Evaluation method [41,42], we next evaluate these two tourism m-commerce platforms.

Application of Fuzzy Comprehensive Evaluation Model
Because it is not easy to accurately quantify each evaluation index in our framework, the instrument of fuzzy mathematics can be applied to the evaluation framework.Specifically, we use the fuzzy comprehensive evaluation method to test the second level indices with a bottom-up evaluation process.Synthesizing the single factor evaluation matrix and the weight vector on each layer, we then conclude the testing results.

Determining the Weight Set (1) Weight set determination of the first level indices
We use contribution rate as the weight for the five main factors extracted by principal components analysis method.If the contribution ratio for factor u i is a i , the weight of a i is The weight of each main factor is obtained accordingly and displayed in Table 3.Therefore, the first level index weight vector is A = (0.370 0.173 0.173 0.157 0.127), which shows that the product and service quality experience is the most important factor for m-commence tourism platforms, followed by the personalization experience and product operations experience both as the second most important factors.The third most important factor is the infrastructure experience and the least important is the interface interaction experience.
(2) Weight sets determination of the second level index The weight set of the second level indices is determined according to the statistical output of the communalities.(See Table 4).The communality of each second level index represents its contribution rate, which reflects the importance of each second level index in the first level index it belongs to.We consider the communality as the weight and then use Equation ( 1) to calculate values.In particular, we fix the extracted factor as one and then normalize the extracted value to obtain the weight.We use U to denote the tourism m-commerce platform overall service quality: where U 1 represents product and service quality experience, U 2 personalization experience, U 3 product operations experience, U 4 infrastructure experience, and U 5 interactive experience.
(2) Construction of the second level factor sets.
We first construct the second level factors as follows for each first level factor.

Determining Comment Sets
The fuzzy evaluation of tourism m-commerce platforms is a collection of different tourists' satisfaction levels to a specific platform.Based on the evaluation results given by tourists, we set up five levels of fuzzy evaluations as where v 1 is very unsatisfied, v 2 unsatisfied, v 3 normal, v 4 satisfied, and v 5 very satisfied.

Determining Judgment Matrix
We select the first 40 responses as samples to the questionnaires of Ctrip and Qunaer to calculate the rating score with Equation ( 8), and then divide the scores by 40 to get the membership grade influencing factors.Finally, we obtain the evaluation matrix based on the selected second-level indices.
(1) Ctrip's evaluation matrix For Ctrip's membership statistics, see Appendix H.According to the evaluation index system and membership statistics, we derive the evaluation matrix for Ctrip as:  (2) Qunaer's judgment matrix Qunaer's membership statistics can be seen in Appendix I. Based on the evaluation index system and membership statistics, we get the evaluation matrix of Qunaer as:  Based on the individual factor of the second level indices, we calculate the comprehensive evaluation value.For instance, the fuzzy comprehensive evaluation set for product and service quality experience can be obtained as follow: = (0.100 0.099 0.101 0.096 0.088 0.088 0.091 0.090 0.079 0.090 0.078) = (0.040 0.036 0.076 0.270 0.570) Similarly, we can get the other four evaluation sets: B 1 2 = A 2 * R 1 2 = (0.017 0.059 0.208 0.367 0.357), B 1 3 = A 3 * R 1 3 = (0.037 0.050 0.211 0.338 0.369), B 1 4 = A 4 * R 1 4 = (0.068 0.026 0.103 0.325 0.479), and B 1 5 = A 5 * R 1 5 = (0.059 0.009 0.123 0.434 0.376).According to the maximum membership grade principle, in the five Ctrip's first-level indices, the product and service quality experience and the infrastructure experience are "v 5 " (very satisfied), and the personalization experience, product operations experience, and the interactive experience are "v 4 " (satisfied).
(2) Qunaer Qunaer's second level fuzzy comprehensive evaluation single factor matrix R 2 is Its second-level fuzzy comprehensive evaluation set is According to the maximum membership grade principle, Qunaer's second level indexes are also "v 5 " (very satisfied).

Fuzzy Comprehensive Evaluation Score
Finally, we normalize the vector of the evaluation matrix by setting different values for v according to five levels respectively, i.e., "v 1 " = 1, "v 2 " = 2, "v 3 " = 3, "v 4 " = 4, and "v 5 " = 5.Therefore, obtaining and using the score vector S = (1 2 3 4 5), we multiple it to the fuzzy comprehensive evaluation matrix and get the final score.

Analysis of Results
Summarizing the results derived from the fuzzy vectors of Ctrip's and Qunaer's m-commerce platforms, Table 5 demonstrates that both Ctrip and Qunaer perform well in terms of product and service quality experience as they both get a high score.Ctrip is better than Qunaer in the aspect of personalization experience, infrastructure experience, and interactive experience.The overall score can be seen as a fuzzy measurement of a platform's performance in general.Ctrip scores 4.125, higher than Qunaer's score (3.990), but the difference is quite small.Ctrip Travel, the most authoritative tourism m-commerce company in China, has an excellent reputation, which is why it can continuously attract tourists and increase customer loyalty.Originated from the early development of mobile terminals, Qunaer Travel started to compete in the tourism market later than Ctrip.However, by fully exploiting the opportunities in the m-commerce market, Qunaer Travel has quickly caught up and diminished its distance with the traditional online enterprises represented by Ctrip Travel.
All membership degrees of the first-level and second-level indices are better than "normal".Since Ctrip Travel and Qunaer Travel are the leading enterprises in China's online travel market, our results show that the consumers in this market are overall satisfied.When China's tourism m-commerce progresses toward its maturity, we will continue to observe the improvement with respect to the quality of tourism products and services to meet the diverse needs of tourists.

Conclusions
Prior research on user experience has mostly focused on website design, recommendation systems, and technology products; the effects of the tourists' experience have not been formally incorporated into mobile travel services.This research makes contribution to the literature by presenting and studying a tourism m-commerce platform evaluation framework.In particular, based on prior literature and relevant theories, we identify 24 preliminary evaluation indices.Using online questionnaires and exploratory factor analysis method, we extract from the 24 preliminary evaluation indices five experience-based principal components, including interactive, infrastructure, personalization, product and service quality, and product operations experience.In addition, we apply the confirmatory factor analysis method to test the robustness of the proposed evaluation framework.Our test result shows that the evaluation framework is both robust and effective.Finally, we empirically evaluate the m-commerce platforms of Ctrip and Qunaer by using our proposed evaluation framework in combination with the fuzzy comprehensive evaluation method.The insights derived from our study, however, are only our initial attempt to understand the factors influencing the performance of tourism m-commerce platforms.Future research may overcome some of the limitations to further extend and improve our evaluation framework.For instance, most of the respondents to our questionnaire were college students, which might result in the partiality of the survey results and our analysis.In addition, we may need to further refine the process of identifying and selecting the preliminary factors to make our evaluation framework more comprehensive.

3.
What are the tourism products that you have purchased through mobile commerce platforms?(choose one or more) Airfare ( ) Hotel ( ) Trip ( ) Resort Ticket ( ) Others__________ 4.
Please evaluate the importance of the following factors based on your traveling experience.

Table 3 .
First level factors and their weights.

Table 4 .
Second level factors and its weight.

Table 5 .
Comparison of the evaluation results.

Table A1 .
Contribution to total variance.

Table A3 .
Evaluation framework with indices.