A Combined Method of r-NPS and t-NPS Evaluations for Identification of Negative Triggers of Detractors’ Experience

Abstract

User experience has garnered increasing interest from researchers in the field of sustainable human–computer interaction (HCI) design. One of the key pillars to building sustainable user experience for digital applications is to be able to identify powerful triggers of detracting behavior and especially those of radical detraction. In the past, traditional methods of user experience analysis have often led to the assumption that pragmatic aspects constitute the priority issues for user experience improvement given the prevalence of these aspects in user feedback surveys. However, our econometric analysis based on net promoter score (NPS) survey results defeats this assumption and reveals that the most powerful degraders of the detractors’ experience are in fact the emotive aspects of experience such as transparency of transactions and customer service interaction. Based on our analysis, we arrive at several insights regarding the building of a sustainable HCI strategy. First, hedonic aspects of user experience are the most significant determinants of the degree of user detraction among the detractors. Second, membership cannot be taken for granted as a token of customer loyalty. Building on the theoretical framework of Hassenzahl, Haines-Gadd, and others, we generate new evidence for the importance of servicing hedonic aspects of user experience for digital applications businesses to form a sustainable customer relation and product strategy.

1. Introduction

Sustainability in the technology industry has gained increasing attention from the media in recent years. Awareness of unsustainable disposal and waste management patterns of electronic devices is growing among businesses, and management methods such as life cycle assessment and triple bottom line cost-benefit analysis have seen growing adoption by corporate sustainability professionals to help ICT businesses improve the overall efficiency and sustainability of their operations [1,2]. However, the sustainability issues of digital products such as digital software applications have in large been left out of this conversation. In fact, digital products have one of the shortest lifecycles among all consumer products. A 2014 study has found that mobile apps in the iTunes app store had an average app lifecycle of 54.84 days [3], and the average app categorical lifecycle length across different app categories was 50.91 days. User abandonment is a major cause of the brevity of app lifecycles. A 2016 report from analytics firm Localytics found that 23% of users use a newly downloaded app only once before abandoning it [4]. The high rate of user abandonment contributes to an unsustainable pattern of investment and consumption in the app economy.

While the majority of authors on sustainability in business focus on the physical environmental externalities of production and usage, our approach to sustainability focus on the long-term relationship between the customer, the product, and the provider. Providing a long-term service to users and cultivating a long-term bond between the product and the user are key indicators of digital sustainability as well as success factors for digital products, which are often overlooked by product managers. The subject of our studies, therefore, has been to explore the enabling factors for facilitating such a relationship, and to identify mitigation strategies for its deterioration. In other words, we are interested in what makes or breaks a sustainable user experience, which is the product of a good long-term customer–product–provider relationship. Previously, we applied the user experience (UX) curve method [5,6] and the AttrakDiff questionnaire [5,7] to identify the most critical user experience issues with a fitness app. Our results reaffirmed our hypothesis that both long-term and first-time user experience are pivotal to the enduring success of the app. In another study, we used user experience data to make business process restructure recommendations to developers of a music streaming app [8].

Apart from the UX evaluation methodologies adopted in our previous studies, there are a few popular UX metrics in the literature, including the usability metric for user experience (UMUX), system usability scale (SUS), American customer satisfaction index (ACSI), Forrester customer experience index (CX Index), and net promoter score (NPS) [9,10,11,12,13]. In this study, we attempt to apply NPS methodologies to understand the critical triggers of users’ negative experiences. Specifically, we combined relationship-NPS (r-NPS) and transactional (t-NPS) methods to study short-term and long-term user experience of our product in focus—an online streaming services platform in both browser, iOS and Android formats. NPS surveys obtain a “would recommend” score from respondents on a scale from one to ten. We focused on the “detractors”, namely, those who gave a recommendation score between zero to six in our NPS surveys. We sought to unveil the principal drivers of negative experience of detractors, using three waves of transactional and relationship-NPS data collected from calibrated selections of users to represent the user population of our targeted product.

The identification of such drivers of detractors’ user experience requires the application of rigorous empirical methods if an argument linking these drivers and sustainable user experience is to be formulated. Prior to this study, following a “what-do-detractors-mostly-say” approach, it appeared that pragmatic and technical issues are the prevailing elements which trouble detractors in their experience with the product. This is because from the relationship-NPS data we collected in the past waves of NPS surveys, the top three most mentioned aspects that users would like to see improved are the diversity of content, smoothness of playback, and content update frequency. For example, over half of all detractors (53.5%) reported “diversity of content” as one of the aspects that they would like to have improved, and 42.3% out of all detractors reported “smoothness of playback” as one of the aspects needing improvement. Given the prominence of these aspects in user feedback, developers have been working with the assumption that they constitute the priority issues for user experience improvement.

However, this assumption is unsafe until the following hypothesis is tested—that the size of a user experience issue, that is the prevalence its reports among the detractors community, is proportional to its significance in determining detractors’ user experience. This forms the single most important research question for the confirmatory part of our study, where we attempt to use econometric methods to establish the exact significance that each aspect of experience has in determining the degree of user detraction. For example, does the prevalence of “diversity of content” warrant that the lack of content diversity is the most powerful undermining factor of detractors’ user experience? Or, since “ease of customer service interaction” is the least reported issue (13.5%) among detractors, can we thus assume that customer service interactions play an insignificant role in shaping detractors’ user experience? Our further analysis, in fact, defeats these assumptions and reveals that the most powerful degraders of the detractors’ experience are in fact the hedonic aspects of experience. In addition, in the exploratory part of our study, we conduct further descriptive analysis on the relationship between various usage features and user experience.

This essay takes on the following structure: The next section will discuss the conceptual and methodological literature referenced and further developed in our study; the third section will unpack the methods employed and main hypotheses tackled in our research; the fourth section will present the results of data analysis and the respective recommendations to the product developers.

2. Literature Review

Sustainability in digital development is a nascent field. An emerging framework capitalizing on user-centered design in achieving digital sustainability is the Principles for Digital Development [14], a system of nine guidelines established by a community of international organizations including UNICEF and the World Bank in 2014. Now endorsed by 56 organizations and firms, the nine principles were originally conceived to promote better practices of digital development to serve technology-enabled development programs. The first principle, “Design with the User”, asks developers to get to know the users through “conversation, observation and co-creation” [11]. The fifth principle “Build for Sustainability” is about building sustainable programs, platforms, and digital tools maintaining user support as well as maximizing long-term impact. Across these principles, a user-centric approach is accentuated, where an emphasis is given to the mobilization of the users’ voice and support to maximize the long term value-added of technology.

These principles would best serve as guidelines for sustainable digital development when complemented with robust methods which enable developers to evaluate how effectively their products are meeting these guidelines. Only when businesses fully understand the mechanics of how various experiential factors trigger users’ negative experience, could they identify the gap between user needs and their current experience, and therefore strive to minimize user abandonment and increase the longevity and long-term utility of products.

The research gap that we are addressing is the identification of impediments to achieving sustainable human–computer interaction (HCI) with the aid of empirical methods for user experience data analysis. In the past, research on sustainable HCI has been predominantly following a materials-focused approach, such as Huang and Truong in their study of sustainable interaction design for mobile phones [15] and Pan et al. in their study of fashion in sustainable HCI [16]. Our study complements this area of research with a UX-focused approach, under a theoretical framework informed by existing literature on sustainable design principles and with the aid of econometric methods for quantitative data analysis.

The net promoter score (NPS) method is at the heart of our study, acting as an instrument for quantifying user experience. The core of our research is the measurement of user experience by the NPS score, complemented by a follow-up questionnaire breaking down the overall experience to multiple dimensions of experience in order to gain a view of those dimensions that trigger the maximal intensity of negative user experience. The NPS methodology was first proposed by Frederick F. Reichheld in 2003 [13]. Reichheld views NPS as the most effective metric to determine customer loyalty, considering its advantages over alternative metrics, such as customer retention or repeated purchases [13]. Reichheld maintained that while a perfectly satisfied and loyal customer may not necessarily repeat their purchase simply due to a decline in need, and vice versa, a repeated purchase may arise purely out of the existence of exit barriers or lack of alternatives, such a customer’s experience would however be captured in the “would recommend” score they give to the product [13]. Therefore, the NPS methodology is viewed to offer a more reliable view of the true state of customer satisfaction. In employing the NPS methodology as the base tool for quantification of user experience for further analysis, this study also contributes to the body of literature which validate the viability of econometric methods such as the ordinary least squares and ordinal logistic regression methods for studying sustainable user experience management via NPS data, such as Chang and Fan; Lartey, Hargiss, and Howard; Jeske, Callanan, and Gui; and Wilberforce et al. [17,18,19,20].

The significance of the role of detractors in relation to long-term product success has been well documented in the NPS literature. Detractors are a “long-term priority” because they spread negative word-of-mouth [21] and are prone to product abandonment if no response or improvement from the firm’s end was initiated [22]. Strategies to counter detraction that have been proposed in the literature include individual-level response to each detractor [23] as well as holistic identification of the cause of detraction [21,24].

However, few researchers have applied a user experience framework to systematically analyze the drivers of detraction. Our approach contributes to closing this gap by borrowing from several innovative user experience frameworks streams of literature in the field of human-centered design (HCD) to buttress our analysis. The first of these is the emotional durability design (EDD) framework developed by Haines-Gadd et al. (2018) to explore the factors which contribute to product longevity [21]. The EDD framework encompasses 38 strategies grouped around nine themes to guide a design approach which maximizes product longevity: 1) relationships; 2) narratives; 3) identity; 4) imagination; 5) conversations; 6) consciousness; 7) integrity; 8) materiality; 9) evolvability. With these themes, Haines-Gadd developed an innovative scheme allowing developers to consider the channels through which they could make their product more “emotionally durable”, namely to have a stronger bond with the user as the volume of experience intensifies. In doing so, the authors also build upon the body of literature on product attachment, to the attainment of which they summarized the following strategies that have been proposed in the literature: memories; pleasure; enjoyment; self-expression; usability; sensory design; superior appearance; utility and reliability; product personality; group affiliation.

Among these strategies, we included the dimensions utility, reliability, usability, and superior appearance in our follow-up questionnaire to the NPS survey, which aims to determine the aspects which most intensely undermine user experience. In our follow-up questionnaire, we asked the detractors to select the aspects of their experience that they would most like to see improved. Two of these options relate to utility and reliability, one relates to the usability of product, and one to the superiority of appearance. Moreover, we included identity and integrity, respectively the third and seventh theme in the EDD framework, in our enquiry. In the EDD framework, the identity theme is about “allowing for self-expression through customization and personalization”, which involves the inclusion of the user’s own identity in the experience of the product [25]. We take this theme into account by including one question on the quality of customized video recommendations in our follow-up questionnaire. The integrity theme encompasses the transparency and openness of production and business processes to the users, such as to “avoid cheap tricks”. We account for this theme in our inclusion of the transparency of transactions as one dimension of experience in our follow-up questionnaire to the detractors.

Our design of the detractors’ questionnaire also broadly aligned with Hassenzahl’s binary categorization of experience. Hassenzahl (2007) distinguished between the two dimensions of experience in his pragmatic/hedonic model: pragmatic experiences anchor those involving users’ functional needs, or “do-goals” such as making a phone call; hedonic experiences, on the other hand, relate to users’ emotive needs, or “be-goals” such as “being special” [25]. Hassenahl also proposed the hypothesis that a user’s perception of pragmatics and hedonics would shift along the passage of time. With repeated usage and learning, a user’s pragmatic experience would improve in time, while the initial experience of hedonics would decline due to disappointment at a later point. In our questionnaire distributed to the detractors, we included six dimensions of pragmatic experiences and five dimensions of hedonics as options when the user is prompted to select any number of dimensions which they identify as the aspects they would like to see improved the most. The “pragmatic” dimensions encompass the quality of content, playback, ease of use, etc., while the “hedonic” dimensions include aesthetics, content customization, process transparency, and relationship with customer service. The adoption of Hassenzahl’s binary framework would help us investigate the relations between these two species of experience and the characteristics of customers, their usage patterns, and their overall experience.

3. Data and Methods

The research method adopted in our study is retrospective non-experimental analysis using one-shot survey data collected from 3 waves of surveys in Q2 2019: (1) relationship-NPS surveys collected from a randomized selection of 5000 mature users (users who first started using the application more than a year prior to the distribution of the survey); (2) relationship-NPS surveys collected from a randomized selection of 500 mature users whose usage profiles were also captured; (3) transactional-NPS surveys collected from a randomized selection of users at the end of 3 types of transactions.

The survey was conducted online after IRB (Institutional Review Board) approval was granted. Participants were randomly selected among mature users of the application. A survey invitation containing the survey link was delivered to each of them via SMS, and those who have successfully answered all questions and submitted them were each awarded an airtime credit top-up in the amount of RMB ¥2 (circa USD $0.30).

3.1. First Wave of Relationship-NPS Survey Data

We collected our first-wave relationship-NPS data from a sample of 8522 users in 2019. Relationship-NPS, or r-NPS, measures customers’ status of satisfaction with its relationship with the product and its provider. It is distinguished from transactional-NPS in the sense that it measures user experience over a sustained period instead of over a discrete touchpoint [26].

We designed a two-step survey that starts with a likelihood to recommend question (Q1), worded as “On a scale of 0 to 10, how likely are you to recommend this platform to your friends and family? (with 0 standing for not likely at all, and 10 for very likely)”. The sampled users are subsequently divided into three groups, based on the NPS score they give in response to Q1. Those who give a score of either 9 or 10 are tagged as “promoters”, while those who either gave 7 or 8 as “passives”. The users who gave an NPS score that fell within 0 to 6 (including 0 and 6) are grouped as “detractors”, who are the focus of our study. Out of the 8522 observations we collected, 2859 fell into the “detractors” group. The second question that follows differs with respect to each user’s group tag. To the detractors, we asked them “Among the following aspects of experience, which one(s) would you most like to see improved?”(Q2) The passives were asked “Which aspect(s) of experience, if improved, would most likely make you recommend the product to your friends and family?”(Q3) The third variant, “Which aspect(s) do you mention when recommending the product to your friends and family?” (Q4) was posed to the promoters. For the purpose of this study, we are only going to make use of response data in the Q1 and Q2 sections which were collected from the detractors.

To obtain our samples, we first identified a pool of 1 million users through a randomized selection from a user population of over 12 million. Before obtaining the pool, we first applied the “mature user” qualification to exclude new or infrequent users who are not likely to have formed a relationship with the business. Users were qualified as a “mature user” under the condition that they had logged on to their account at least once and used the app for over 10 min in the preceding month. The survey was distributed to the selected pool of users via a mobile SMS text, which contained an NPS evaluation prompt and a weblink directing the user to the online survey. The attrition rate between the preliminary pool and the final sample is high: only 0.85% of users (8522 out of 1 million) who received the SMS text invite completed the online survey.

Table 1. First-wave questionnaire structure.

	Q1	Q2 **	Q3 **	Q4 **
Question wording	On a scale of 0 to 10, how likely are you to recommend this platform to your friends and family?	Among the following aspects of experience, which one(s) would you most like to see improved?	Which aspect(s) of experience, if improved, would most likely make you recommend the product to your friends and family?	Which aspect(s) do you mention when recommending the product to your friends and family?
Question is distributed to:
Promoters *	●			●
Passives *	●		●
Detractors *	●	●

* User group is determined by answer to Q1. ** Aspects are listed in Table 3.

and

Table 2. First-wave data summary.

Feature	Description
Sample size	8522
User distribution by NPS	51.1% Promoters
	15.3% Passives
	33.5% Detractors
Number of users by NPS	4356 Promoters
	1307 Passives
	2859 Detractors
NPS data summary, all users	Mean: 7.33
	Median: 9
	Mode: 10
NPS data summary, detractors	Mean: 3.27
	Median: 4
	Mode: 5

Abbreviations: NPS, net promoter score.

show the structure and data summary of first-wave questionnaire survey respectively.

There are 2 key threats to the external validity of this study: First, users self-select into the final sample, since they exercise discretion in choosing whether to take the survey when prompted. Incentive to nudge user towards participation was provided in the form of an RMB ¥2 (circa USD $0.30) airtime credit top-up. Given the meagerness of this incentive, we acknowledge a potential selection bias whereby the users who self-select into the sample are opinionated about the product in either a particularly positive or negative way, which incentivizes the voicing of such views. Secondly, since the study was exclusively done in China, we do not claim the results to be generalizable beyond this market.

Each user is expected to complete the survey in the privacy of their own devices, so users are not influenced by each other in their likelihood to recommend or in their choices of the aspects that they consider most urgently require improvement. Therefore, standard errors are expected to be independent across observations and no clustering problem should be present. In this first wave of the survey, we did not collect any behavioral or usage profile associated with each observation. The survey was fully anonymous and sampled users were not asked to self-identify at any stage. Therefore, they were free from any bias resulting from the Hawthorne effect [27].

Our hypothesis is that the detractor’s NPS score is determined by individual aspects of experiences which collectively trigger the behavior of “detracting”.

We model users’ NPS outcome by fitting the following OLS equation:

$$y_i={\beta }_0+{\beta }_1{X}_{1i}+{\beta }_2{X}_{2i}+{\beta }_3{X}_{3i}+{\beta }_4{X}_{4i}+{\beta }_5{X}_{5i}+{\beta }_6{X}_{6i}+{\beta }_7{X}_{7i}+{\beta }_8{X}_{8i}+{\beta }_9{X}_{9i}+{\beta }_{10}{X}_{10i}+{\beta }_{11}{X}_{11i}+{\beta }_{12}{X}_{12i}+{\epsilon }_i$$

(1)

where $y_i$ denotes each sampled user’s NPS score outcome and $X_k$ represents an aspect of experience. $X_k$ is constructed as a dummy variable which registers 1 when selected by the user, and 0 when not selected.

Table 3. Covariates to determine detractors’ NPS score.

Variable	Aspect of Experience	Types of Experience
$X_1$	Smoothness of playback	Pragmatic
$X_2$	Diversity of content	Pragmatic
$X_3$	App stability	Pragmatic
$X_4$	Content update frequency	Pragmatic
$X_5$	Practicality of app functions	Pragmatic
$X_6$	Aesthetic appeal of interface	Hedonic
$X_7$	Ease of use	Hedonic
$X_8$	Transparency of transactions	Hedonic
$X_9$	Member benefits	Pragmatic
$X_{10}$	Ease of customer service interaction	Hedonic
$X_{11}$	Personalized recommendations	Hedonic
$X_{12}$	Others, please specify	N/A

lists all variables of detractor’s NPS score and corresponding aspect of experience.

The benefit of this approach as opposed to descriptive statistics is that it enables us to assess the relation between each aspect of experience and the degree of intensity of the detractors’ dissatisfaction. It is important to note that the detractors are not a homogeneous community, as to detract is not a binary decision on the part of the detractor. Those who gave an NPS score of 6 have a qualitatively much different experience from those who gave a score of 0, and are more likely to convert to the passives camp. Instead of trying to find out what makes users detract, we are interested in what triggers intense negativity and differentiates such behavior from mild detraction. The aspects which are most frequently highlighted among all detractors as needing improvement are not necessarily identical with the ones which are the most powerful drivers of negativity. It is possible, for example, that the most prevalent flaw only in fact slightly undermines the likelihood to recommend, whilst a rarely picked-out flaw has a significant impact on the “would recommend” decision. To differentiate, quantify, and attribute the exact degree of influence of each aspect of experience on overall user experience requires us to use econometric methods to explore the relation between each aspect and the intensity of detraction.

Our 12th option is an open-ended question to account for any omitted aspects of experience which users find also in need of improvement. In addition to the structured part of the survey, this contingent option provides additional qualitative input for user research [28].

3.2. Second Wave of r-NPS Survey Data with Usage Profiles

The first wave of r-NPS survey was originally conducted to identify the user approval level of the application. When the survey was concluded, we noticed that the characteristics of each user was still unclear, especially for radical detractors (users who gave an NPS score between 0–3), which means that the data contributed very little to customized product experience improvement. In order to identify those users’ characteristics, we further conducted a second wave of r-NPS survey with prompts on usage profiles. The second wave questionnaires were distributed to the same standard group of active users in Q2 2019 through the same SMS messaging instrument, which resulted in a final effective sample size of 500. The questionnaires collection was terminated once 500 valid questionnaires were obtained, where we identified questionnaires that had incomplete answers as invalid. This sampling process was done in parallel to the first wave of r-NPS data collection, so the second-wave pool is not a subset of the sample pool obtained in the first wave. We applied the same “mature user” controls to disqualify new or unexperienced users from the preliminary pool.

For this wave of data, we incorporated additional metrics into the survey including user characteristics and usage profiles. Specifically, these included the user’s 1) gender; 2) age group; 3) province/municipality of origin; 4) membership status; 5) monthly active days; 6) mobile operating system—Android or iOS. The rest of the survey was identically structured as the first wave surveys, that is a single likelihood to recommend question followed by one of the three variants of customized questionnaires based on user group tags “promoters”, “passives”, and “detractors”.

The collection of the second-wave data enables us to test various hypotheses regarding the relations between the depth of usage and user experience. For instance, we are interested in exploring any potential gap between paying and non-paying users in their average experience, or differentiation between the most active users and relatively inactive users in their loyalty and the aspects of experience that they are most concerned about. These data would add explanatory power to our inferences drawn from the first-wave survey data where we establish the power of each aspect of experience in shaping long-term user experience.

3.3. Transactional-NPS Survey

Whereas the relationship-NPS surveys provide strategic decision support by signaling the overall user experience and level of customer loyalty, transactional-NPS data gives tactical support to such strategies by providing actionable insights on how to improve specific dimensions of service to ameliorate these aspects of experience [29]. This is because transactional-NPS data is always triggered over a specific touchpoint such as a service transaction or a usage scenario. Moreover, transactional-NPS data offers the possibility of learning about the dynamic relationship between customer experience and various touchpoints that constitute the overall experience over time. The shifts in overall experience, reflected relationship-NPS data, could be explained by the improvement of deterioration of service performance at certain touchpoints reflected in transactional-NPS data.

To collect transactional-NPS data, we designed a one-question likelihood to recommend survey prompt after these scenarios: 1) the purchase of the platform membership; 2) the end of a video download; 3) the end of interaction with a customer service representative. The survey was distributed to a randomized selection of 3% of all users in the above 3 scenarios in Q2 2019 through an in-app push prompt, and resulted in 827 effective samples for the membership purchase scenario; 735 for the video download scenario; and 689 for the customer service interaction scenario.

In our study, transactional-NPS survey data also enables the comparison of transient and long-term perceptions of a certain aspect of experience, since we both collect in-situ evaluation data over specific service touchpoints and retrospective experience evaluation data on these dimensions of service in the second section (Q2-detractors) of our relationship-NPS surveys. The role of memory in biasing the perception of user experience has been extensively addressed in the literature, such as Larson and Csikszentmihaly’s experience sampling method for collecting in-situ experiences to eliminate disturbance from memory effects, Burmester et al.’s Valence method that captures feelings during the exploration phase with an interactive product, and the MyExperience system designed by Froehlich et al. to collect real-time user feedback on their experience with mobile phones [30,31,32]. By comparing transactional-NPS data and retrospective evaluation of the respective experience touchpoints, we can deduce the difference in the structures of short- and long-term user experience and how the power of various aspects of experience are reinforced or diminished through the axis of time.

4. Analysis of Results

4.1. First-Wave Data

Comparing pragmatic and hedonic aspects of experience as drivers of detraction, we performed two regression operations of the overall NPS outcome on the 12 variables. The first was an ordinary least squares (OLS) regression directly using the NPS score collected from first-wave surveys as a continuous variable; the second was a logistic regression where we dichotomized detraction into “mild” (0) and “radical” (1), and regressed the detraction outcome, that is the likelihood of being a radical detractor, on the 12 variables. The rationale for employing the binary logistic method in addition to OLS regression is to find out triggers of identification on the radical end of detraction, represented by an NPS score of 0–3, which is more interesting than aspects which are significant, but only in so far as to trigger mild detraction. The results of the two operations are tabulated in

Table 4. Regression output on NPS outcomes.

Type of Experience	Variable	Aspect of Experience	Frequency of Appearance among All Responses	OLS Regression Coefficient on Continuous NPS Outcome (0–6)	Logistic Regression Coefficient on Dichotomous Detraction Outcome ****
Pragmatic	$X_1$	Smoothness of playback	1,081 (42.1%)	0.055 (−0.137, 0.246)	−0.030 (−0.205, 0.145)
Pragmatic	$X_2$	Diversity of content	1,373 (53.5%)	0.237 ** (0.045, 0.430)	−0.209 ** (−0.385, −0.034)
Pragmatic	$X_3$	App stability	690 (26.9%)	0.060 (−0.168, 0.287)	0.013 (−0.195, 0.221)
Pragmatic	$X_4$	Content update frequency	964 (37.6%)	0.292 ** (0.091, 0.493)	−0.172 * (−0.357, 0.012)
Pragmatic	$X_5$	Practicality of app functions	630 (24.6%)	−0.108 (−0.348, 0.131)	0.037 (−0.182, 0.255)
Hedonic	$X_6$	Aesthetic appeal of interface	448 (17.5%)	−0.383 *** (−0.651, −0.115)	0.279 ** (0.034, 0.523)
Hedonic	$X_7$	Ease of use	494 (19.3%)	0.100 (−0.163, 0.363)	−0.101 (−0.342, 0.140)
Hedonic	$X_8$	Transparency of transactions	744 (29.0%)	−0.357 *** (−0.573, −0.140)	0.182 * (−0.015, 0.380)
Pragmatic	$X_9$	Member benefits	587 (22.9%)	0.216 * (−0.018, 0.450)	−0.192 * (−0.408, 0.024)
Hedonic	$X_{10}$	Ease of customer service interaction	346 (13.5%)	−0.406 *** (−0.715, −0.097)	0.364 *** (0.083, 0.645)
Hedonic	$X_{11}$	Personalized recommendations	378 (14.7%)	0.092 (−0.188, 0.372)	−0.116 (−0.373, 0.141)
Customary	$X_{12}$	Others, please specify	443 (17.3%)	−1.084 *** (−1.324, −0.844)	0.885 *** (0.663, 1.107)

* Significant at 90% confidence level. ** Significant at 95% confidence level. *** Significant at 99% confidence level. **** NPS outcomes are dichotomized into mild detractors (0) who gave a score of 4, 5, or 6; and radical detractors (1) who gave a score of 0, 1, 2 or 3; (1 for radical detractors & 0 for mild detractors). Abbreviations: OLS, ordinary least.

.

Since the 12 variables are dimensions of dissatisfaction that should inversely affect individual user experience, we expect all variables to have a negative correlation with the overall NPS score outcome. However, there appears to be two statistically significant positive drivers: “Diversity of content” and “Content update frequency”. It seems that not only do dissatisfaction with these two aspects not add to the intensification of detraction, they appear to have a moderating effect on the level of user detraction than would otherwise be the case. This is a puzzling discovery, since these are the top two aspects most frequently cited by sampled users as needing improvement, respectively appearing in 53.5% and 37.6% of the first-wave surveys. On the other hand, “Aesthetic appeal of interface”, “Transparency of transactions”, and “Ease of customer service interaction” emerge as three prominent drivers of detraction exerting highly statistically significant downward pressure on the NPS outcome.

This discovery suggests the non-equivalence between the prevalence of a user experience issue and its impact on the overall experience among detractors. For example, although “Diversity of content” is the most frequently selected aspect of dissatisfaction, it does not in fact trigger the most intensely negative experience. This is validated by our observation of the variation in NPS outcomes when we stratify users into single-issue groups, which consist of users who picked out exactly only one aspect of experience as needing improvement at Q2. Since Q2 was designed as a multiple-selection survey, by using single-issue user data, we can isolate the influence on user experience from other aspects of experience which the user potentially also has issues with, making the NPS outcome entirely attributable to one issue only. This approach, though, is subject to a degree of bias given that the mean NPS scores obtained for the least-often singled-out aspects are based on a smaller sample size than the most popular ones. Figure 1 visualizes the mean NPS scores given by single-issue users distributed over the prevalence of issue.

Table 5. Quantitative and statistical significance on continuous NPS outcome.

	High Statistical Significance *	Low Statistical Significance
High quantitative significance **	Diversity of content (+) Content update frequency (+) Member benefits (+) Aesthetic appeal of interface (−) Transparency of transactions (−) Ease of customer interaction (−) Others, please specify (−)	Practicality of app functions Ease of Use
Low quantitative significance	None	Smoothness of playback App stability Personalized recommendations

* Significant at >90% confidence level. ** Coefficient >0.1 in absolute value.

summarizes the quantitative and statistical significance of the aspect variables driving continuous NPS outcome.

Over half of the sampled users selected “Diversity of content” as one of the issues they would like to have improved. However, the average NPS score given by users who solely selected “Diversity of content” as the aspect they would like to see improved was 3.55, which is in fact 0.21 higher than the mean NPS score of 3.34 across all detractors, and 0.67 higher than the mean score (2.89) among all single-issue detractors. In contrast, several hedonic dimensions of experience appear to be much more powerful underminers of the overall NPS outcome. For example, those who picked exclusively “Ease of customer service interaction” gave on average an NPS score of 1.85, indicating the considerable power that a negative customer service interaction has in undermining a user’s relationship with the product, although only 13.5% of detractors mentioned this aspect of experience in their survey responses, making it the least visible problem. Users who made only one selection singling out “Transparency of transactions” as the aspect they would like to see improved gave an average NPS of only 2.02, which is 1.32 lower than the mean NPS score of 3.34 across all detractors.

What may explain the significant positive relationships between the NPS outcome and the variables “Diversity of content”, “Content update frequency”, and “Member benefits”? We can hypothesize that users who expressed desire for improvement in these areas should have already developed a degree of loyalty, since all these aspects of experience are longitudinal in the sense that the flaws in these dimensions take time to be detected and improvement requires time to benefit consumers. For example, those who are concerned with content diversity and update frequency are likely to expect to use the product for an extended period in the future, which justifies the expectations for more diversity and new releases in the catalogue. Likewise, those who are dissatisfied with member benefits are likely to be either already a member, or considering becoming one, showing an inclination to keep using the product. It could be such loyalty that exerts a moderation or de-radicalizing effect on the NPS outcome, since the voicing of these dimensions of dissatisfaction implies an existing positive relationship with the product. However, the validity of the reasoning presented here would need to be supported by evidence from user and usage characteristics, which will be addressed in Section 4.2.

4.1.1. Comparing Aspects Driving Continuous NPS Outcome vs. Dichotomous Detraction Outcome

In addition to the ordinary least squares operation using directly the NPS scores as the outcome variable, we dichotomized negative experience into a binary characterization as either “mild” (who gave an NPS score of 4, 5, or 6) or “radical” (who gave 0, 1, 2, or 3) to help us further delineate the experiential profile of the most radical detractors. Through this operation, we are seeking to find the variables that triggers “radicalization” within the detractor cohort, that is the qualitative differentiation taking place specifically at the transition from the more neutral detractors to the radicals.

As the logistic regression output in Table 2 showed, we found that the most significant drivers of radicalization largely coincide with those of overall degree of detraction, that is the three hedonic aspects “Aesthetic appeal of interface”, “Transparency of transactions”, and “Ease of customer service interaction”. However, several observations are interesting:

First, the only highly significant (at 99% confidence level) driver of radicalization is “Ease of customer service interaction”. Both “Aesthetic appeal of interface” and “Transparency of transactions” have dropped in quantitative and statistical significance. This indicates that although the latter two are powerful underminers of detractors’ user experience, they are not as significant as customer service interaction in converting a user from a mild to a radical detractor.

Second, the pragmatic aspects of experience “Practicality of app functions” and “App stability” are substantially less quantitatively significant in influencing the dichotomized outcome than the continuous NPS variable. This shows that these aspects have little power in radicalizing users once they already identify as detractors.

4.1.2. The Most Powerful Drivers of Negative Experience

(1) Aesthetic Appeal of Interface

Among all detractors, 17.5% selected “Aesthetic appeal of interface” as one aspect of their experience which they would like to see improved. It is the second-most quantitatively significant variable, contributing a 27.9% likelihood to radical detraction. The power of visual design in shaping user’s overall experience could be partly explained by the nature of our product in focus, which is a content streaming platform that relies on visual channels of delivery.

On the other hand, the significant power that visual quality has in moving users between degrees of detraction confirms the hypothesis that hedonic features of digital products are increasingly important in determining the competitive edge and product differentiation that businesses can have competing in this market [33]. More interestingly, the aesthetics of products could be not only valued over functionality, but may subjectively influence users’ perception of functionality itself: researchers from the Hitachi Design Center found in their experiments with ATMs in 1995 that there was a high correlation between users’ ratings of aesthetic appeal and ease of use [34]. Good aesthetics could intrinsically motivate users to be more willing to learn during the earliest exposure and develop a more positive perception of usability [35]. In our sample, the correlation between the dummy variables “Aesthetic appeal of interface” and “Ease of use” is 38.2%, almost at the same level as that (40.8%) between the two functional variables “Ease of use” and “Practicality of app functions”.

(2) Transparency of Transactions

Over 1 in 4 detractors (29%) took issue with the “Transparency of transactions”, making it the most frequently selected option among the hedonic aspects of experience. We also note that dissatisfaction with the transparency aspect has the most robust negative relationship with the NPS outcome among all 11 specified aspects of experience, with the highest statistical significance at the 99.9% confidence level.

The transparency aspect ties into the several core tenets of the user’s relationship with the business, including trust, closeness, openness, and the feelings of having choice and control [36]. Under Haines-Gadd et al.’s emotionally durable design framework, it relates to the “integrity” theme, which is one of the 9 dimensions of design for product longevity. The transparency of pricing and charges, in particular, is a crucial determinant of trust. UX pioneer Jakob Nielsen has called-out price as “one of the most critical pieces of information a user has” for decision-making about a product [37]. Deceptions or withhold of information on the pricing of services, whether intentional or unintended, is an immediate trigger of distrust [38]. Therefore, establishing a transparent pricing mechanism and correcting for any unintended confusion with pricing and charges on the front-end is critical for minimizing detraction.

(3) Ease of Customer Service Interaction

Only 13.5% of detractors selected ease of customer service interaction when asked which aspects of experience they would like to have improved, which gives this aspect the appearance of a low-salience issue. However, it turned out to be the most quantitatively significant aspect among all dimensions of experience in determining the NPS outcome among detractors. The customer service aspect alone contributes a 36.4% likelihood to radical detraction.

The quality of customer service interaction is a critical shaper of negative experience firstly in the sense that the need for this interaction most likely arises in the context of existing experience of trouble or friction while using the product. Therefore, it is not surprising that customer service interaction emerged as a gatekeeper to radical detraction, since a negative interaction on top of existing negative experience would easily radicalize users.

Secondly, in the technology industry, customer service is one of the most visible constituents of what the product is in the customer’s mind. Castrounis [39] noted that customers in general are not interested, do not understand or are apprehensive of the “blackboxes” of digital products such as backend architecture or algorithms. Rather, the only aspects they consider to have on the product are user interface, any deliverables the product produces and customer service. Therefore, investment in customer service is anticipated with particularly high returns in terms of its prominent influence on radical detraction.

(4) Open-Ended Option

The open-end option is highly significant in determining both the continuous and dichotomous outcome, more so than any of the specified aspect of experience. The act of filling out the open-ended option itself drops the continuous NPS outcome by over 1 full point. It also contributes an 88.5% likelihood to the identification as a radical detractor. This confirms the value of this open-ended option as a pointer to the exact motivations behind radical detraction, which was our initial expectation when designing the survey. As a result, we expect further value for consumer insight generation from content analysis on the feedback collected in the open-ended option.

4.2. Second-Wave Data

Table 6. Participants’ features.

Feature	Distribution
NPS	52.2% Promoters
	19.8% Passives
	28.0% Detractors
Membership status	33.6% Member
	66.4% Non-member
New user status	9.0% New user *
	91.0% Old user
Gender	57.4% Male
	14.2% Female
	28.4% Undeclared
Province of origin	17.0% Jiangsu
	15.2% Guangdong
	6.2% Sichuan
	6.2% Shanghai
	52.4% Rest (Declared)
	3.0% Undeclared
Age group	5.8% Under 25 years old
	8.6% 26–30 years old
	8.4% 31–35 years old
	10.0% 36–40 years old
	10.6% 41–45 years old
	10.4% 46–50 years old
	7.4% 51–55 years old
	6.0% Over 55 years old
	32.8% Undeclared
Monthly active days	39.8% 0–10 days
	27.2% 11–20 days
	32.8% 21–32 days

* New users are identified as accounts created in the preceding month.

shows the features of 500 randomly selected users of second-wave survey.

4.2.1. Relationship between Intensity of Usage and Overall Experience

For both members and non-members, we observe an upward growth trajectory for r-NPS outcome along the increase of monthly active days. The result is depicted in Figure 2. This signals that either a) user experience progressively improves as the user engages in higher volume of usage, and/or b) users who choose to increase usage have a higher level of satisfaction with the overall experience. In either case, this indicates the creation of customer loyalty over time.

4.2.2. Relationship between Membership Status and Overall Experience

According to Hassenzahl’s binary model, the nature of an experience is defined with respect to the purpose that it serves [40]. Experiences that deliver functional goals are pragmatic, and those which result in an emotive achievement are hedonic. Membership could be either a hedonic or pragmatic aspect of experience, depending on whether it is programmed to provide additional functionality and/or pleasure in service. In our case, membership benefits of the product include access to additional titles, channels, and free data. Since all these benefits revolve around functional needs, “Member benefits” is recognized in our framework under the pragmatic group of experiences.

Figure 3 shows the distribution of NPS among members and non-members. One surprising finding which emerged when we modelled the distributions of NPS score outcomes with respect to membership status is that there is no clear differentiation between the distributions of NPS outcomes between members and non-members. In other words, users 1) are having vastly similar levels of experience be they members or non-members, and secondly 2) have not become more evangelist of the product after committing to the membership. A two-sample t-test was conducted to test the equivalence of the means of the member and non-member distributions. The result of this test is t(498) = 0.361, p > 0.05, which leads us to fail to reject the null hypothesis of mean equivalence from the two distributions. In addition, we ran a two-sample Kolmogorov-Smirnov test to test the equivalence of the shapes of the two distributions. We obtained a p-value of 1.000 for the combined two-tailed K-S test, on which basis we fail to reject the null hypothesis of the equality of the two distributions. In other words, no significant difference was found between the either the mean or the shape of the NPS distributions of members and non-members.

Membership, therefore, appears to be a purely utilitarian transaction in our case. Users’ decisions to commit to the membership is not triggered by a superior level of experience with the product, and also does not in itself trigger loyalty.

4.2.3. The Relationship between Intensity of Usage and Salience-Attribution to Aspects of Experience

We hypothesized in Section 4.1 that several aspects of experience have a de-radicalizing effect on r-NPS outcome. Specifically, there are “Diversity of content”, “member Benefits”, which feature respectively as the 2nd and 4th aspects in our second-wave survey Q2, and “Content update frequency”, which was not included in the second-wave questionnaire. This hypothesis gains some support from the evidence on the correlation that emerged between usage intensity and salience-attribution to certain aspects of experience modelled using second-wave data.

Several patterns appear after we distributed the detractors’ selections of the aspects of experience that they would like to see improved most across their respective monthly active days. In Figure 4 we observe the prevalence of blue dots around the 2nd and 7th aspects “Diversity of content” (61.3% blue) and “Member benefits” (75% blue), indicating a dominance of mild detractors around these issues, in contrast to some other aspects such as the 8th “Ease of customer service interaction” (67% red) and 10th “Others” (78% red), which are dominated by the red colors.

Table 7. Label and corresponding Aspect of experience.

Label	Aspect of Experience User Would Like to See Improved
1	Smoothness of playback
2	Diversity of content
3	Practicality of app functions
4	Aesthetic appeal of interface
5	Ease of use
6	Transparency of transactions
7	Member benefits
8	Ease of customer service interaction
9	Personalized recommendations
10	Others, please specify

details the corresponding aspects of experience in Figure 4.

However, there is little evidence for an existing high level of commitment with users voicing concerns with “Diversity of content”. If this community has a higher degree of loyalty serving as a de-radicalizer as we hypothesized, it could be more to do with expectations than current commitment. In Figure 5, we see a concentration of relatively inactive non-members at the first three pragmatic aspects. This means that the majority of users who expressed concerns with these aspects, including “Diversity of content”, are those with little depth of usage and little investment in the product.

At higher values of monthly active days, salience across the 10 aspects is more evenly distributed than at the lower ranges, where there is a gravitation towards the pragmatic aspects. This means that the more mature users are, the more likely they are attentive to hedonic aspects of experience, and form their NPS outcome decision based on these aspects. Relationship between usage intensity and salience-attribution is visualized in Figure 5.

Table 8. Label and corresponding Aspect of experience.

Label	Aspect of Experience User Would Like to See Improved
1	Smoothness of playback
2	Diversity of content
3	Practicality of app functions
4	Aesthetic appeal of interface
5	Ease of use
6	Transparency of transactions
7	Member benefits
8	Ease of customer service interaction
9	Personalized recommendations
10	Others, please specify

details the corresponding aspects of experience in Figure 5.

4.3. Transactional NPS Analysis

There are three main observations from the t-NPS results:

First, at each touchpoint, users exhibited vastly similar response behavior, with the interquartile range placed between 5 and 10, and the highest densities found at the scores of 10, 5, and 1. This same distribution is also found with the r-NPS survey data. Two of the three touchpoints exhibited the identical distribution as the r-NPS surveys, with the same interquartile range, density peaks, and the median found at the score of 9. The persistence of this structure, that is the lack of differentiation between the response distributions in answer to different prompts, implies the possibility that the research design itself biased the results towards a certain distribution. For example, the response prompted by the single likelihood-to-recommend question at a certain touchpoint may not necessarily represent user experience only at that touchpoint, but could also be diluted by the user’s prior experience at other touchpoints, thus biasing the t-NPS outcome towards the overall r-NPS score. To establish the extent of this bias, we need to employ alternative methods of data collection in order to determine if the t-NPS data collected at each touchpoint faithfully reflect customer experience at that point in time. Distributions of transactional NPS across 3 touchpoints is visualized in Figure 6.

Second, on average, users have a more negative experience at the membership transaction touchpoint than the other two. t-NPS outcomes at the membership transaction touchpoint has the lowest median score at 8, one point lower than the other two. It also has a flatter distribution than the other two touchpoints, showing a highest share of detractors (36.9%) among the sampled. This shows the relative deficiency of the quality of service at this touchpoint independent of interference from memory effects.

Third, the customer service touchpoint appears to generate more polarized experiences under transient in-situ condition. NPS score outcomes of 52.1% for this touchpoint were either 10 or 0, compared to 49.7% at video download and 40.1% at membership transaction. Therefore, although t-NPS evidence for the customer service touchpoint does not show a significantly inferior quality of service on average, we can suppose that negative experience at this touchpoint is more likely to be reinforced through the axis of time given the tendency of feedback towards polarities. This corroborates with our finding in Section 4.1 that customer service interaction has a highly quantitatively and statistically significant negative relation with relationship NPS outcome among detractors.

5. Discussion

In this study, we have applied results from three waves of data collection on NPS scores to generate insights on the drivers of negative user experience. We aimed to identify the most powerful triggers of detracting behavior and especially those of radical detraction. We found firstly that the existing management approach equating the size of a user experience issue with its priority is unwarranted, as the most significant drivers of the degree of detraction appeared to be those with weaker prevalence among survey outcomes. Second, we identified the most significant triggers of detraction as three hedonic aspects of experience, respectively “Aesthetic appeal of interface”, “Transparency of transactions”, and “Ease of customer service interaction”. Among these, we identified customer service interaction as the most significant trigger of radical detraction. Next, we used user characteristics data to further study the drivers of user experience. We found that the more mature users with higher intensity of usage and investment in the product are more likely to be concerned with hedonic aspects of experiences, whereas the population seeking improvement of the pragmatic aspects are dominated by relatively inactive non-members. Although members and non-members differ in their identification of the most salient user experience issues, their NPS distributions show largely similar patterns, indicating the failure of the membership mechanism as a trigger of loyalty. Finally, our analysis of transactional NPS data demonstrated an inferior level of user experience upon membership transaction and polarized experiences at the customer service touchpoint.

Based on our findings, we arrive at the following conclusions for building a sustainable business model in the digital applications industry. First, developers should pay due attention to hedonic aspects as long-term shapers of user experience. Past approaches which narrowly focused on the most frequently reported triggers of dissatisfaction have failed to reveal the true significance of hedonic aspects in determining user detraction. As our analysis demonstrated, hedonic aspects of user experience are the most important determinants of the user experience of mature users and are the most significant drivers of radical detraction. This shows that to improve on hedonic aspects of experience is essential to customer retention and building customer loyalty. Second, product managers cannot rely on the membership scheme as a sustainable customer relationship strategy. As we found out, the distribution of NPS outcomes of members exhibits little heterogeneity with that of non-members. Members do not have a higher likelihood of recommending the app to their relatives and friends, signaling a failure to create customer loyalty simply by drawing financial commitment to the membership.

Our findings are in congruence with the stream of sustainable user experience design literature that emphasizes the relation between hedonic pillars of user experience and product longevity, such as the dimension of integrity in Haines-Gadd’s emotional durability design framework and Nielsen’s principles for web design [24,37]. On the other hand, we also demonstrated a quantitative method combining evaluations of r-NPS and t-NPS to assess and implement the idea of a sustainable digital business model, which was conceptualized in the core tenet “Build for Sustainability” under the Principles for Digital Development [14]. Our method specifically actionized the set of expectations under the principles’ framework including partnering with users throughout the product lifecycle, ensuring user needs are being met, and gathering feedback from users to understand how the business model could be better sustained [14].

6. Conclusions

A summary of our findings regarding several hypotheses discussed in this paper is given in

Table 9. Summary of findings on hypotheses explored in this paper.

Hypothesis	Evidence	Support *
Hedonic aspects of user experience are significant in shaping the experience of detractors (Section 4.1)	Both OLS and logistic regressions showed the statistical and quantitative significance of the following hedonic aspects in determining the NPS outcome of detractors: “Aesthetic appeal of interface”, “Ease of customer service interaction”, and “Transparency of transactions”	✓
User experience improves proportionately to the volume of usage (Section 4.2.1)	An overall positive correlation between usage intensity and NPS outcome was found in both member and non-member sections	✓
There is a gap between paying and non-paying users in their user experience (Section 4.2.2)	Although members and non-members show a different pattern in salience attribution to aspects of experience that they find most needing improvement, no significant difference was found between the NPS distributions or NPS means between members and non-members	X
Several aspects of experience “Diversity of content”, “Member benefits” have a de-radicalizing effect on r-NPS outcome (Section 4.2.3)	There is a relatively thin concentration of radical detractors amid those who took issue with these two aspects. However, there is no evidence that concur with our speculation that users who took issues with these two aspects are more active mature users on average	†

* ✓ Evidence support hypothesis; † Evidence partially support hypothesis; X Evidence does not support hypothesis. Abbreviations: r-NPS, relationship-NPS.

:

There are several potential sources of bias that we should consider in future designs of NPS studies. First, in relation to internal validity, there is a worry about the arbitrariness of the classification rule for tagging users as promoters, passives and detractors. Within different cultural contexts, users may perceive the threshold “6” differently, resulting in a potentially erroneous operationalization of detraction. In the context where we operate, we retain a reasonable doubt that our users’ threshold of identification with detraction may fall well below the score of six, which more likely aligns with our users’ perception of a passive experience. In this case, classification should be adapted with respect to psychometric evidence on users’ subjective interpretation of each score threshold. Second, the study suffers from poor external validity due to a 99.15% attrition rate. To correct for this, future design needs to optimize the channel of delivery to aim for a higher response rate, for stronger representation of the total user population. Lastly, the aspects of experience we included in our Q2 questionnaires for the r-NPS surveys could be better specified according to feedback in the open-ended option. In the responses to the customary option in Q2 that we collected in the first-wave survey, users either reiterated their dissatisfaction with an already specified aspect of experience and expanded on the details, or introduced a new aspect that is not found among the 11 specified dimensions. Some of the prevailing new aspects that users mentioned such as streaming data usage, audio quality, and limited non-member benefits could be designed into the survey to enable quantitative analysis on the significance of these aspects.