Using Decision Tree to Predict Response Rates of Consumer Satisfaction, Attitude, and Loyalty Surveys

: Response rate has long been a major concern in survey research commonly used in many ﬁelds such as marketing, psychology, sociology, and public policy. Based on 244 published survey studies on consumer satisfaction, loyalty, and trust, this study aimed to identify factors that were predictors of response rates. Results showed that response rates were associated with the mode of data collection (face-to-face > mail / telephone > online), type of survey sponsors (government agencies > universities / research institutions > commercial entities), conﬁdentiality (conﬁdential > non-conﬁdential), direct invitation (yes > no), and cultural orientation (individualism > collectivism). A decision tree regression analysis (using classiﬁcation and regression Tree (C&RT) algorithm on 80% of the studies as the training set and 20% as the test set) revealed that a model with all above-mentioned factors attained a linear correlation coe ﬃ cient (0.578) between the predicted values and actual values, which was higher than the corresponding coe ﬃ cient of the traditional linear regression model (0.423). A decision tree analysis (using C5.0 algorithm on 80% of the studies as the training set and 20% as the test set) revealed that a model with all above-mentioned factors attained an overall accuracy of 78.26% in predicting whether a survey had a high ( > 50%) or low ( < 50%) response rate. Direct invitation was the most important factor in all three models and had a consistent trend in predicting response rate.


Introduction
A high response rate is essential to the success of survey research in many fields such as marketing, sociology, psychology, public policy, and public health [1]. A low response rate is likely to introduce nonresponse bias and hence lower the external validity or generalizability of the survey results, because respondents and nonrespondents typically differ in the variables of interest [2]. The importance of high response rates is especially salient for consumer and marketing survey research because service organizations and firms need to gauge accurately customers' attitudes toward services or products (e.g., satisfaction, loyalty, and trust). These attitudes play an important role in determining long-term customer behavior and subsequently the success of firms and service organizations in terms of customer retention and market share [3,4]. Attracting new customers is considerably more expensive than retaining old customers [5]. Firms and service providers need valid survey results in order to make decisions about their efforts at increasing customer retention. Unfortunately, past decades have witnessed declining response rates in all forms of business and academic research [6,7], so it is imperative to understand factors that influence response rates.
Some of the above studies were experimental studies that investigated whether one or more factors were effective in raising response rates [38,48,49], whereas other studies were either quantitative or qualitative analysis of response rates of existing studies as a function of factors such as mode of data collection as mentioned earlier [14,16,24,39,[50][51][52][53]. Some researchers [24,39] have also integrated previous findings and constructed regression models to predict response rates. Although there has been much research on the influence of various factors on response rates, Helgeson et al. [54] concluded that "any reading of this large body of works leads to the conclusion that few survey design factors have a consistent and significant effect on observed response rates" (p. 305).
In order to improve on previous attempts at understanding the factors influencing response rates, this study used a decision tree approach (with a training set and a test set of studies) to quantitatively analyze the influence of survey attributes on response rates in a large number of published studies. A decision tree approach has several advantages over the traditional logistic regression: Automatic consideration of potential interactive effects among predictors, high tolerance of multicollinearity, intuitive guidance for application of the results in decision making (ranked priority or importance of factors, cutoff point for continuous predictors), and automatic exclusion of unimportant factors. Two main sets of analyses were conducted, with the first treating the dependent variable-response rate-as a continuous variable (the decision tree regression model) and the second treating the response rate as a dichotomized variable (the decision tree classification analysis). These results were further compared to those based on traditional linear and logistic regression analyses.

Sample of Studies
In this study, we focused on previous survey studies of consumer satisfaction, loyalty, and attitudes about particular topics or trust in service organizations. We conducted searches using major electronic reference databases, including ERIC, JSTOR, MEDLINE, PsycARTICLES, PsycINFO, Science Direct, Social Science Abstracts, Sociological Abstracts, and Web of Science (Social Science Citation Index). We used three clusters of key words for the searches: (1) "questionnaire" or "survey"; (2) "response rate", "return rate", or "participation rate"; and (3) "satisfaction", "loyalty", "attitude", or "trust". The search results were examined for their relevance and the selected studies' lists of references were further investigated for additional potentially relevant studies. The searches yielded 450 studies, covering a wide range of topics in many sectors of the service industry such as restaurants, hotels, hospitals, telecommunications, websites, etc. These studies were closely examined for their reports of survey attributes. Studies with limited information on the survey attributes (see below) were excluded from further analyses, yielding a final sample of 244 studies.

Coding of Main Variables
The main outcome variable of this study was response rate. Its calculation is deceptively simple-the number of valid surveys returned over the total number of people surveyed. As Groves and Lyberg [55] pointed out, there were actually many ways of calculating response rates because the total number of people surveyed was subject to different interpretations (those who were initially contacted or those who expressed initial interest or those who were followed up with the actual surveys). In this study, we used the minimum response rate, which is the number of completed surveys divided by the number of surveys returned (completed or partially completed) plus the number of nonsurveys (refusals, breakoffs, or noncontacts) plus all cases of unknown eligibility. Although many studies did not report a breakdown of the specific numbers of partially completed surveys, refusals, break-offs, noncontacts, etc., the minimum response rate could be calculated based on the total number of surveys initially sent out and the final number of completed surveys.
We recorded and coded nine survey attributes or features of each survey study: Mode of data collection, type of survey sponsors, incentives, questionnaire length, relevance of the survey topic to the respondents, sensitivity of the topic, confidentiality, direct invitation, and country or region. These features were selected because they have been shown to be important in previous studies [21,56] and because they could be coded in the studies we found. Table 1 describes the coding categories of these survey attributes as well as their descriptive statistics. Although there are a number of modes of data collection or survey methods including face-to-face interviews, mails, telephone, internet, e-mail, or other social media platforms, previous research has demonstrated that response rates varied from high to low for three modes of data collection: Face-to-face interviews, mail surveys, and telephone or web surveys [57][58][59].We coded our studies into four major categories: Face-to-face interviews, mail surveys, telephone, and online surveys.
The type of survey sponsors was classified into three categories following Reference [31]: Government agencies, universities and other research institutions or non-profit organizations, and companies and other commercial institutions.
Incentives for participation were coded in terms of the amount of money offered to the respondents.
Questionnaire length was coded based on the number of questions or items included in the surveys reported in the studies.
Whether the survey topic was of relevance to the respondents was determined according to whether potential respondents had direct experience with the particular services or products covered by the surveys. Goyder [60] showed that such past experience increased survey participation.
The surveys were coded for whether they covered sensitive topics. Albaum and Smith [59] defined sensitive topics as those that possess a substantial threat to the survey participants either because they are intrusive to their privacy or because they could raise fears about potential repercussions/consequences of disclosing the information requested, including financial assets, sexual behaviors, illegal drug use, etc. Singer [61,62] has argued that privacy-related sensitive topics would seem intrusive to potential respondents and thus lower participation rates.
Confidentiality was coded based on whether the survey was confidential (or anonymous) or not confidential.
Direct invitation for surveys meant that researchers invited participants via face-to-face contact or telephone, whereas indirect invitation was via mail or e-mail, following the definition used by Bowling [57]. In an earlier, study, Groves and Couper [8] found that direct invitation led to higher participation rates.
Finally, we coded the country or region based on where the surveys were conducted. Most of the studies were conducted in USA (31.89%), followed by China (11.89%), Taiwan/China (10.81%), and Korea (9.73%), with the remaining studies conducted in other countries or regions. Previous research has focused mainly on the influence of individualism/collectivism on response rates [9,21]. These studies simply classified European countries and America as individualistic countries, and Asian countries (e.g., China) as collectivistic countries. In this study, we used specific values of the Hofstede's individualism/collectivism dimension [63] for each country.

Methods
We first analyzed the influence of each survey attributes on response rates by performing ANOVA in SPSS 22.0. Then, we constructed a decision tree regression model by applying the classification and regression tree (C&RT) algorithm and a linear regression model with the response rate as the dependent variable and all attributes as predictors in IBM SPSS Modeler 18.0. Finally, we used 50% as the cutoff point to divide our sample studies into those with 'high' response rates and those with 'low' response rates, and the decision tree classification model was constructed with all useful attributes as predictors by applying the C5.0 algorithm. The cutoff of 50% was selected because it was near the mean response rate in our study (also see Reference [64]). C5.0 is one of the classic decision tree algorithms. It can learn to predict discrete outputs based on the values of the inputs it receives. Whether the inputs are linear data or nonlinear data, and even with missing values, C5.0 could perform robustly. C5.0 splits the sample based on the field of the maximum information gain (entropy) brought by the input variable. Information entropy reflects the degree of information clutter, and the more impure the information is, the larger the information entropy is. Finally, C5.0 generates a decision tree or rule set with very straightforward explanations.

Results
After coding the data, two survey attributes were found to lack information or variations. Only 37 of the 244 studies provided information about incentives, and very few studies (2.87%) covered sensitive topics such as financial assets, sexual behaviors, and illegal drug use (see Table 1). These two variables were not used in subsequent analyses.
We performed ANOVA by using SPSS 22.0 and found that five of the remaining seven factors showed significant associations with response rates in the expected direction (Figures 1-5). First, the mode of data collection was a significant factor, F(3, 238) = 60.460, p < 0.001, with highest response rates for face-to-face interviews (M = 77.81%), followed by mail surveys (M = 56.19%) and telephone surveys (M = 55.77%), and lowest response rates for online surveys (M = 24.20%) (Figure 1). Post hoc comparisons (all Bonferroni corrected) showed that the difference in response rates between telephone surveys and mail surveys was not significant (p = 0.128), but the differences between other modes of data collection were significant, p ≤ 0.001. Second, the type of survey sponsors was also a significant factor, F (2, 240) = 9.233, p < 0.001. As shown in Figure 2, the mean response rate of surveys sponsored by government agencies was the highest (M = 93.50%), followed by those sponsored by universities and research institutions (M = 52.05%), and then by those sponsored by commercial institutions (M = 29.79%). Post hoc analysis showed that the three types of sponsors differed significantly from one another, p < 0.05. Third, confidential surveys showed higher response rates (M = 60.33%) than non-confidential surveys (M = 47.44%), F(1, 173) = 4.342, p = 0.039 ( Figure 3). Fourth, direct invitation was significantly associated with higher response rates, F(1, 235) = 109.315, p < 0.001. The mean response rate was 65.41% when invitations for participation were through either face-to-face contact or phone, but only 29.65% when they were sent by mail or email ( Figure 4). Finally, the individualism and collectivism index of the countries or regions where surveys were conducted was correlated with response rates, r = −0.379, p < 0.001. Greater individualism was linked to lower response rates.
Contrary to our hypothesis, relevance of the topics was associated with lower response rates, F (1, 240) = 4.339, p = 0.038, with a mean response rate of 59.95% for studies on non-relevant topics and 49.43% for studies of relevant topics ( Figure 5). Finally, the correlation between the number of items and response rates was not significant, r = 0.046, p = 0.478. We also examined non-linear relations between the two variables and found no significant results.  Figure 3). Fourth, direct invitation was significantly associated with higher response rates, F(1, 235) = 109.315, p < 0.001. The mean response rate was 65.41% when invitations for participation were through either face-toface contact or phone, but only 29.65% when they were sent by mail or email ( Figure 4). Finally, the individualism and collectivism index of the countries or regions where surveys were conducted was correlated with response rates, r = −0.379, p < 0.001. Greater individualism was linked to lower response rates.            Contrary to our hypothesis, relevance of the topics was associated with lower response rates, F (1, 240) = 4.339, p = 0.038, with a mean response rate of 59.95% for studies on non-relevant topics and 49.43% for studies of relevant topics ( Figure 5). Finally, the correlation between the number of items

Predicting Response Rates
In the previous section, we examined bivariate relations between survey attributes and response rates and identified significant survey attributes influencing response rates. In this section, we used all attributes to construct a decision tree regression model to predict response rate as a continuous variable. The classification and regression tree (C&RT) algorithm implemented in IBM SPSS Modeler 18.0 was used to construct the prediction model. The data were divided into the training set (80%) and the test set (20%). Results showed that the linear correlation coefficient between the predicted values of decision tree regression model and actual values was 0.722 in the training set and the linear correlation coefficient was 0.578 in the test set.
Of all the predictors, direct invitation had by far the highest importance, whereas all other attributes (i.e., mode of data collection, confidentiality, relevance of topics, type of survey sponsors, and questionnaire length) had similarly low levels of importance. After pruning, only direct invitation remained as a significant factor (see Figure 6). and response rates was not significant, r = 0.046, p = 0.478. We also examined non-linear relations between the two variables and found no significant results.

Predicting Response Rates
In the previous section, we examined bivariate relations between survey attributes and response rates and identified significant survey attributes influencing response rates. In this section, we used all attributes to construct a decision tree regression model to predict response rate as a continuous variable. The classification and regression tree (C&RT) algorithm implemented in IBM SPSS Modeler 18.0 was used to construct the prediction model. The data were divided into the training set (80%) and the test set (20%). Results showed that the linear correlation coefficient between the predicted values of decision tree regression model and actual values was 0.722 in the training set and the linear correlation coefficient was 0.578 in the test set.
Of all the predictors, direct invitation had by far the highest importance, whereas all other attributes (i.e., mode of data collection, confidentiality, relevance of topics, type of survey sponsors, and questionnaire length) had similarly low levels of importance. After pruning, only direct invitation remained as a significant factor (see Figure 6). We then compared our decision tree regression model with a prediction model based on traditional linear regression. The results of the traditional linear regression were: F (7,88) = 9.273, p < 0.001, adjusted R 2 = 0.379. The correlation coefficient between the predicted values of the traditional linear regression model and actual values was 0.615 in the training set and 0.423 in the test set, both of which were lower than the corresponding coefficients of our regression tree model (see Table 2). Consistent with the results of the decision tree model, direct invitation had the highest importance (β = 0.498, p < 0.001). But the second most important attribute was confidentiality (β = We then compared our decision tree regression model with a prediction model based on traditional linear regression. The results of the traditional linear regression were: F (7,88) = 9.273, p < 0.001, adjusted R 2 = 0.379. The correlation coefficient between the predicted values of the traditional linear regression model and actual values was 0.615 in the training set and 0.423 in the test set, both of which were lower than the corresponding coefficients of our regression tree model (see Table 2). Consistent with the results of the decision tree model, direct invitation had the highest importance (β = 0.498, p < 0.001). But the second most important attribute was confidentiality (β = 0.159, p = 0.058) and the third was mode of data collection (β = −0.154, p = 0.066).

Predicting High and Low Response Rates
In the previous section, we used all attributes to construct a decision tree regression model to predict response rates as a continuous variable. Because survey researchers typically aim for a response rate above a threshold [64] and the average response rate of published surveys was 52.7% [64], we used 50% as the cutoff point to divide our studies into those with "high" response rates (i.e., at or above the threshold, n = 110 studies) and those with "low" response rates (i.e., below the threshold, n = 134). C5.0 algorithm implemented in IBM SPSS Modeler 18.0 was used to construct the prediction model. The data were divided into the training set (80%) and the test set (20%).
Results showed that the prediction model's overall accuracy in the test set was 78.26% with the C5.0 algorithm (Table 3). Table 3. Recall and precision of the prediction model.

Recall Rate a Precision Rate b
High response rates 78.57% 78.96% Low response rates 71. Of all the predictors, direct invitation had the highest importance, followed by mode of data collection and individualism. The length of survey was also important. Figure 7 shows the details of the decision tree and Figure 8 was the decision tree after pruning. Results showed that "high" response rates were obtained when using direct invitation (70% with direct invitations vs. 17% without). Of those with direct invitations, face-to-face interview and mail survey led to "high" response rates for 78% and 73% of the studies as compared to 50% and 0% for telephone and online surveys. For telephone surveys, individualism was a key factor, with lower individualism countries (59 or lower scores) leading to "high" response rates (69%), whereas all five surveys conducted in higher individualism countries showed "low" response rates. For face-to-face interviews, the length of survey was a key factor, with shorter surveys (fewer than 39 items) leading to "high" response rates (all seven studies, 100%), whereas all longer surveys (more than 39 items) showed "low" response rates.
Finally, we compared our decision trees model with a prediction model based on traditional logistic regression. Results showed that the overall accuracy with the testing dataset was 73.10%, which was lower than the accuracy of our model (78.26%). Consistent with the results of the decision tree model, direct invitation had the highest importance (β = 2.506, p = 0.002), followed by mode of data collection (p = 0.582), and type of survey sponsors (p = 0.741).
response rates for 78% and 73% of the studies as compared to 50% and 0% for telephone and online surveys. For telephone surveys, individualism was a key factor, with lower individualism countries (59 or lower scores) leading to "high" response rates (69%), whereas all five surveys conducted in higher individualism countries showed "low" response rates. For face-to-face interviews, the length of survey was a key factor, with shorter surveys (fewer than 39 items) leading to "high" response rates (all seven studies, 100%), whereas all longer surveys (more than 39 items) showed "low" response rates.  Finally, we compared our decision trees model with a prediction model based on traditional logistic regression. Results showed that the overall accuracy with the testing dataset was 73.10%, which was lower than the accuracy of our model (78.26%). Consistent with the results of the decision tree model, direct invitation had the highest importance (β = 2.506, p = 0.002), followed by mode of data collection (p = 0.582), and type of survey sponsors (p = 0.741).

Discussion
Previous studies have identified a number of factors influencing participation rates in survey research, including societal factors, individual participants' characteristics, and surveys' attributes [8]. The current study aimed to improve on previous research by using a decision tree approach to predict both continuous and dichotomized response rates from surveys' attributes. After identifying 244 relevant surveys and extracting nine survey attributes from each study, we found seven usable attributes (i.e., mode of data collection, type of survey sponsor, relevance of the topic, confidentiality, direct invitation, length of survey, and collectivism/individualism orientation of the country or region). First, a decision tree regression model was trained with 80% of the data and tested with the remaining 20% of the data to predict response rate. Results showed that the decision tree regression model achieved a higher linear correlation coefficient (0.578) between the predicted values and actual values than a traditional linear regression model, and direct invitation had the highest importance in

Discussion
Previous studies have identified a number of factors influencing participation rates in survey research, including societal factors, individual participants' characteristics, and surveys' attributes [8]. The current study aimed to improve on previous research by using a decision tree approach to predict both continuous and dichotomized response rates from surveys' attributes. After identifying 244 relevant surveys and extracting nine survey attributes from each study, we found seven usable attributes (i.e., mode of data collection, type of survey sponsor, relevance of the topic, confidentiality, direct invitation, length of survey, and collectivism/individualism orientation of the country or region). First, a decision tree regression model was trained with 80% of the data and tested with the remaining 20% of the data to predict response rate. Results showed that the decision tree regression model achieved a higher linear correlation coefficient (0.578) between the predicted values and actual values than a traditional linear regression model, and direct invitation had the highest importance in both models predicting response rate. We further trained a decision tree model with 80% of the data and tested with the remaining 20% of the data to predict high or low response rate with 50% as the cutoff point. Results showed an accuracy of 78.26% on the test dataset. The most important factor was also whether surveys used direct invitation. Within surveys with direct invitations, mode of data collection was also a crucial factor, favoring face-to-face or mail surveys over telephone or online surveys. Within telephone surveys, surveys conducted in low-individualism countries were more likely to obtain high response rates. Within face-to-face interviews, the length of survey became a crucial factor, with 39 items as the cutoff point.
Although the current study did not examine any new factors affecting response rates as compared to previous studies, it used a new approach-the decision tree approach-that has several advantages over previous studies: Automatic consideration of potential interactive effects among predictors, high tolerance of multicollinearity, and intuitive guidance for application of the results in decision making (ranked priority or importance of factors, cutoff point for continuous predictors). Indeed, our results differed from previous studies in several ways. Our models performed better than traditional models. Furthermore, our models are simpler (after pruning). For example, several factors (i.e., mode of data collection, type of survey sponsors, confidentiality, individualism vs. collectivism) showed significant effects at the bivariate level, which was consistent with previous literature [57][58][59], but they were not of importance in our decision tree models.
Our results can be explained by Furse's [65] extension of the classic cognitive dissonance theory [66] to survey participation. According to the cognitive dissonance theory, when people sense a dissonance in their own cognitive elements (e.g., views, thoughts, opinions, desires, or intentions), they feel an unpleasant state of tension, which would motivate them to change some of the cognitive elements in order to reduce or eliminate the dissonance-induced tension. Although Hackler and Bourgett [67] initially suggested to increase survey participation by creating "a dissonance that could be resolved by returning the questionnaire to the researcher", Furse [65] provided a detailed description of the role of cognitive dissonance in survey participation as a sequence of decisions. Albaum's [57] model is able to integrate much of the empirical literature on inducement techniques used in survey research. In terms of our results, the decision process within the potential respondents' mind was triggered by the invitation for participation, with a direct invitation generating a greater tension (or dissonance with a person's self-belief of being helpful) than an indirect invitation. After agreeing to the direct invitation, there was a dissonance if the potential participants were thinking of not following through and completing the survey. This dissonance seemed to be greater for face-to-face interviews and mail surveys than for telephone and online surveys, perhaps due to direct and personalized contact for the former. Finally, for face-to-face interviews, the length of the survey was a major factor. Peytchev [47] also found that page and question characteristics (e.g., number of items) had a stronger influence on survey break-off than respondent characteristics. Based on our results, the critical length was 39 items.
Although our study shed new light on important factors influencing survey response rates, two main limitations of this study need to be mentioned to guide future research. First, our study focused on a subarea of survey research (mainly related to consumer satisfaction and product loyalty and trust), so it is not clear whether our findings can be generalized to broader social research, which may involve more sensitive questions, greater compensations or incentives, or longer surveys. Second, we focused on survey attributes because of they are within the control of survey researchers, but other types of factors (e.g., societal and individual participants' characteristics [8]) may also need to be considered for particular types of surveys.