Abstract
Background: Patient experience is a critical component of continuous quality improvement and value-based health-care delivery. This study aimed to identify a simple, cost-effective means of administering a validated patient experience survey in ambulatory-care settings. Methods: Patients were randomly assigned to groups to complete the validated Consumer Assessment of Healthcare Providers and Systems (CAHPS) clinician and group patient satisfaction survey using a Web-based platform. The response rate was assessed for in-clinic and e-mail distribution and was compared with the historical response rates. Patients were able to change groups after randomization to assess effect on response rate and potential bias. The cost of survey administration was compared. Results: Of 132 participants, 87 completed surveys (65.9%), with no significant differences among distribution methods. Twenty-three participants self-selected the in-clinic survey after being randomized to the e-mail cohort. Survey responses were statistically significantly different in only three of 34 questions. Web-based survey administration costs two to four times less than standard mail, phone, and mixed-modal survey administration. Conclusions: We recommend that ambulatory clinics use Web-based technology to administer CAHPS clinician and group surveys, using both e-mail and in-clinic distribution to enhance the response rate.
The Triple Aim serves as a focal point of change and innovation in health care and is at the core of the Patient Protection and Affordable Care Act of 2010. One of the tenants of the Triple Aim—improving the patient experience—is intertwined with the Institute of Medicine six dimensions of care for quality improvement.
Meanwhile, except for the patient-centered medical home and accountable care organizations, few ambulatory clinics have implemented or transformed care delivery or implemented methods of obtaining patient experience data. Without this patient information, evaluation of the processes and outcomes of the health-care system and care delivery, objective analysis, and continuous quality improvement (CQI) are impossible, leaving change to guesses and assumptions. Mandated by the Agency for Healthcare Research and Quality (AHRQ) and endorsed by the National Quality Forum as a measure of patient experience, the Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey is a validated assessment tool for patients to evaluate doctors, staff, and the overall patient experience. However, unlike hospitals, health systems, and insurance carriers, clinicians and provider groups are not required to conduct such surveys despite the current health-care system transformation and care delivery trends shifting health care to ambulatory clinic settings.
Medical organizations, regulatory agencies, and insurance companies recognize patient experience as a key factor in value-based health care due to the patient-centered focus of care delivery. Satisfactory patient experiences lead to improved clinical outcomes and affect care processes and quality, resulting in many downstream effects. Clinically, greater patient satisfaction significantly improves clinical outcomes and compliance with treatment plans and medication adherence.[1,2] From an operational standpoint, higher satisfaction significantly reduces medical malpractice claims and increases patient loyalty, staff satisfaction, and financial performance.[2]
Survey administration requires substantial resources that may exceed the fiscal and operational resources of many ambulatory clinics. However, technological advances often enhance efficiency and decrease costs. This study aimed to identify a simple, cost-effective means of administering a validated patient experience survey to generate credible data for quality reporting and CQI in ambulatory-care settings. We expect increased response rates with more efficient use of resources using Web-based technology.
Methods
The study was conducted at the Foot and Ankle Center (FAC) at the Western University of Health Sciences Patient Care Center (Pomona, California). All of the patients of the FAC who reached the age of majority and were deemed to have the capacity to complete the survey were invited to participate. The Western University of Health Sciences institutional review board approved this study.
The 34-question survey included composite and individual measures from the 2012 CAHPS Clinician and Group Survey, with additional questions from the cultural competence, health literacy, and patient-centered medical home supplemental item sets. The survey was administered using Qualtrics (Qualtrics LLC, Provo, Utah), Web-based survey software housed at the Western University of Health Sciences Department of Institutional Research and Effectiveness.
All of the patient interactions, from the invitation to participate to survey completion, exclusively involved FAC staff not involved in direct patient care. The FAC staff advised participants that the surveys were anonymous and that all of the responses were blinded to providers, staff, and students. Before randomization, and on agreement to participate, informed consent was obtained by nonclinical staff. By blindly selecting one of 20 pieces of paper labeled with the cohort name, patients were randomly assigned to survey administration in the FAC or via e-mail. After randomization, patients who requested the other cohort were allowed to change groups in an effort to assess the effect of preferences on response rate and potential bias.
The in-clinic cohort completed the survey in a nonclinical room in the clinic after the staff member entered patient demographic information. Patients completed the online survey and submitted their responses before notifying the staff to preserve confidentiality. After completion of this process, patients checked out of the FAC as they would normally do after a patient encounter.
The e-mail cohort provided their e-mail address before checking out of the FAC. At the end of the same day as their appointment, patients received an e-mail containing a link to the Web-based survey and a message reiterating the anonymity of the survey. Patients were asked to enter the same demographic data that the staff completed for the in-clinic cohort.
Surveys were conducted for 4 consecutive months. The primary outcome was response rate, defined by the number of completed surveys divided by the number of patients invited to participate in the survey. Secondary outcomes were 1) the potential bias associated with survey completion in the ambulatory clinic compared with the privacy of a location of the patient's choosing at a potentially more convenient time and 2) the cost of Web-based survey administration.
Response rates were compared between the cohorts and against historical data from the FAC patient satisfaction surveys. The historical method involved distribution of surveys at the completion of checkout protocols and required patients to mail surveys back to the FAC. Descriptive analysis was used to describe patient demographics. Shapiro-Wilk tests were used to determine the normality of the data, and Levene tests were used to determine the equality of variances. Mann-Whitney U tests were conducted to determine differences between groups for continuous variables and χ2 tests for categorical variables, with P < .05 considered significant. Surveys were also evaluated for attitudes of participants to help determine whether bias exists by group, sex, or ethnicity.
After the standard economic analysis comparing methods of services, a cost-minimization analysis was conducted.[3] Incremental cost was determined by comparing the Web-based survey administration costs with the approved AHRQ methods of mail, phone, or mixed-mode distribution on an annual basis. The total cost of using the Web-based software was based on the direct costs associated with survey administration (software licensing costs and medical assistant and data analyst wages). The expenses attributed to phone, mail, and mixed-mode survey administration were determined from proposals obtained from five AHRQ-certified companies. The average expense for each distribution method was calculated. Expenses for both groups relied on the distribution of 250 and 500 surveys per month. The average cost was calculated by dividing the average cost by the number of surveys distributed for the Web-based and AHRQ-approved methods.
Results
Response Rate
Of the 132 people asked to participate in the survey, 25 refused to participate, 20 agreed to participate but only activated the survey and did not answer items in the survey, and 87 agreed to participate and completed the survey, rendering a response rate of 65.9% (Fig. 1). The FAC group had a 72.2% response rate and the e-mail group a 58.3% response rate (Table 1). The historical method of survey administration at the FAC (n = 706) had a response rate less than 1.0%, with five people responding. There is a significant improvement in the response rate between the Web-based survey administration methods and the historical method (χ21,838 = 490.37; P < .001).
Figure 1.
Participant flow diagram.
Table 1.
Patient Population and Response Rates
Participants ranged in age from 18 to 75 years (mean ± SD, 41.0 ± 13.4 years). Of the participants, three were Spanish-speaking women who completed the survey in the clinic (Table 2). These three participants were in agreement on all items, giving the most favorable positive response, except for two items. One participant indicated “never” and two indicated “always” for the item “Saw provider within 15 minutes of appointment time,” and on the dichotomous item “Did you have a blood test,” two indicated “yes” and one indicated “no.” These participants were not included in the between-group analysis of condition.
Table 2.
Demographic Characteristics of the 87 Study Participants
Attitudes
Participants who completed the survey in the clinic trusted their provider more (U79 = 736.00; Z = 2.01; P < .05; r = 0.22), believed that their provider cared more about their health (U78 = 720.00; Z = 1.99; P < .05; r = 0.22), and believed that their provider cared more for them as a person (U79 = 713.00; Z = 2.33; P < .05; r = 0.26) than participants who completed the survey via e-mail (Table 3). No other statistically significant differences were found by condition, sex, or race/ethnicity.
Table 3.
Responses to Survey Items Significantly Different Between Groups
Cost of Survey Administration
The cost of survey administration using traditional CAHPS methods ranged from $13,253 to $31,895 for 250 surveys per month and from $21,530 to $37,752 for 500 surveys per month. The Web-based survey method projected an annual savings of 51.7% to 79.9% for 250 surveys per month and 54.5% to 74.0% for 500 surveys per month. The savings realized per survey ranges from $27.42 to $101.99 and $23.47 to $55.91 for 250 and 500 surveys per month, respectively (Table 4).
Table 4.
Cost Analysis Based on Survey Distribution Method
Discussion
Patient experience is a key factor in the establishment of value-based care and health-care transformation. The need to strive for improving the patient experience in ambulatory clinics is critical to providing the highest-quality patient care. In addition, health-care providers are, in part, incentivized to improve patient satisfaction and the patient experience. However, data collection for quality reporting, including patient experience metrics and CQI, lags behind the hospitals, health systems, and insurance companies in the ambulatory-care setting. Browne et al[2] expressed the need for evaluation of the patient experience to improve primary-care delivery and the potential use of technology in this area of health care.
Whenever surveys are conducted, the response rate is always a concern. Inadequate response volume negatively affects the credibility of the results secondary to nonresponse bias, participation bias, and selection bias.[4,5] Patient experience surveys are no exception, thus threatening the external validity of the results. Using a Web-based platform for survey administration can enhance response rates, resulting in credible data for CQI. Compared with mail-based surveys, Web-based surveys significantly shorten response times and provide immediately accessible data for analysis and CQI. Other benefits include greater survey completion, improved response accuracy due to greater item response rates, improved accuracy on sensitive questions, and fewer errors due to ease of completion and simplified data entry.[4,6-8] Meanwhile, survey administration by mail and phone produces lower response rates.[1,9] Telephone-based surveys also face increasing difficulties due to greater cell phone and voice mail use, and mail surveys have shown decreasing trends in response rates.[7] Similar to the report of enhanced response rates using Web-based surveys, we found a 65.9% response rate with Web-based surveys compared with a historical measure of 0.7% (P < .001), thus greatly enhancing the data to conduct CQI to ensure a good patient experience.
In the present study, we used the ambulatory adult CAHPS questionnaire, which is traditionally administered via mail, telephone, or mixed mode. Despite the proven external validity of the CAHPS survey methods, low response rates and the lag time between the patient encounter, survey administration, and survey completion can result in selection bias and recency bias.[9,10]
Although the use of Web-based survey distribution significantly increased response rates, it creates a new potential for biased responses. In general, Web-based surveys affect socially disadvantaged groups, individuals with limited computer/Internet knowledge or access, lower income levels, racial and ethnic minorities, and those less educated.[1,5,7] Although an inverse relationship between age and Internet access has been observed, this is becoming less prevalent.[9] This potential cause of selection and participation bias was mitigated, or potentially negated, because online access was provided to participants in the in-clinic cohort and crossover was permitted after randomization. The in-clinic offering was consistent with the recommendation of Broadwater-Hollifield that emergency departments use Web-based surveys with multiple sampling strategies or provide patients the option of completing the survey in the emergency department at the time of discharge to avoid bias.
Completion of the survey in the clinic where care is provided can promote nonblinding bias and faking good bias.[10] The restricted access to survey responses using software permissions eliminates nonblinding bias. We addressed the need to answer favorably through verbal and written reinforcement of the anonymous nature of survey responses before the participants initiated the survey. To further prevent the potential underreporting of undesirable answers, patients could change groups to participate via e-mail in a seemingly safer environment. However, no participant switched to the e-mail cohort (Table 2).
In support of the lack of bias from Web-based surveys, we observed only three questions with significant differences in the responses between the cohorts: trust in the provider, whether the participant felt that the provider cared about his or her health, and whether the provider cared about the participant as a person. Although statistically significant, the effect size of all three items indicates the presence of some effect but only to a small extent in this sample. This effect could be due to the three to four participants of 35 in the e-mail cohort who responded “always,” unlike most of the participants in this group. One participant answered “sometimes” to all three questions, indicating that this individual's experience was different from that of the broader population. Despite the potential for various types of bias created by the new method of survey administration, we found no statistically significant differences in the responses by race/ethnicity of participants.
Leveraging technological advancements frequently improves efficiency. Previous studies of Web-based survey administration also report decreased costs.[4,7,8] Harewood et al[8] reported that mail and phone surveys use substantial resources, rendering them time-consuming and costly. The present findings support this notion, with annual Web-based survey costs 51.7% to 79.9% less than those for phone, mail, and mixed-mode distribution survey formats. The Web-based survey administration was found to be 15.6% and 3.6% higher than the AHRQ-certified companies using e-mail survey distribution exclusively. However, the average costs show that the incremental difference is significantly reduced with greater volume, and the Web-based technology likely demonstrates a savings in all groups in high-volume clinics.
This study has several limitations. First, conducting the study negatively affected staff time. In an effort to complete their primary job responsibilities, the nonclinical staff responsible for all of the patient interactions was not always available to recruit all of the patients. Although this time limitation depended on the staff daily operational responsibilities, independent of patient demographics, there is still the potential for selection bias. This limitation is unlikely to factor in to practical implementation of this survey administration method outside of the research setting because the staff would not be required to follow time-intensive aspects of research protocols involving patient recruitment and informed consents. This aspect of the study was done to meet ethical research requirements and is not applicable to daily clinical operations for the purpose of obtaining patient feedback.
Inconsistent provider participation limited the number of patients invited to participate. However, as federal and state agencies mandate that health-care provider participation and payers align reimbursement and patient experience, this limitation will become moot.
The last limitation is that some of the nonclinical staff involved were not bilingual. This probably negatively affected the Spanish-speaking population, resulting in participation and selection bias, creating the lower response rate in the Spanish-speaking patients. We recommend addressing these limitations by conducting a follow-up study with greater cultural sensitivity to determine the external validity of the findings in larger populations by involving communities that have greater linguistic, ethnic, racial, and economic variance.
Conclusions
Regulatory and market forces are driving implementation of the CAHPS Clinician and Group Survey so that more widespread use of patient experience measures is anticipated with health-care delivery transformation.[11] The substantial, and potentially prohibitive, demand on resources to conduct these surveys in ambulatory-care settings will likely lead to inadequate response rates and a lack of credible data for CQI and quality measure reporting. Ultimately, this likely creates the scenario whereby ambulatory-care delivery is the rate-limiting step in providing value-based health care.
Leveraging Web-based technology when conducting patient experience surveys can improve efficiency and outcomes in a cost-effective manner because it enhances validity by significantly increasing response rates. Survey distribution via e-mail affords patients convenience with greater privacy, and in-clinic survey participation reduces participation and selection bias. Because we found no difference in response rates between survey administration in the clinic or via e-mail, except for three questions with a small effect, we suggest that ambulatory clinics use Web-based CAHPS surveys using either method to maximize the response rate and mitigate potential bias.
Financial Disclosure
None reported.
Conflict of Interest
None reported.
References
- Broadwater-Hollifield C, Fair J, Podolsky S, et al: Web-based emergency department patient satisfaction surveys may introduce potential for bias. J Emerg Med46: 404, 2014.
- Browne K, Roseman D, Shaller D, et al: Analysis & commentary: measuring patient experience as a strategy for improving primary care. Health Aff29: 921, 2010.
- Brouwer W, Rutten F, Koopmanschap M: “Costing in Economic Evaluations,”inEconomic Evaluation in Healthcare: Merging Theory with Practice, edited byMDrummond, AMcGuire, p68, Oxford University Press, New York, 2001.
- Bowling A: Mode of questionnaire administration can have serious effects on data quality. J Public Health27: 281, 2005.
- Schonlau M, van Soest A, Kapteyn A, et al: Selection bias in web surveys and the use of propensity scores. Soc Methods Res37: 291, 2009.
- Heiervang E, Goodman R: Advantages and limitations of web-based surveys: evidence from a child mental health survey. Soc Psychiatry Psychiatr Epidemiol46: 69, 2011.
- Pealer LN, Weiler RM, Pigg RMJr, et al: The feasibility of a web-based surveillance system to collect health risk behavior data from college students. Health Educ Behav28: 547, 2001.
- Harewood GC, Wiersma MJ, de Groen PC: Utility of Web-based assessment of patient satisfaction with endoscopy. Am J Gastroenterol98: 1016, 2003.
- Shih T-H, Fan X: Comparing response rates from web and mail surveys: a meta-analysis. Field Methods20: 249, 2008.
- Choi BC, Pak AW: A catalog of biases in questionnaires. Prev Chronic Dis2: A13, 2005.
- Aligning Forces for Quality—Improving Health & Health Care in Communities Across America: Primer/Brief: Forces Driving Implementation of the CAHPS Clinician & Group Survey,Robert Wood Johnson Foundation, Princeton, NJ, March2013.
© 2017 American Podiatric Medical Association