1. Introduction
The Patient-Centered Medical Home (PCMH) care model is gaining prominence as a way of improving primary care. The PCMH is commonly defined by an emphasis on comprehensive, team-based care; patient-centered care; coordination across different aspects of the health care system; access to care; and a commitment to quality and safety [
1]. The adoption of PCMH functions have been encouraged under the Affordable Care Act, and multiple payers have provided financial incentives for practices to become medical homes [
2,
3].
As PCMH adoption expands, the ability to evaluate patient experiences has become critical in evaluating the impact of the medical home [
4]. With Commonwealth Fund support, NCQA collaborated with the Consumer Assessment of Healthcare Providers and Systems (CAHPS
®, Agency for Healthcare Research and Quality, Rockville, MD, USA) Consortium, overseen by the Agency for Healthcare Research and Quality (AHRQ), to develop a survey instrument functionally aligned with key PCMH functions. This instrument, the CAHPS PCMH, finalized in late 2011, evaluates patient experiences on key domains of care associated with the medical home [
5], and has since been used in a variety of settings, including National Committee for Quality Assurance’s (NCQA) PCMH recognition program [
6]. A few studies have provided some support for the reliability and validity of the survey [
3,
7], though their results were based on initial data or smaller samples than those used in our current study (almost 7500 respondents in two census regions across prior studies [
3,
7]).
Nonetheless, even as several national and state initiatives have adopted the CAHPS PCMH version for public reporting or evaluation efforts [
8,
9], there have been concerns raised about the length of the survey. For example, low uptake rates have persisted since the survey was first introduced for use in the NCQA’s PCMH recognition program, the most widely used method for qualifying practices for rewards in multi-payer PCMH demonstrations [
3,
5]. While over 11,000 practices, representing an estimated 15%–18% of primary care physicians, are currently recognized by NCQA [
10], fewer than 3% of them submit patient experiences surveys to NCQ when applying for recognition under NCQA’s PCMH recognition program.
Based on these concerns, and growing attention to survey length, NCQA evaluated and shared initial psychometric results, and gathered qualitative input from multiple stakeholders to make recommendations for shortening the survey used for NCQA’s programs. Stakeholders included about a dozen clinicians, researchers, survey implementers, those who work with practices to improve patient experiences, and those who use the survey for public reporting purposes, as well as about a dozen patient advocates and a separate broad-based advisory panel.
The growing attention to survey length is not limited to NCQA efforts, and attempts to address this concern have also grown. For example, in the months since NCQA initiated its evaluation, the CAHPS Consortium also released a slightly shorter version of the CAHPS Clinician and Group survey (version 3.0), which forms the core of the CAHPS PCMH survey, reducing the length from 34 to 31 items [
11,
12]. Additional possibilities for shortening the CAHPS Clinician and Group survey were also published recently [
13].
This paper compares the psychometric properties of the original and proposed shortened survey based on the CAHPS PCMH survey. We used data reported in 2013 by 326 practices (including over 30,000 patients from three census regions), using data reported by medical practices in 2013. The proposed reductions shorten the adult survey from 52 to 26 items (from 10 to 5 pages), and the child survey from 66 to 31 items (from 12 to 6 pages).
2. Methods
2.1. Data and Materials
Data were from practices that voluntarily submit CAHPS PCMH data to NCQA as part of NCQA’s PCMH recognition program. All practices used NCQA-certified survey vendors to collect data. NCQA requires all survey vendors participate in annual training and monitoring of survey administration procedures.
Practices may submit data on the adult or child versions of the CAHPS PCMH survey items. The CAHPS PCMH survey uses the CAHPS Clinician and Group (C&G) core survey (version 2.0), plus an additional PCMH set of items covering topics beyond the core. The core survey includes multi-item composites assessing access to care, communication with providers, office staff, and an overall provider rating scale. The PCMH item set assesses shared decision making, self-management support, comprehensiveness of care, coordination of care, information about care, and additional aspects of access. All questions assess care in the past 12 months. Complete details of the CAHPS PCMH survey are available at AHRQ [
14], including details of the new slightly shortened CAHPS C&G core survey (version 3.0) [
12]. Our proposed PCMH survey is not tied to these changes in version 3.0 of the C&G core survey (although items dropped from the C&G version 3.0 are also dropped from our proposed survey).
2.2. Sample and Survey Protocol
Practices voluntarily submitting survey data to NCQA must follow procedures that NCQA requires for sample selection. For each survey administered, a random sample of patients is drawn based on the number of clinicians at a practice site (1 clinician in a practice = a required sample size of 128; 2–3 clinicians = 171 sample size; 4–9 clinicians = 343 sample size; 10–13 clinicians = 429 sample size; 14–19 clinicians = 500 sample size; 20–28 clinicians = 643 sample size; 29 or more clinicians = 686 sample size).
Practices choose a random selection of adults (aged ≥18 years) and pediatric (aged <18 years) patients who had at least 1 visit to a provider in the past 12 months prior to survey completion. A parent or guardian is asked to complete the survey for eligible children. In 2013, 268 practices submitted data for adult patients (n = 27,896 respondents); 58 practices submitted data for child survey patients (n = 4277 respondents). Data were submitted to NCQA in April and September 2013, and surveys had to be administered within the 15 months prior to submission. The last month of data collection allowed was August 2013. The survey administration protocol included mail only, telephone only, mail with telephone follow-up and Internet only administration options. The majority of practices used mail only administration (78% adult, 79% child), with smaller proportions using Internet only (12% adult, 15% child), telephone only (10% adult, 5% child) or mail with telephone follow-up (none in adult, 1% child) administration.
2.3. Analysis
We calculated internal consistency reliability (Cronbach’s alpha) of multi-item composites; practice-level unadjusted mean scores for each composite; and site-level reliabilities for each item and composite. Following prior methods used to report CAHPS PCMH results in the literature, we calculated scores using proportional scoring and the summated rating method—
i.e., we calculated the mean responses to each item, after transforming each response to a 0–100 scale (100 representing the most positive response on any given item response scale; 0 representing the least positive) [
3,
7]. For example, on a Yes/No response scale, if “Yes” represents the most positive response, then Yes = 100 and No = 0; on an Always/Usually/Sometimes/Never response scale, if “Always” represents the most positive response, then Always = 100, Usually = 67, Sometimes = 33 and Never = 0. A higher score means that practices were rated more positively for care on that item. We use this 0–100 scale to facilitate comparison of our results to prior, peer-reviewed published CAHPS PCMH results that were reported based on a 0–100 possible range off scores [
3,
7]. We examined site-level reliabilities by differentiating between-site and within-site variance in one-way ANOVAs [
3,
7].
We also assessed the extent to which shortening the access and communication composites resulted in changes to the relative ranking of practices. Specifically, we examined the extent to which the ranking of practices shifted under the revised survey composites using two statistical tests. First, we conducted a Pearson’s Chi-Squared test that examined the relationship between (categorical) quintile rankings of practices in the revised versus original composites. Second, we examined the rank order correlations among practices using the short and long versions of each composite. Both of these analyses were conducted on each composite (access and communication) for all samples (child and adult).
2.4. Proposed Revisions to Shorten the Survey
Based on initial psychometric analysis and stakeholder input, we propose a shorter survey—reducing the adult tool from 52 to 26 items, and the child tool from 66 to 31 items. We consulted 22 stakeholders, representing a variety of perspectives: 11 were clinicians, researchers, survey implementers, those who work with practices to improve patient experiences, and those who use the survey for public reporting purposes; another 11 were patient advocates identified in collaboration with the National Partnership on Women and Families and the Institute for Patient and Family Centered Care. We asked all stakeholders to provide input and select items for a shortened survey based on several key principles: Which items are psychometrically sound (i.e., site-level reliability of 0.70 or higher)? Which items are conceptually central to the PCMH model? Which items are important to consumers? Which items are actionable?
We gathered qualitative input during discussions with stakeholders, including the rationale for prioritizing items based on the above principles. As part of this process, we also asked stakeholders to vote to either “keep” or “drop” items for a shortened survey. The final selection of items was based on this input, including items that were prioritized by stakeholders and garnered the largest number of “keep” votes.
Based on stakeholder input, key changes include reductions in access, communication and comprehensiveness of care composites for the adult and child tool. Because stakeholders did not prioritize the shared decision-making and office staff composites, or several individual (non-composite) items related to access, information, and coordination of care, the proposed shortened survey drops these composites and items (further detail on all items retained for the shortened survey are in the Results).
Item-level results often informed stakeholder input regarding which items could be dropped for a proposed shorter survey. Generally, stakeholders agreed that items achieving estimated reliabilities of less than 0.70 at the practice level could be dropped. For example, an item in the access composite—getting answers to medical questions as soon as needed when phoning one’s provider after-hours—did not achieve 0.70 reliability (0.45 adult, 0.42 child) and was dropped. Self-management support items also did not achieve 0.70 reliability and were dropped.
There were some exceptions, however, including if the item met other guiding principles, such as being conceptually important to the PCMH model or to consumers. For example, a coordination of care item—provider seemed informed and up-to-date about care received from specialists—did not achieve 0.70 reliability (0.66 adult, 0.20 child). However, most stakeholders deemed this item too conceptually important to the PCMH model to be dropped; thus, the item was retained. Conversely, some items achieved 0.70 site-level reliability, but based on concerns over survey length and other guiding principles, stakeholders did not prioritize these items. For example, two items in the access composite (got appointment for routine care; saw provider within 15 min of appointment time) achieved site-level reliabilities above 0.70, but most stakeholders did not deem these two items as conceptually important relative to others in the composite; one of the items also had a lower item-scale correlation with the total composite. Thus, the proposed shortened survey did not include these items.
We sought public comment on the proposed changes in October and November 2014, and received 635 comments—the majority (88%) voted in support the proposed changes [
15].
3. Results
A total of 268 practices submitted data on the adult survey and 58 practices submitted data on the child survey. The mean number of respondents per practice was 104 for the adult survey and 74 for the child survey. The overall response rate was 27% for adults and 23% for children. Respondent characteristics are presented in
Table 1. For the adult survey, the majority of respondents were female (61%) and aged 55–64 years (25%). Most self-rated their general health as good (36%) and their mental health as very good (35%). For the child survey (filled out by the child’s parent or guardian), the majority of respondents were also female (89%). Parental ratings of child health on the child survey were better overall than self-rated health on the adult survey, with excellent general and mental health ratings of 57% and 56%, respectively, for the child sample. The majority of practices, for both adult and child samples, were comprised of multiple providers (four or more), with ownership under a hospital, health system, or health plan (rather than physician owned) and located in the Northeast census region. Below we describe key results for the current PCMH composites and items for both adults (
Table 2) and children (
Table 3), as well as the impact of shortening the survey (
Table 4 and
Figure 1,
Figure 2,
Figure 3,
Figure 4,
Figure 5 and
Figure 6).
Table 2 and
Table 3 indicate, in italics, all items retained for the shortened survey.
Table 1.
Characteristics of adult survey respondents and children who were the patients asked about in child surveys, CAHPS PCMH 1, 2013.
Table 1.
Characteristics of adult survey respondents and children who were the patients asked about in child surveys, CAHPS PCMH 1, 2013.
Characteristic | Category | Respondents for Adult Survey, n = 268 Practices (27,896 Individuals) | Children Asked about, n = 58 Practices (4277 Individuals) |
---|
N | Percent | N | Percent |
---|
Age | <18 years | 0 | 0 | 3341 | 99% |
| 18–24 years | 947 | 4% | 26 | 1% |
| 25–34 years | 1936 | 7% | - | - |
| 35–44 years | 2540 | 9% | - | - |
| 45–54 years | 4808 | 18% | - | - |
| 55–64 years | 6764 | 25% | - | - |
| 65–74 years | 5823 | 21% | - | - |
| ≥75 years | 4550 | 17% | - | - |
Gender | Male | 10,682 | 39% | 435 | 11% |
| Female | 16,697 | 61% | 3683 | 89% |
Ethnicity | Hispanic | 1729 | 7% | 318 | 8% |
| Not Hispanic | 24,685 | 94% | 3783 | 92% |
General Health | Excellent | 3638 | 13% | 2393 | 57% |
| Very Good | 9485 | 35% | 1383 | 33% |
| Good | 9773 | 36% | 340 | 8% |
| Fair | 3761 | 14% | 51 | 1% |
| Poor | 717 | 3% | 6 | 0% |
Mental Health | Excellent | 6990 | 26% | 2319 | 56% |
| Very Good | 9533 | 35% | 1217 | 29% |
| Good | 7489 | 27% | 485 | 12% |
| Fair | 2752 | 10% | 135 | 3% |
| Poor | 545 | 2% | 19 | 1% |
No. of visits past year | 1 | 6147 | 23% | 1221 | 30% |
| 2 | 7269 | 27% | 1151 | 28% |
| 3 | 5038 | 19% | 760 | 18% |
| 4 | 3880 | 15% | 434 | 11% |
| 5–9 | 3325 | 13% | 500 | 12% |
| ≥10 | 981 | 4% | 64 | 2% |
3.1. Internal Consistency Reliabilities
The majority of multi-item composites formed an internally consistent scale in current versions of both adult (
Table 2) and child surveys (
Table 3)—with four composites meeting the recommended standard of a 0.70 or higher Cronbach’s α: communication with providers, six items (Cronbach’s α = 0.92 adult and 0.91 child); office staff interaction, two items (0.84 adult and 0.85 child); access to care, five items (0.81 adult; 0.70 child); and comprehensiveness of behavioral care, three items (0.79 adult-only composite). Only two composites did not achieve the 0.70 level: self-management support (0.66 adult; 0.60 child) and shared decision making (0.65 adult-only composite).
Table 2.
2013 results for the adult CAHPS PCMH Survey (n = 268 practices). Items in italics recommended for revised survey.
Table 2.
2013 results for the adult CAHPS PCMH Survey (n = 268 practices). Items in italics recommended for revised survey.
Item | Item # | Core or PCMH 1 | Response Set 2 | Correlation with Total (Composite) 3 | Composite | Practice Level Reliability 5 |
---|
With Original Composite | With Revised Composite | Mean 4 | SD |
---|
Rating of provider | 32 | Core | 0–10 | - | - | 89.47 | 4.20 | 0.80 |
Access (5 items original, 2 items revised) | | | | | | 76.02 | 9.69 | 0.94 |
Got appointment for urgent care as soon as needed | 6 | Core | N-A | 0.74 | 0.74 | 82.97 | 9.34 | 0.84 |
Got appointment for check-up or routine care as soon as needed | 9 | Core | N-A | 0.76 | - | 85.20 | 7.15 | 0.87 |
Got answer to medical question the same day you phoned | 14 | Core | N-A | 0.68 | 0.74 | 79.24 | 8.96 | 0.75 |
Got answer to medical question as soon as you needed when phoned after hours | 16 | Core | N-A | 0.33 | - | 73.31 | 19.05 | 0.45 |
Saw provider within 15 min of appointment time | 18 | Core | N-A | 0.57 | - | 65.56 | 14.79 | 0.95 |
Items not scored in composite | | | | | | | | |
Days you had to wait for an appointment for urgent care | 7 | PCMH | 0–7 Days | - | - | 25.52 | 12.65 | 0.90 |
Got needed care on evenings, weekends, or holidays | 12 | PCMH | N-A | - | - | 48.76 | 19.99 | 0.67 |
Information (2 items, not scored as a composite) | | | | | | | | |
Got information about what to do if you needed care on evenings, weekends, or holidays | 10 | PCMH | Y-N^ | - | - | 70.64 | 10.56 | 0.78 |
Received reminders between visits | 17 | PCMH | Y-N | - | - | 66.25 | 11.83 | 0.79 |
Communication (6 items original, 2 items revised) | | | | | | 91.24 | 4.15 | 0.82 |
Provider explained things in a way that was easy to understand | 19 | Core | N-A | 0.92 | 0.79 | 92.68 | 4.00 | 0.78 |
Provider listened carefully | 20 | Core | N-A | 0..93 | - | 92.33 | 3.96 | 0.73 |
Provider gave easy to understand instructions about taking care of health problems or concerns | 22 | Core | N-A | 0.92 | - | 91.23 | 4.38 | 0.71 |
Provider seemed to know important information about your medical history | 23 | Core | N-A | 0.83 | 0.79 | 87.53 | 5.61 | 0.81 |
Provider respected what you had to say | 24 | Core | N-A | 0.91 | - | 93.85 | 3.78 | 0.74 |
Provider spent enough time with you | 25 | Core | N-A | 0.87 | - | 90.41 | 4.85 | 0.80 |
Coordination of Care (3 items, not scored as a composite) | | | | | | | | |
Provider’s office followed up to give you results of blood test, X-ray, or other test | 27 | Core | N-A | - | - | 82.46 | 9.08 | 0.85 |
Provider seemed informed and up-to-date about care you got from specialists | 34 | PCMH | N-A | - | - | 79.53 | 7.32 | 0.66 |
Talked with you about your prescriptions | 38 | PCMH | N-A | - | - | 84.56 | 9.50 | 0.82 |
Comprehensiveness-Behavioral/whole person (3 items original, single item revised) | | | | | | 43.82 | 13.23 | 0.91 |
Talked about personal or family problem/alcohol or drug use | 39 | PCMH | Y-N | 0.86 | - | 45.94 | 16.11 | 0.92 |
Talked about worry and stress in your life | 40 | PCMH | Y-N | 0.89 | - | 49.35 | 14.33 | 0.89 |
Talked about feeling sad or depressed | 41 | PCMH | Y-N | 0.87 | - | 35.98 | 11.74 | 0.83 |
Self-Management Support (2 items original & revised) | | | | | | 45.45 | 10.52 | 0.81 |
Work with you to set specific goals for your health | 35 | PCMH | Y-N | 0.78 | 0.78 | 56.09 | 11.81 | 0.78 |
Ask if there are things make it hard to take care of your health | 36 | PCMH | Y-N | 0.78 | 0.78 | 34.53 | 10.57 | 0.75 |
Shared Decision Making (3 items) | | | | | | 79.51 | 6.46 | 0.58 |
Provider talked about reasons to take a medicine | 29 | PCMH | Not-A lot | 0.68 | - | 85.56 | 4.62 | 0.39 |
Provider talked about reasons not to take a medicine | 30 | PCMH | Not-A lot | 0.73 | - | 71.42 | 7.96 | 0.49 |
Provider asked what you thought was best for you regarding medicine | 31 | PCMH | Y-N | 0.65 | - | 81.56 | 9.66 | 0.52 |
Office Staff (2 items) | | | | | | 87.43 | 5.78 | 0.87 |
Office staff at this office were as helpful as you though they should be | 42 | Core | N-A | 0.92 | - | 84.09 | 6.68 | 0.86 |
Office staff at this office treated you with courtesy and respect | 43 | Core | N-A | 0.92 | - | 90.82 | 5.12 | 0.84 |
Table 3.
2013 results for the child CAHPS PCMH Survey (n = 58 practices). Items in italics recommended for revised survey.
Table 3.
2013 results for the child CAHPS PCMH Survey (n = 58 practices). Items in italics recommended for revised survey.
Item | Item # | Core or PCMH 1 | Response Set 2 | Correlation with Total (Composite) 3 | Composite | Practice Level Reliability 5 |
---|
With Original Composite | With Revised Composite | Mean 4 | SD |
---|
Rating of provider | 35 | Core | 0–10 | | | 89.47 | 4.21 | 0.80 |
Access (5 items original, 2 items revised) | | | | | | 80.53 | 7.29 | 0.88 |
Got appointment for urgent care as soon as needed | 13 | Core | N-A | 0.61 | 0.61 | 90.06 | 5.62 | 0.69 |
Got appointment for check-up or routine care as soon as needed | 16 | Core | N-A | 0.51 | -- | 87.02 | 6.10 | 0.75 |
Got answer to medical question the same day you phoned provider’s office | 21 | Core | N-A | 0.54 | 0.61 | 90.18 | 5.99 | 0.70 |
Got answer to medical question as soon as you needed when phoned provider’s office after hours | 23 | Core | N-A | 0.40 | -- | 87.41 | 10.47 | 0.42 |
Saw provider within 15 min of appointment time | 25 | Core | N-A | 0.60 | -- | 67.64 | 11.59 | 0.88 |
Items not scored in composite | | | | | | | | |
Days you had to wait for an appointment for urgent care | 14 | PCMH | 0–7 Days | -- | -- | 91.42 | 6.45 | 0.83 |
Got needed care on evenings, weekends, or holidays | 19 | PCMH | N-A | -- | -- | 69.37 | 15.71 | 0.71 |
Information (2 items, not scored as a composite) | | | | | | | | |
Got information about what to do if you needed care on evenings, weekends, or holidays | 17 | PCMH | Y-N | -- | -- | 81.02 | 9.08 | 0.72 |
Received reminders between visits | 24 | PCMH | Y-N | -- | -- | 58.24 | 15.00 | 0.85 |
Communication (6 items original, 2 items revised) | | | | | | 93.29 | 3.78 | 0.70 |
Provider explained things in a way that was easy to understand | 26 | Core | N-A | 0.89 | 0.86 | 94.94 | 3.69 | 0.66 |
Provider listened carefully | 27 | Core | N-A | 0.93 | -- | 94.41 | 3.22 | 0.51 |
Provider gave easy to understand instructions about taking care of health problems or concerns | 29 | Core | N-A | 0.77 | -- | 94.00 | 3.82 | 0.52 |
Provider seemed to know important information about your medical history | 30 | Core | N-A | 0.90 | 0.86 | 89.88 | 5.83 | 0.74 |
Provider respected what you had to say | 31 | Core | N-A | 0.92 | -- | 94.96 | 3.10 | 0.49 |
Provider spent enough time with you | 32 | Core | N-A | 0.89 | -- | 92.54 | 4.01 | 0.58 |
Coordination of care (3 items, not scored as composite) | | | | | | | | |
Provider’s office followed up to give you results of blood test, X-ray, or other test | 34 | Core | N-A | -- | -- | 84.88 | 9.80 | 0.48 |
Provider seemed informed and up-to-date about care you got from specialists | 37 | PCMH | N-A | -- | -- | 77.59 | 10.69 | 0.20 |
Talked with you about your prescriptions | 52 | PCMH | N-A | -- | -- | 87.63 | 8.75 | 0.5 |
Comprehensiveness-Child Development (6 items original, single item revised) | | | | | | 61.79 | 12.08 | 0.87 |
Talked about child’s learning ability | 38 | Core | Y-N | 0.85 | - | 47.11 | 12.58 | 0.73 |
Talked about behaviors that are normal for child at this age | 39 | Core | Y-N | 0.85 | - | 68.79 | 14.06 | 0.83 |
Talked about how your child’s body is growing | 40 | Core | Y-N | 0.62 | - | 79.67 | 12.26 | 0.81 |
Talked about child’s moods and emotions | 41 | Core | Y-N | 0.87 | - | 60.09 | 14.24 | 0.81 |
Talked about how much time child spends in front of a computer/TV | 44 | PCMH | Y-N | 0.78 | - | 49.96 | 19.26 | 0.91 |
Talked about how child gets along with others | 47 | Core | Y-N | 0.78 | - | 53.39 | 16.26 | 0.85 |
Comprehensiveness-Child Prevention (5 items original, 2 items revised) | | | | | | 58.84 | 13.76 | 0.88 |
Talked about things to do to keep child from getting injured | 42 | Core | Y-N | 0.82 | - | 58.14 | 14.49 | 0.81 |
Given information about keeping child from getting injured | 43 | Core | Y-N | 0.77 | - | 51.45 | 16.35 | 0.83 |
Talked about food your child eats | 45 | Core | Y-N | 0.82 | - | 78.68 | 13.89 | 0.85 |
Talked about exercise your child gets | 46 | Core | Y-N | 0.75 | 0.62 | 67.20 | 13.55 | 0.80 |
Talked about if there were problems in household that might affect child | 48 | Core | Y-N | 0.73 | 0.62 | 46.18 | 18.20 | 0.88 |
Self-Management Support (2 items original & revised) | | | | | | 33.43 | 10.65 | 0.72 |
Work with you to set specific goals for your health | 49 | PCMH | Y-N | 0.82 | - | 45.43 | 13.13 | 0.68 |
Ask you if there are things that make it hard for you to take care of your health | 50 | PCMH | Y-N | 0.82 | - | 21.29 | 9.14 | 0.65 |
Office Staff (2 items) | | | | | | 87.43 | 5.67 | 0.84 |
Office Staff at this office were as helpful as you though they should be | 53 | Core | N-A | 0.90 | - | 84.92 | 6.36 | 0.82 |
Office Staffs at this office treated you with courtesy and respect | 54 | Core | N-A | 0.90 | - | 89.93 | 5.29 | 0.82 |
Table 4.
Comparison of original composites and revised composites: Adult (n = 268 practices) and child (n = 58 practices) CAHPS PCMH Survey, 2013. (Items in italics recommended for revised survey).
Table 4.
Comparison of original composites and revised composites: Adult (n = 268 practices) and child (n = 58 practices) CAHPS PCMH Survey, 2013. (Items in italics recommended for revised survey).
Item | # of Items | Mean | SD | Internal Consistency Reliability 1 | Practice Level Reliability | Number of Responses per Practice Needed to Achieve 0.70, 0.80, and 0.90 Reliability 2 |
---|
Reliability = 0.70 | Reliability = 0.80 | Reliability = 0.90 |
---|
Adult Survey | | | | | | | | |
Original Access | 5 | 76.02 | 9.69 | 0.81 | 0.94 | 14 | 24 | 55 |
Revised Access | 2 | 81.50 | 8.41 | 0.67 | 0.85 | 26 | 45 | 101 |
Original Communication | 6 | 91.24 | 4.15 | 0.92 | 0.82 | 52 | 89 | 200 |
Revised Communication | 2 | 90.11 | 4.55 | 0.72 | 0.82 | 50 | 85 | 191 |
Shared Decision Making | 3 | 79.51 | 6.46 | 0.65 | 0.58 | 84 | 144 | 324 |
Self-Management Support | 2 | 45.45 | 10.52 | 0.66 | 0.81 | 55 | 94 | 211 |
Comprehensiveness-Behavioral | 3 | 43.82 | 13.23 | 0.79 | 0.91 | 21 | 37 | 83 |
Revised Comprehensiveness-Adult Behavioral (Single item: Talked about worry and stress in your life) | 1 | 49.35 | 14.33 | NA | 0.89 | 29 | 50 | 113 |
Office Staff | 2 | 87.43 | 5.78 | 0.84 | 0.87 | 35 | 59 | 134 |
Child Survey | | | | | | | | |
Original Access | 5 | 80.53 | 7.29 | 0.70 | 0.88 | 22 | 37 | 84 |
Revised Access | 2 | 90.03 | 5.27 | 0.56 | 0.77 | 36 | 62 | 139 |
Original Communication | 6 | 93.29 | 3.78 | 0.91 | 0.70 | 71 | 121 | 272 |
Revised Communication | 2 | 92.40 | 4.54 | 0.68 | 0.75 | 55 | 94 | 211 |
Comprehensiveness-Child Development | 6 | 61.79 | 12.08 | 0.81 | 0.87 | 18 | 32 | 71 |
Revised Comprehensiveness-Child Development (Single item: Talked about behaviors that are normal for child at this age) | 1 | 68.79 | 14.06 | NA | 0.83 | 34 | 58 | 129 |
Comprehensiveness-Child Prevention | 5 | 58.84 | 13.76 | 0.81 | 0.88 | 17 | 30 | 67 |
Revised Comprehensiveness-Child Prevention | 2 | 56.65 | 14.35 | 0.59 | 0.88 | 21 | 37 | 83 |
Self-Management Support | 2 | 33.43 | 10.65 | 0.60 | 0.72 | 61 | 104 | 234 |
Office Staff | 2 | 87.43 | 5.67 | 0.85 | 0.84 | 31 | 53 | 120 |
Reducing the number of items in existing composites generally led to reductions in internal consistency reliability (
Table 4). For the access composite for adults, reducing from five to two items led to reduction in internal consistency reliability from 0.81 to 0.67. For the communication composite in adults, reducing from six to two items changed the internal consistency reliability from 0.92 to 0.72. These findings are to be expected since the Cronbach’s alpha increases as the number of items in a scale increases. Only the internal consistency of the access item for adults (0.67) fell below the recommended level of 0.70.
These patterns were also generally found in the child results (
Table 4). However, three child composites fell below the recommended internal consistency level of 0.70 when revised: access (0.56 for the two-item scale), communication (0.68 for the two-item scale), and comprehensiveness in child prevention (0.59 for the two-item scale).
Figure 1.
Change in practices’ quintile ranking from original to revised adult composites, CAHPS PCMH 2013 (n = 268 practices).
Figure 1.
Change in practices’ quintile ranking from original to revised adult composites, CAHPS PCMH 2013 (n = 268 practices).
Figure 2.
Change in practices’ quintile ranking from original to revised child composites, CAHPS PCMH 2013 (n = 58 practices).
Figure 2.
Change in practices’ quintile ranking from original to revised child composites, CAHPS PCMH 2013 (n = 58 practices).
Figure 3.
Practice ranking, revised vs. original adult access composite, CAHPS PCMH 2013 (n = 268 practices).
Figure 3.
Practice ranking, revised vs. original adult access composite, CAHPS PCMH 2013 (n = 268 practices).
Figure 4.
Practice ranking, revised vs. original adult communication composite, CAHPS PCMH 2013 (n = 268 practices).
Figure 4.
Practice ranking, revised vs. original adult communication composite, CAHPS PCMH 2013 (n = 268 practices).
Figure 5.
Practice ranking, revised vs. original child access composite, CAHPS PCMH 2013 (n = 58 practices).
Figure 5.
Practice ranking, revised vs. original child access composite, CAHPS PCMH 2013 (n = 58 practices).
Figure 6.
Practice ranking, revised vs. original child communication composite, CAHPS PCMH 2013 (n = 58 practices).
Figure 6.
Practice ranking, revised vs. original child communication composite, CAHPS PCMH 2013 (n = 58 practices).
3.2. Practice Level Reliabilities
Practice-level reliabilities achieved the recommended level of 0.70 or higher for most current versions of multi-item composites in the adult (
Table 2) and child (
Table 3) surveys, with the exception of the shared decision making composite (0.58).
Reducing the number of items in existing composites led to reductions in practice level reliability for only the access composite (
Table 4). Reducing the access composite from five to two items led to reduction in practice level reliability from 0.94 to 0.85 for adults, and 0.88 to 0.77 for children. There was no reduction in practice level reliability for the communication composite in adults or children (e.g., 0.82 for both the six-item and two-item scale in adults), nor for the child comprehensiveness of preventive care composite (0.88 for both the five-item and two-item scale).
3.3. Item to Scale Correlations and Unadjusted Mean Scores
Item-scale correlations (
Table 2 and
Table 3) provided support for the reduced composites (all correlations achieving levels of 0.50 or higher, and correlating about as highly as many items in the original), with some exceptions. For the adult communication and the child comprehensiveness of preventive care composite, items in the reduced two-item composite correlated more weakly than the lowest correlation in the original multi-item composites (e.g., for adult communication: 0.79 for the two-item scale and 0.83 for the weakest correlation in the six-item scale; for child comprehensiveness of preventive care: 0.62 for the two-item scale and 0.73 for the weakest correlation in the five-item scale). However, these item-scale correlations still achieved 0.50. We also note that the correlations for two-item scales should not be interpreted as correlations with a true “scale” as they relate one item to only one other item.
Unadjusted mean scores were also generally stable with the reductions; the largest difference being a five-point score improvement for the adult access composite (76.0 for the original six-item scale to 81.5 for the short two-item scale).
3.4. Responses Estimated to Obtain Site-level Reliabilities of 0.70, 0.80, 0.90
The number of estimated responses per practice needed to achieve reliabilities of 0.70, 0.80, and 0.90 are presented in
Table 4. For the adult survey, the number of responses for a reliability of 0.70 ranged from 14 to 84, with a higher minimum number of responses needed to achieve the same reliability in the revised composites (minimum of 26, maximum of 50). For the child survey, the number of responses for a reliability of 0.70 ranged from 17 to 71, with a higher minimum number of responses needed to achieve the same reliability in the revised composites (minimum of 21, maximum of 55).
3.5. Relative Ranking of Practices under Revised Survey Composites
Shortening both the adult and child composites for access and communication resulted in more changes in the relative ranking of practices for the access composite compared to the communication composite. Specifically, for the communication composite, results from the quintile analysis showed that 74% of adult practices did not change rank while 25% changed one quintile rank (
Figure 1); 66% of child practices did not change quintile rank while 35% changed one quintile rank (
Figure 2). For the access composite, however, there were more changes based on quintile ranks: 51% of adult practices did not change rank while 40% changed one quintile rank (
Figure 1); 52% of child practices did not change rank while 31% changed one quintile rank (
Figure 2).
Results from the (full) rank order correlation analysis were consistent with results from the quintile ranking analysis. Specifically, for the communication composite, long and short versions of the composite resulted in similar practice rankings for both the adult (
r = 0.97,
p < 0.001) and child (
r = 0.96,
p < 0.001) versions of the composites (
Figure 4 and
Figure 6). For the access composite, long and short versions of the composite also resulted in more changes in rankings for the adult (
r = 0.83,
p < 0.001) and child (
r = 0.76,
p < 0.001) composites (
Figure 3 and
Figure 5) compared to the communication composite—although these still meet common recommended levels of 0.70 or higher for a strong, positive correlation.
4. Discussion
This study provides further support for the reliability and validity of the current CAHPS PCMH survey, based on updated data across a larger sample, and characterizes the psychometric impact of shortening the survey. Importantly, our findings suggest that a shorter adult survey is possible. Unadjusted mean scores were also generally stable with the reduction; the largest difference being a five-point score improvement for the adult access composite. For both the adult and child surveys, the reductions did result in more changes in the relative ranking of practices for the access composite, compared to the communication composite.
In general, internal consistency reliability for the multi-item composites exceeded or equaled original published field test results [
3]. Estimates of site-level reliability also indicate that a reliability of 0.70 or higher can generally be achieved for most multi-item composites. However, borderline site-level reliability among select composites and items suggest that, despite their salience to the PCMH care model, these items and composites may be considered for removal to streamline the survey and its effectiveness and uptake. Previous research by the CAHPS Consortium suggests that survey length generally does not affect survey response rates, with prior findings suggesting that the number of survey questions that respondents were required to answer, from as few as 23 to as many as 95, had little effect on response rates and respondents were as likely to answer a relatively longer survey as a shorter one [
16]. However, recent input from a diverse group of stakeholders under NCQA’s PCMH recognition program have suggested a need to consider shortening the survey in order to increase response rates. Both NCQA and the CAHPS Consortium have conducted research to re-evaluate the PCMH survey, and, as of the time of this present study, have each put forth their own proposals for changes to the survey [
11,
15], with the CAHPS Consortium finalizing their revised version of the CAHPS Clinician and Group survey (version 3.0) in July 2015, reducing the length from 34 to 31 items [
12]. Additional possibilities for shortening the CAHPS Clinician and Group survey have also since been published [
13].
The results here suggest that reduction in length are possible; despite some reduction in psychometric properties, the reduced adult survey would still generally meet standard definitions of a psychometrically sound survey; however, given three child composites fell below the recommended internal consistency level of 0.70 when revised, further testing is recommended to establish appropriate criteria for shortening the child survey. For example, further work could investigate whether internal consistency reliability suffered because these composites may not have reflected “true” scales, whether the smaller child survey samples may have influenced practice-level reliability—which in turn influenced item-level results and decisions to drop item, or whether there may be something else altogether beyond these psychometric concerns—such as the possibility of more variability in the kind of care pediatric populations require.
Additionally, while a shorter survey addresses ongoing concerns about survey length, further work should also investigate related issues of survey response and uptake, including whether a shorter survey facilitates meaningful improvement in response rates, or facilitates opportunities for customization of the survey to fit practice needs. Input from consumers and families about the relevance of these measures for decision-making as well as practice input on the usefulness in quality improvement are also key considerations.
In recommending any further potential changes to shorten the survey, several overarching principles should be taken into account, including some of those used in the current study. First, any reduction needs to be weighed not only against its impact on psychometric attributes, but also against goals for survey use. During stakeholder discussions and public comment fielding, many indicated the importance of having a shorter survey that meets a mix of both accountability and quality improvement needs. One useful principle already used in the current study is to consider whether an item is actionable, which speaks to its usefulness from both an accountability and quality improvement perspective.
Second, reductions may need to be considered for only certain composites versus all composites. In the current study, some reductions achieved higher internal consistency reliability than others, begging the question of whether a broad approach of shortening all composites is too “blunt”, and if reductions should instead be customized to only some portions of the survey.
Finally, relevant to these concept of customization, any survey change needs to consider the increasing attention towards flexibility. Although there was overwhelming support for shortening the survey, there were also diverse opinions regarding which items should be dropped. Given the CAHPS Consortium, NCQA and other groups (including the Massachusetts Health Quality Partners) have each proposed slightly different approaches for shortening the survey, this begs the further question of whether the route to a shorter survey should emphasize not so much the selection of specific items, but rather the creation of a flexible route to assessment. The literature has already begun to acknowledge the need to strike this balance, calling for patient surveys, such as the CAHPS surveys, to allow for variation, while retaining common core elements as a “foundation” to facilitate alignment and standardization [
17].
This study had some limitations. First, response rates were lower than seen in some other surveys, although they are similar to response rates in some implementations of CAHPS surveys [
3]. While a low response rate may not have affected the psychometric results presented in this study, this is an important limitation. As we were unable to examine differences between non-responders and responders, the study results must be interpreted with caution and may not be generalizable. Second, the majority of practices were from the northeast area, which also affects the generalizability of our results. However, unlike prior published findings of the CAHPS PCMH survey, practices from most major census regions (west, midwest, northeast), except the south, submitted data. Despite these limitations, this study provides important information on the psychometric impact of shortening the survey, and opens up possibilities for assessing patient experiences in medical home settings where survey length or burden may be a concern.
As PCMH adoption expands, the ability to evaluate the PCMH promise of improving patient experiences and other aspects of care remains essential. The current literature acknowledges that more evidence is generally needed to determine the effects of the PCMH on select outcomes [
2]. Given the concerns around survey length, opportunities to shorten the CAHPS PCMH survey would complement current measurement efforts to evaluate PCMH settings. Further research should address and further delineate the approaches needed to ensure that the CAHPS PCMH plays a useful role in optimizing patient experiences in PCMH and other efforts to reform the health system, whether it is investigating approaches to improving survey response or uptake, the relevance of survey items and composites to inform quality improvement, or the incorporation of new methods to efficiently assess priority domains, while retaining opportunities for shortening and customizing the survey.