Using the Nursing Culture Assessment Tool (NCAT) in Long-Term Care: An Update on Psychometrics and Scoring Standardization

An effective workforce performing within the context of a positive cultural environment is central to a healthcare organization’s ability to achieve quality outcomes. The Nursing Culture Assessment Tool (NCAT) provides nurses with a valid and reliable tool that captures the general aspects of nursing culture. This study extends earlier work confirming the tool’s construct validity and dimensionality by standardizing the scoring approach and establishing norm-referenced scoring. Scoring standardization provides a reliable point of comparison for NCAT users. NCAT assessments support nursing’s ability to evaluate nursing culture, use results to shape the culture into one that supports change, and advance nursing’s best practices and care outcomes. Registered nurses, licensed practical nurses, and certified nursing assistants from 54 long-term care facilities in Kentucky, Nevada, North Carolina, and Oregon were surveyed. Confirmatory factor analysis yielded six first order factors forming the NCAT’s subscales (Expectations, Behaviors, Teamwork, Communication, Satisfaction, Commitment) (Comparative Fit Index 0.93) and a second order factor—The Total Culture Score. Aggregated facility level comparisons of observed group variance with expected random variance using rwg(J) statistics is presented. Normative scores and cumulative rank percentages and how the NCAT can be used in implementing planned change are provided.

study takes the NCAT to its next level of development by reconfirming the tool's construct validity and dimensionality in a larger sample and by standardizing the scoring approach through norm-referenced scoring. NCAT users can now compare LTC clinical unit or facility assessment results to standardized scores providing a common perspective from which to use assessment results to evaluate and shape the culture into one that could better support project success when implementing planned change, thereby advancing nursing's best practices and care outcomes. We report on the results scoring standardization process and offer insights about how the NCAT can be used by LTC leaders to conduct a baseline assessment of the nursing culture and design and implement intervention strategies aimed at bringing about a positive nursing culture prior to introducing change in the workplace.

Design
A cross-sectional design and a survey approach was used to collect demographics and survey data. Confirmatory factor analysis of the NCAT's items and determination of percentile scores were conducted to establish a norm-referenced approach to NCAT scoring and interpretation. This study builds on the already established face and content validity of the 19-item NCAT by analyzing the performance of the measure when completed by a representative sample of the RNs, LPNs, and CNAs working in LTC facilities.

Setting and Participants
LTC facilities (n = 54) from a convenience sample in urban and rural areas of North Carolina, Kentucky, Nevada, and Oregon participated. Researchers contacted the facility director of nursing for permission to include the facility in the study. All willing and consented part-time or full-time nursing staff (RN, LPN, and CNA) with the ability to read and understand English and who had worked at the respective facility for 3 or more months were eligible to participate in the study. Staff ages (≥18-years-old) and staff of all races, ethnicities, physical ability, and gender were eligible for inclusion. The norming process was implemented through administration of the NCAT to at least 5 nursing staff (RNs, LPNs, and CNAs) at each facility. Before agreeing to participate in this study, participants were informed of the purpose, procedures, risks, and potential benefits of the study through a statement of informed consent. The University of North Carolina Charlotte Institutional Review Board approved the study prior to data collection.

Power Analysis
Based on a minimum sample of 300 nurse and CNA survey respondents, there was 80% power to demonstrate that the Cronbach's alpha (α) statistic calculated for the NCAT and its subscales or a correlation of the NCAT or its subscales with the IES-R is significantly different from zero if any of these statistics exceed 0.14 and 90% power for statistics that exceed 0.17. The precision of an α statistic, as measured by its 95% confidence interval, that is in the 0.70 to 0.90 range is approximately ±0.04 to ±0.01, indicating that 95% of the time we would not overestimate a (population value of) Cronbach's α of less than 0.66 as greater than 0.7 with a sample size of 300.

Measurement-Survey Instrument
The survey instrument was composed of demographic and NCAT items. An investigator developed demographic data collection tool was used to ascertain, age, gender, type of licensed/certified job role, years in current position, and LTC facility name from each participant. The NCAT [19,20] is a 19 item questionnaire composed of declarative statements about nursing and culture in the work setting. Respondents were instructed to rate each item from his/her own perceptions about the healthcare work environment in which currently employed. Each respondent was asked to assign a rating of each of the items according to an ordinal scale, including Strongly Disagree (1); Disagree (2); Agree (3); and Strongly Agree (4). Six core dimensions were hypothesized as being represented within the 19 items. These six subscales were proposed as Expectations (Items 1-3), Behavior (Items 4-6), Teamwork (Items 7-10), Communication (Items 11-13), Satisfaction (Items 14, 15), and Professional Commitment (Items 16-19).

Data Collection/Methods/Procedures
LTC facilities were identified from the Centers for Medicare and Medicaid list of Medicare certified LTC facilities. This list is publically available from www.medicare.gov/nursinghomecompare data regarding the facility size, 5 star rating, and quality measures, such as staff care hours per resident per day were collected. Each facility's director of nursing was contacted by phone and followed up with an email including a description of the study and procedures and a consent form for signature that formally requested the facility's participation. The study was explained, sample study materials provided, and permission requested to distribute study materials to nursing staff. Once the facility administrator had agreed to participation, a time was determined for administering the survey to staff and a staff recruitment flier was posted at the facility. Data were collected via either printed survey or electronically from study participants using Qualtrics online software (Qualtrics, Provo, UT, USA, 2014). All data were combined into a single Excel database for statistical analyses. Use of a subject identification number permitted grouping of staff participants by the respective facility.

Data Analysis
Data were transferred into SPSS or M-Plus for data analysis. NCAT scores were calculated for each item, subscale, and total score by assigning the numerical value to the answer marked for each item (strongly disagree = 1, disagree = 2, agree = 3, strongly agree = 4). Reverse scoring is not indicated for any item, so subscale and total score calculation was achieved by a simple summing process. Statistical analysis included analysis of demographic data, psychometric evaluation of the NCAT, and norming procedures to obtain percentile ranks. Multigroup Equivalence/Invariance analysis (ME/I) was used to evaluate whether the factor structure of the culture measure was the same in the data collection groups. The psychometric evaluation of the NCAT included an examination of Cronbach's Alpha reliability and factor structure using Confirmatory Factor Analyses (CFA). The chi-square statistics comparative fit index (CFI), the root mean-square error of approximation (RMSEA) were used as indicators of first and second order factors. Structural analyses of item functioning, including item cross-loadings and inter-item correlations were also examined. Aggregation from individual level responses to facility level scores was supported by examination of the rwg(J) index of agreement.

Results and Discussion
Empirical construct validity and dimensionality of the NCAT tool were confirmed and norm-referenced scoring was established with a total of 1025 predominately female nursing staff (204 RNs, 243 LPNs, and 578 CNAs) from 54 LTC facilities participating in the study. The mean LTC facility size was 117 Medicare and Medicaid certified beds with a mean of 99 total residents and mean 5 Star rating of 3 (range = 1 to 5). Mean nursing hours per resident day for RN, LPN, and CNA care were 46, 49, and 139, respectively.

NCAT's Dimensionality Supported
We examined the fit of the data to a hypothesized model that included items as indicators of six first order factors (the subscales) and a second-order factor representing Overall Culture. The model was found to fit the data well (CFI = 0.932; RMSEA = 0.078; SRMR = 0.048) ( Table 1). The standardized loadings for the first-order factor ranged from 0.631 to 0.912 and the loadings for the second-order factor ranged from 0.398 to 0.965 (Table 2). Cronbach's alpha coefficients of items in each subscale and in the overall scale ranged from 0.71 to 0.94. The nursing staff's mean NCAT Total score was 59.21 with the full range of NAT scores from 19 to 76 represented. Mean subscale scores were Expectations 9.32 (3 items), Behavior 9.31 (3 items), Communication 9.22 (3 items), Teamwork 11.22 (4 items), Satisfaction 5.75 (2 items), and Commitment 14.37 (4 items). Standardized loadings for each item ranged from 0.631 to 0.912 (Table 2). Pearson product-moment correlation coefficients between each pair of NCAT subscales was at a medium (≥0.05) to high (≥0.8) level ranging from 0.60 to 0.71 for all subscales except for Commitment (r = 0.30 to 0.335; low <0.5), which focuses on perceptions of self and professional commitment rather than relationships with other staff in the facility. The correlation between Commitment and the NCAT Total Score is 0.553.

NCAT's Normative Scores
Normative results are based on the aggregation of data to the facility level (n = 54). Using rwg(J) statistics, aggregated facility level comparisons showed acceptable ≥0.50 values for each scale supporting the inclusions of all 54 facilities in the aggregation. Normative scores for the overall Nursing Culture Total Score (summative for all 6 subscales) are presented in Table 3. The tabled values indicate the percentage of 54 LTC facilities with scores < that level. For example, at a score of 69, 98.2% of sites had scores at or below 69. Similarly, Cumulative Rank Percentages are listed in Table 4 for each subscale. A Total subscale score of 7 for Expectations shows that 1.8% of all sites had scores of <7 for Expectations. The percentages in Tables 4 and 5 can be applied by NCAT users to compare how their respective NCAT results compare to those of other LTC sites.

Using the NCAT to Guide Change
Positive work and care environments provide a safe, respectful, and civil atmosphere in which to implement job responsibilities. A "negative healthcare work environment" has been consistently cited by licensed nurses as being among the top reasons difficulty is encountered in recruiting and retaining licensed nurses in the workforce [4]. Similarly, difficulty recruiting and retaining CNAs occurs in most states [11]. Literature shows stabilized, lower turnover rates may be linked to effectual cultures [7]. In long-term care organizations, culture is thought to be a strong factor in determining the organization's ability to achieve Quality Measures and Quality Indicators (QI), such as prevention of pressure ulcers and falls. In fact, one QI is staff turnover rate, which is believed to be a key disruptor of culture and a strong influence on quality and environment. Disenchanted staff will often generate sub-cultures that deepen dissatisfaction and contribute to high turnover rates [5]. In contrast, a positive culture leads to teamwork, safe behavior, satisfaction, commitment, and good communication [22]; therefore, leading to decreased stress, thereby reducing the impact risks both short and long-term on the ability of licensed nurses and CNAs to safely do their jobs. The NCAT is a valid and reliable nursing culture assessment tool that nurse leaders can use in first assessing the nursing work culture. This study's results specific to scoring and score interpretation offer a common point for comparison and could lead to changes in workplace environments that will be effective in improving care outcomes.
The NCAT's normative subscale scores and cumulative rank percentages can be used in interpreting baseline nursing culture assessment and periodic reassessment results when planning change and implementing improvements in care practices. Table 5 presents seven core steps that we believe are foundational for NCAT users to include when interpreting NCAT results and then applying them to facilitate a transition in their own facility's nursing culture to a more positive one. Administering the NCAT (Items available in earlier publication [20]) to all nursing staff and scoring the NCAT as described above will provide a baseline picture of staff perceptions of their workplace nursing culture. Next, compare the facility performance and that of individual clinical units as indicated to the normative NCAT total score in Table 3 and note the Cumulative Rank Percentage corresponding to your facility/unit mean score. If your mean score is below 65, only 10.9% of facilities scored higher. You may find it useful to further explore your performance and discover which subscale(s) contributed to the lower scores at your facility. Compare facility/unit mean subscale scores to the Cumulative Rank Percentage given in Table  4 for each subscale in order to further identify potential areas in which improvement in staff relationships might be indicated. For example, Teamwork in the change exemplar has a cumulative percentage of only 58.2, which could be considered less than desirable if planning to launch a QI improvement project that requires a focused team effort. Targeting this nursing culture dimension for improvement can be facilitated by examining staff ratings for Teamwork subscale items. For example, team building activities might best be focused on clarifying group concerns if among staff performance on the Teamwork subscale items you find that the lowest rated item by staff is "Staff show respect for one another". Strategies for clarifying expectations regarding respect and defining group activities to help staff share in achievement of mutually set goals will be essential components of an improvement plan. After an appropriate period of time working on improvement strategies, such as 3 to 6 months, re-administer the NCAT and evaluate the impact of strategies on nursing culture. Benchmark your facility's/unit's progress and determine whether your next step is to revise your improvement plan and/or move to introduce your QI planned change project to advance nursing's best practices and care outcomes.

Conclusions
The NCAT item structure yields a reliable and valid instrument that is representative of the construct of nursing culture. With this instrumentation available, LTC leaders are able to assess the nursing culture, thereby enabling them to design and implement intervention strategies aimed at bringing about a positive nursing culture. Results specific to scoring and score interpretation from this research will offer a common point for comparison and could lead to changes in workplace environments that will be effective in improving care outcomes. Furthermore, by better understanding the workplace culture and its effect on licensed nurses and CNAs, it will be possible to design intervention strategies to reduce psychosocial stressors, thus reducing turnover and its latent impact on workload, worker safety, and healthcare practices.