Development, Pilot Study, and Psychometric Analysis of the AHRQ Surveys on Patient Safety Culture™ (SOPS®) Workplace Safety Supplemental Items for Hospitals

Workplace safety is critical for advancing patient safety and eliminating harm to both the healthcare workforce and patients. The purpose of this study was to develop and test survey items that can be used in conjunction with the Agency for Healthcare Research and Quality (AHRQ) Surveys on Patient Safety Culture™ (SOPS®) Hospital Survey to assess how the organizational culture in hospitals supports workplace safety for providers and staff. After conducting a literature review and background interviews with workplace safety experts, we identified key areas of workplace safety culture (workplace hazards, moving/transferring/lifting patients, workplace aggression, supervisor/management support for workplace safety, workplace safety reporting, and work stress/burnout) and drafted survey items to assess these areas. Survey items were cognitively tested and pilot tested with the SOPS Hospital Survey 2.0 among providers and staff in 28 U.S. hospitals. We conducted psychometric analysis on data from 6684 respondents. Confirmatory factor analysis results (item factor loadings and model fit indices), internal consistency reliability, and site-level reliability were acceptable for the 16 survey items grouped into 6 composite measures. Most composite measures were significantly correlated with each other and with the overall rating on workplace safety, demonstrating conceptual convergence among survey measures. Hospitals and researchers can use the Workplace Safety Supplemental items to assess the dimensions of organizational culture that support provider and staff safety and to identify both strengths and areas for improvement.


Introduction
In 2020, the Institute for Healthcare Improvement's (IHI) National Steering Committee for Patient Safety (NSC) released a National Action Plan to Advance Patient Safety in the United States (U.S.) that identified four foundational areas for patient safety: Culture, Leadership and Governance; Patient and Family Engagement; Learning Systems; and Workforce Safety [1]. The report identified workforce safety-the physical and psychological wellbeing and safety of the healthcare workforce-as a necessary precondition to advancing patient safety and highlighted the need for a systems approach to eliminate harm to both patients and the workforce. Further, IHI's National Action Plan identified several priority areas for workforce safety assessment and program implementation, including safe patient handling as well as prevention of exposures (e.g., to pathogens and chemicals), sharps and needlestick injuries, slips/trips/falls, and workplace violence [1].
U.S. healthcare workers experience over 582,000 injuries each year [2]. Based on the 2020 data from the Bureau of Labor Statistics (BLS), the overexertion injury rate for hospital workers is more than twice the national average of U.S. full-time workers [3]. The most The COVID-19 pandemic has further compromised the safety of healthcare workers. Shortages of appropriate personal protective equipment (PPE) in the early phases of the pandemic increased the risk of provider and staff exposure to SARS-CoV-2 and exacerbated their fears of both becoming infected and infecting co-workers and family members [17,18]. The physical and psychological well-being of healthcare workers was further strained by increased patient loads and staffing shortages, all of which significantly contribute to work stress and burnout in the healthcare workforce [19,20]. Thus, the COVID-19 pandemic underscored the need for ongoing organizational support to protect the physical, mental, and emotional safety and well-being of the healthcare workforce [21].
In response to increased concern about the safety of healthcare workers during the COVID-19 pandemic and recognizing the importance of workplace safety in ensuring patient safety, Westat was contracted by the Agency for Healthcare Research and Quality (AHRQ) to develop survey items focusing on workplace safety for providers and staff in the hospital setting. The items were designed and tested as a supplement to the AHRQ Surveys on Patient Safety Culture™ (SOPS ® ) Hospital Survey 2.0 that assesses the extent to which organizational culture in hospitals supports patient safety [22]. Organizational culture refers to the beliefs, values, and norms shared by clinicians and staff within healthcare organizations which influence their actions and behaviors [23]. Similar to the SOPS Hospital Survey, the supplemental items would enable hospitals to assess the extent to which their organizational culture supports the safety of providers and staff.
In healthcare, a comprehensive safety culture includes both patient and workplace safety [24]. Since workplace safety culture and patient safety culture are related [25], hospitals can use these supplemental items to obtain provider and staff perspectives on workplace safety, with a goal of identifying strengths and areas of improvement related to both workplace and patient safety. Herein we present information on the survey development process, pilot study data collection, and results from psychometric analysis of the supplemental items. Figure 1 shows the development process for the Workplace Safety Supplemental Items. The supplemental items were developed using an iterative process starting with a non-systematic review of published and grey literature in various areas of workplace safety, including slips, trips, and falls; sharps injuries; psychological safety and wellbeing; safe patient handling; violence and aggression; workplace safety reporting; and work stress/burnout. We also reviewed extant literature on organizational culture factors that affect the workplace safety of providers and staff, in addition to any existing survey measures on workplace safety. To supplement the results of literature searches and better understand the key elements of workplace safety in healthcare settings, we conducted in-depth interviews with eight workplace safety experts selected based on their content expertise from diverse settings, including universities, healthcare systems, and professional healthcare associations. Figure 1 shows the development process for the Workplace Safety Supplemental Items. The supplemental items were developed using an iterative process starting with a non-systematic review of published and grey literature in various areas of workplace safety, including slips, trips, and falls; sharps injuries; psychological safety and well-being; safe patient handling; violence and aggression; workplace safety reporting; and work stress/burnout. We also reviewed extant literature on organizational culture factors that affect the workplace safety of providers and staff, in addition to any existing survey measures on workplace safety. To supplement the results of literature searches and better understand the key elements of workplace safety in healthcare settings, we conducted indepth interviews with eight workplace safety experts selected based on their content expertise from diverse settings, including universities, healthcare systems, and professional healthcare associations. After synthesizing the information from these sources, we identified key dimensions of workplace safety that were widely applicable to hospital staff, relevant to organizational culture in hospitals, and could be assessed using closed-ended, self-reported survey items. Next, we drafted survey items to assess those key dimensions. As the funding agency, AHRQ provided ongoing input and feedback throughout survey development. In addition, an 18-member Technical Expert Panel (TEP), which included several workplace safety experts, provided feedback on survey dimensions and items at several stages of survey development, starting with the first draft. Based on feedback from the TEP, we revised the initial draft dimensions and survey items in preparation for cognitive testing.

Supplemental Item Development
To assess item comprehension, relevance, and ease of responding, we cognitively tested the draft supplemental items in three rounds of cognitive interviews with 15 providers and staff in English and one round with five providers and staff in Spanish. Participants included physicians, nurses, and administrative/support staff in order to assess the applicability of the items across different positions in hospitals. After each round of cognitive testing, we revised, added, or dropped items based on the results of the cognitive  After synthesizing the information from these sources, we identified key dimensions of workplace safety that were widely applicable to hospital staff, relevant to organizational culture in hospitals, and could be assessed using closed-ended, self-reported survey items. Next, we drafted survey items to assess those key dimensions. As the funding agency, AHRQ provided ongoing input and feedback throughout survey development. In addition, an 18-member Technical Expert Panel (TEP), which included several workplace safety experts, provided feedback on survey dimensions and items at several stages of survey development, starting with the first draft. Based on feedback from the TEP, we revised the initial draft dimensions and survey items in preparation for cognitive testing.
To assess item comprehension, relevance, and ease of responding, we cognitively tested the draft supplemental items in three rounds of cognitive interviews with 15 providers and staff in English and one round with five providers and staff in Spanish. Participants included physicians, nurses, and administrative/support staff in order to assess the applicability of the items across different positions in hospitals. After each round of cognitive testing, we revised, added, or dropped items based on the results of the cognitive interviews or continued testing them, as appropriate. Members of the TEP provided feedback on the draft items after each round before we tested the revised versions.
Although we identified important areas of hospital workplace safety through our literature review, there were some areas that were difficult to assess with survey items. During cognitive interviews, we learned that sharps injuries did not apply to all respondents. Additionally, when a staff member experienced a sharps injury, it was often attributed to staff error rather than facility policies, which was overly punitive for staff and not adequately assessing organizational culture. Another area that was difficult to assess was slips, trips, and falls, as staff were thinking about both patients and workers when answering those questions. Therefore, we did not include questions about those areas in the pilot survey. After cognitive testing and TEP input, the survey items were finalized for the pilot study.

Measures
The pilot study supplemental items consisted of 31 survey items assessing seven a priori dimensions or composite measures of workplace safety in hospitals (see Table 1). Composite measures are groups of two or more survey items that measure the same conceptual domain. Multiple survey items were developed to assess each area more comprehensively, maximize content validity and reliability of measurement, and provide more specific and actionable information to hospitals. 3

Addressing Workplace Aggression from Patients or Visitors *
Physical and verbal aggression from patients or visitors is problematic; and effective policies, procedures, and training are in place to manage workplace aggression.

Addressing Workplace Aggression from Providers or Staff *
Physical and verbal aggression among providers and staff is problematic; and effective policies and procedures are in place to manage workplace aggression. 3

Supervisor, Manager, or Clinical Leader Support for Workplace Safety
Supervisors, managers, or clinical leaders monitor the workplace, seriously consider suggestions for improving safety, encourage providers and staff to report their concerns, and can be trusted to keep providers and staff safe. 4

Hospital Management Support for Workplace Safety
Hospital management shows that workplace safety is a top priority, provides adequate resources for workplace safety, and takes action to address concerns. 3

Workplace Safety and Reporting
Whether or not providers and staff notice or experience workplace safety hazards, injuries, or verbal or physical aggression and whether they report them; providers and staff can report concerns about workplace safety without fear of negative consequences. 9 * These are final measure names and differ slightly from the pilot test version.
In addition to these composite measures, the survey also included one single-item measure, Work Stress/Burnout, and one Overall Rating on Workplace Safety for Providers and Staff, and several background questions on respondent characteristics.
Most questions were asked using five-point Likert-type scales with response options from Strongly disagree to Strongly agree or Never to Always and included a Does Not Apply or Don't Know as the sixth response option. Eight of the Workplace Safety and Reporting items included a set of screener questions asking respondents if they noticed or experienced a safety issue in the past 12 months (e.g., unsafe working conditions, aggression) and, if the response was "Yes", how often they reported the safety issue. The Work Stress/Burnout item was adapted from an item on the Mini-Z instrument (we modified the first response option from I enjoy my work. I have no symptoms of burnout to I have no symptoms of burnout, as we found during cognitive testing that some respondents indicated enjoying their work but also reported experiencing some symptoms of burnout) [26]. Providers and staff who indicated feeling burned out (i.e., The symptoms of burnout that I am experiencing won't go away. I think about work frustrations a lot, or I feel completely burned out. I am at the point where I may need to seek help) were encouraged to consider seeking assistance (e.g., from their insurance provider or employee assistance plan [EAP]). At the beginning of the Workplace Aggression section, we provided definitions and examples of physical and verbal aggression.

Pilot Study
We conducted a pilot study in 28 hospitals across 16 states in the U.S. from May to June 2021. The purpose of the pilot study was to obtain data for psychometric analyses to examine the reliability and construct validity of the items, with the goal of retaining the best performing items. Hospitals were recruited from AHRQ SOPS email listserv subscribers, SOPS Hospital Survey users, webinars, and through outreach to hospital stakeholder organizations. Hospitals were selected to vary by several characteristics (e.g., bed size, region, ownership, teaching status), but were not statistically representative of all U.S. hospitals. The workplace safety items were added toward the end of the SOPS Hospital Survey 2.0 [22] (just before the background questions) and administered by Westat as a web-based survey to a census of all providers and staff in the hospitals. Each provider and staff member received an email with a unique survey link. Data collection procedures were the same across all hospitals.

Analyses
Psychometric analyses included: (1) item analysis to examine the variability of responses and percentages of missing data, (2) internal consistency reliability, (3) confirmatory factor analysis, (4) site-level reliability, and (5) site-level percent positive scores and correlations among composite measures and items. Each of these analyses is described in more detail below.

Item Analysis
We first examined item frequencies at the respondent level to review response variability and identify items with high percentages of missing data or Does Not Apply/Don't Know (DNA/DK) responses. Items with little response variability may not be helpful in differentiating higher-scoring from lower-scoring hospitals. Accordingly, any items with more than 90 percent of respondents responding positively (e.g., those answering Strongly agree/Agree or Always/Most of the time for positively worded items, and Strongly disagree/Disagree for negatively worded items) were considered to have low variability. If more than 30% of respondents left an item missing or answered DNA/DK, the item was considered for removal because it may not be relevant to a large proportion of respondents. However, we did not rely solely on items flagged based on item analysis to determine which items to drop. We also examined results from other psychometric analyses and TEP feedback, weighing the importance and relevance of item content.

Internal Consistency Reliability
Internal consistency reliability indicates how consistently respondents answer items within a composite measure by assessing how closely those items are related or correlated. Internal consistency reliability was assessed for each composite measure using Cronbach's alpha (α). Cronbach's alpha ranges from 0.00 to 1.00, with higher values indicating greater internal consistency. The minimum criterion for acceptable reliability is an alpha of 0.70 [27].

Confirmatory Factor Analysis (CFA)
The purpose of CFA is to confirm a particular pattern of relationships among survey items predicated on past research and theory by assessing how well a proposed factor structure fits the data [28]. A CFA was conducted on the final proposed composite measures and their associated items. Full-information maximum likelihood estimation was employed to address missingness resulting from either nonresponse or those answering Does not apply/Don't Know [29,30].
We examined standardized factor loadings for each item based on its respective composite measure. Factor loadings above 0.40 indicate that the item's relationship to the a priori composite measure is acceptable [31]. Several model fit indices were also examined to determine how well the hypothesized factor structure fits the data ( Table 2). The chi-square goodness-of-fit statistic assesses the discrepancy between the sample covariance matrix and the model-specified covariance matrix. Lower and non-significant values indicate good model fit. Since chi-square tends to be larger and statistically significant in larger samples, we examined the chi-square divided by its degrees of freedom, which is less sensitive to sample size. Values less than 5.00 indicate good model fit [32]. The comparative fit index (CFI) compares the existing model fit with a null model that assumes the factors in the model are uncorrelated. A CFI value of 0.95 or above indicates adequate model fit [33]. The root mean square error of approximation (RMSEA) is a parsimony-adjusted index that favors the simplest model possible [34]. A value less than 0.06 for RMSEA indicates good model fit [33]. The standardized root mean square residual (SRMR) is the standardized difference between the observed covariance and predicted covariance. A value of less than 0.08 for the SRMR indicates good model fit [35].

Site-Level Reliability Analysis
We computed site-level reliability to examine the variability of item and composite measure scores among hospitals. Site-level reliability indicates the extent to which responses given by providers and staff within the same hospital are more similar to each other than they are to responses given by providers and staff from other hospitals. In other words, site-level reliability helps to assess how well a measure differentiates hospitals. It does so by comparing between-hospital variability to within-hospital variability of items. Site-level reliability was computed using the following formula [36]: Site − level Reliability = between hospital variance between hospital variance + within hospital variance n where n is the number of respondents in a given hospital. Site-level reliability for each measure was first computed for individual hospitals and then averaged across hospitals. This approach helps to take into consideration the differences in the number of respondents in each hospital. Similar to internal consistency reliability, values of 0.70 or higher are considered acceptable for site-level reliability [27].

Hospital-Level Percent Positive Scores and Correlations
We calculated hospital-level percent positive scores as the percentage of respondents within a site who answered positively (% Strongly agree/Agree or Always/Most of the time for positively worded items, and % Strongly disagree/Disagree for negatively worded items) for each item. These site-level percent positive scores for the items within each composite measure were equally weighted and averaged to compute site-level composite measure scores. Percent positive scores can range from 0 to 100. We examined Spearman's rank order correlations among the composite measures, single-item measures, and the overall rating. The Workplace Safety measures should be intercorrelated, as they are all designed to assess aspects of culture focusing on workplace safety. Moderate to moderately high correlations typically indicate a correspondence or convergence among similar concepts.

Results
Across the 28 hospitals that participated in the pilot study, 7037 providers and staff out of 19,979 responded, for an overall response rate of 35%. On average, there were 251 respondents per hospital (ranging from 21 to 1373). Of the 7037 respondents, 353 did not answer any Workplace Safety items and thus were excluded from the final analysis dataset, which resulted in a final analytic sample of 6684 respondents.

Item Analysis
We examined the variability of responses to survey items at the individual or respondent level. Table 5 shows the average percent positive and percent missing and Does not apply/Don't know (DNA/DK) responses for items in the a priori composite measures, as well as one Workplace Safety and Reporting item and one Overall Rating on Workplace Safety for Providers and Staff. While there were also eight other items measuring Workplace Safety and Reporting, these items are not shown in Table 5 because they were dropped based on their complexity, length, and some redundancy in content with items in other sections. Four items had low variability: three items in Protection from Workplace Hazards (range, 91% to 92%) and one workplace aggression item ("In this unit, there is a problem with providers or staff being physically aggressive toward other providers or staff" (91% of respondents Strongly Disagreed/Disagreed)). The remaining items had acceptable variability, with percent positive scores below 90% (range 45% to 89%). We did not find any items with a high percentage of missing responses; all items had < 5% missing responses (range <1% to 4%).
The percentages of DNA/DK responses ranged from 5% to 43%. Two items in Moving, Transferring, or Lifting Patients had high percentages of DNA/DK: "Equipment or assistive devices are available when needed to help move, transfer, or lift patients in this unit" (42%) and "In this unit, staff use equipment or assistive devices when needed to help move, transfer, or lift patients, even if it takes more time" (43%). When these items were further investigated, the percent of DNA/DK was not as high (<30%) for the majority of positions in which staff are likely to engage in moving, transferring, or lifting patients, such as registered nurses; patient care aides, hospital aides and nursing assistants; physical, occupational, or speech therapists; respiratory therapists; and transporters. Table 6 shows responses for the item assessing Work Stress/Burnout. While 69% of respondents indicated having no symptoms of burnout, 32% reported some symptoms of burnout, including 3% who reported feeling completely burned out. This item had 3% missing responses.

Addressing Workplace Aggression from Patients or Visitors (five items)
In this unit, there is a problem with patients or visitors being physically aggressive toward providers or staff (negatively worded) * 59% 3% 25% In this unit, there is a problem with patients or visitors being verbally aggressive toward providers or staff (negatively worded) * 45% 3% 24% In this unit, there are effective policies and procedures to keep providers and staff safe from aggressive patients or visitors 71% 3% 22% In this unit, providers and staff are trained to recognize early signs of aggressive behavior from patients or visitors 72% 3% 20% In this unit, providers and staff are trained on how to deescalate or calm down aggressive behavior from patients or visitors 67% 3% 20%

Addressing Workplace Aggression from Providers or Staff (three items)
In this unit, there is a problem with providers or staff being physically aggressive toward other providers or staff (negatively worded) * 91% 3% 14% In this unit, there is a problem with providers or staff being verbally aggressive toward other providers or staff (negatively worded) * 74% 4% 13% In this unit, there are effective policies and procedures to address providers and staff who behave aggressively toward other providers or staff 65% 4% 19%

Supervisor, Manager, or Clinical Leader Support for Workplace Safety (four items)
My supervisor, manager, or clinical leader regularly monitors the workplace to identify unsafe working conditions for providers and staff 77% 2% 8% My supervisor, manager, or clinical leader seriously considers provider or staff suggestions for improving workplace safety 78% 2% 6% My supervisor, manager, or clinical leader encourages providers and staff to report their concerns about workplace safety 83% 2% 5% My supervisor, manager, or clinical leader can be trusted to do the right thing to keep providers and staff safe 82% 2% 5%

Hospital Management Support for Workplace Safety (three items)
The actions of hospital management show that the safety of providers and staff is a top priority 70% 2% 5% Hospital management provides adequate resources to ensure the safety of providers and staff 70% 3% 5% Hospital management takes action to address provider and staff concerns about workplace safety 69% 3% 6% Workplace Safety and Reporting (one item) I can report my concerns about workplace safety without fear of negative consequences for me 76% 2% 2%

Overall Rating on Workplace Safety for Providers and Staff (one item)
How would you rate your unit/work area on workplace safety for providers and staff? 50% 3% N/A Table 6. Item analysis results: Work Stress/Burnout (N = 6508).

Work Stress/Burnout (one item) % % No Burnout and Burnout
Using your own definition of "burnout", please select one of the answers below: I have no symptoms of burnout. 32% 69% (No symptoms of burnout) I am under stress, and don't always have as much energy as I did, but I don't feel burned out. 37% I am definitely burning out and have one or more symptoms of burnout, e.g., emotional exhaustion. 21% 32% (Symptoms of burnout) The symptoms of burnout that I am experiencing won't go away. I think about work frustrations a lot. 8% I feel completely burned out. I am at the point where I may need to seek help. 3% Note: Percentages may not add to 100 percent due to rounding.

Internal Consistency Reliability Analysis
All a priori composite measures had initial internal consistency reliability at or above criterion (α ≥ 0.70). Cronbach's alpha ranged from 0.70 on Addressing Workplace Aggression from Providers or Staff to 0.96 on Hospital Management Support for Workplace Safety.

TEP Review and Input
We presented the initial analysis results to the Workplace Safety TEP to obtain their input on whether to retain or drop items. To shorten the survey, we also asked the TEP to identify other items to drop based on content relevance. After TEP feedback, five survey items were dropped: one item each from Protection from Workplace Hazards, Addressing Workplace Aggression from Patients or Visitors, and Supervisor, Manager, or Clinical Leader Support for Workplace Safety, and two items from Addressing Workplace Aggression from Providers or Staff. Details about which items were dropped and reasons for dropping these items are provided in Table S1.

Confirmatory Factor Analysis (CFA)
When we conducted a confirmatory factor analysis (CFA) on the retained survey items, the resulting model showed poor fit to the data based on all fit indices. In addition, standardized factor loadings for items in Addressing Workplace Aggression from Patients or Visitors were lower relative to factor loadings obtained for other composite measures. Thus, we conducted an exploratory factor analysis on the four remaining items in that measure using iterated principal axis factors as the method of extraction, with varimax (orthogonal) rotation. Results supported a two-factor solution rather than one overall factor. The first factor had two items focusing on physical and verbal aggression from patients or visitors (Addressing Workplace Aggression from Patient or Visitors). The second factor had two items focusing on effective policies and procedures and training on how to deescalate or calm down aggressive behavior from patients or visitors (Workplace Aggression Policies, Procedures, and Training). Since only one aggression item remained in the Addressing Workplace Aggression from Providers and Staff a priori composite measure, "In this unit, there is a problem with providers or staff being verbally aggressive toward other providers or staff", this composite measure became a single-item measure (Addressing Verbal Aggression from Providers or Staff ).
To test the fit of the six proposed composite measures to the data, we reran the CFA. Table 7 displays standardized factor loadings for each of the Hospital Workplace Safety Supplemental Items on their respective composite measures. All factor loadings were statistically significant (p < 0.05) with magnitudes greater than 0.40, indicating that the items adequately loaded on the composite measures. The factor loadings ranged from 0.59 to 0.95, with an average of 0.85.

Addressing Verbal Aggression from Providers or Staff
In this unit, there is a problem with providers or staff being verbally aggressive toward other providers or staff --0.80 Workplace Safety and Reporting I can report my concerns about workplace safety without fear of negative consequences for me --0.76

Work Stress/Burnout
Using your own definition of "burnout", please select one of the answers below --0.62

Overall Rating
How would you rate your unit/work area on workplace safety for providers and staff? --0.86 Notes: Composite measure scores at the respondent level were calculated as means of their respective constituent items. Survey measures are shown in the order they appear in the survey within each category: composite measures, single-item measures, and overall rating. All factor loadings were statistically significant (p < 0.05).
Three of the four goodness-of-fit indices for the six-factor model satisfied the criteria for acceptable fit of the model to the data (Table 8). Specifically, the CFI was 0.98 (criterion is ≥0.95), the SRMR was 0.05 (criterion is <0.08), and the RMSEA was 0.04 (criterion is <0.06). The chi-square value divided by the degrees for freedom was 13.5, which is above the criterion of <5.00. After confirming the measurement structure, we re-calculated Cronbach's alpha for the six composite measures. As presented in Table 7, the reliability estimate for Workplace Aggression Policies, Procedures, and Training was 0.67, slightly under the criterion of 0.70. The remaining alpha coefficients ranged from 0.83 for Moving, Transferring, or Lifting Patients to 0.96 for Hospital Management Support for Workplace Safety. Dropping one item "In this unit, enough staff are available when needed to help move, transfer, or lift patients" from the Moving, Transferring, or Lifting Patients would have increased the alpha from 0.83 to 0.88. However, upon careful review, we determined that the conceptual importance of this item outweighed any increase in reliability that would result from its removal. Dropping any other items from their respective composite measures would not have resulted in increases in reliability.

Discussion
The Hospital Workplace Safety Supplemental Items demonstrated good psychometric properties. Internal consistency reliability estimates were acceptable for five of the six composite measures. At 0.67, the alpha for Workplace Aggression Policies, Procedures, and Training composite measure fell only slightly under the criterion of 0.70. All composite measures and all but two items had acceptable site-level reliability, indicating that the survey measures differentiated hospitals. Standardized factors loadings and model fit indices from the CFA provided support for the construct validity of the final six composite measures.
In addition, most composite measures were moderately, yet significantly, intercorrelated and also related to the Overall Rating on Workplace Safety for Providers and Staff, indicating adequate conceptual convergence among these measures. Only one composite measure, Workplace Aggression Policies, Procedures, and Training, was not associated with other composite measures or the overall rating. However, the small number of hospitals (N = 28) used in the analysis could have contributed to the non-significant findings. Hospital Management Support for Workplace Safety was strongly associated with the Overall Rating on Workplace Safety for Providers and Staff (r s = 0.90). Although this association is bivariate, and therefore unadjusted, it suggests the importance of support for workplace safety from hospital management on the overall rating of workplace safety.
Almost one-third of providers and staff across the study hospitals indicated experiencing at least one symptom of work stress/burnout. Furthermore, experiencing work stress/burnout was related to lower protection from workplace hazards and lower support for workplace safety from supervisors, managers, or clinical leaders and from hospital management. These could be important areas for future investigations, as burnout is a significant threat to the health and well-being of healthcare professionals [37].
While Protection from Workplace Hazards resulted in a high average percent positive score for the hospitals in our pilot study (90% positive), average percent positive scores for the remaining five composite measures were not as high, ranging from 58% to 82%.
In addition, only about half of providers and staff rated their unit/work area as Excellent or Very good on workplace safety. Although the number of study hospitals was small and hospitals were not randomly selected, taken together, this evidence suggests ample room for improvement in multiple areas of workplace safety culture.
Two of the three items in the Moving, Transferring, or Lifting Patients composite measure had relatively high percentages of respondents answering, Does not apply/Don't know (DNA/DK). Further analyses indicated that the percent of DNA/DK for these items was generally lower for staff expected to engage in moving, transferring, or lifting patients, such as those in clinical positions requiring direct patient care. Because staff seemed to be appropriately selecting DNA/DK, and given the importance of this aspect of workplace safety, we decided to retain this composite measure in the final item set.

Limitations
The hospitals were not randomly selected and thus are not representative of all U.S. hospitals. However, the hospitals were selected using purposive sampling to vary by region, bed size, ownership, and hospital teaching status to ensure diversity on those key characteristics. In addition, respondents represented diverse positions within hospitals. These factors helped to assure that our findings are based on a fairly representative set of hospital respondents. In addition, while the supplemental items were added to the end of the SOPS Hospital Survey and made it longer, and our data collection occurred during the COVID-19 pandemic, we were able to obtain a sufficient overall response rate of 35%. In fact, the questions about workplace safety are likely to have been particularly salient to hospital providers and staff as a result of COVID-19-related concerns about PPE and stress/burnout.

Conclusions
The final Hospital Workplace Safety Supplemental Items released in October 2021 [38] extend the SOPS Hospital Survey by assessing additional dimensions of organizational culture that affect workplace safety for providers and staff. The supplemental item set consists of twenty-two supplemental items, including six composite measures, three single-item measures, one overall rating, and two background items. The final items and composite measures are not only psychometrically sound, but also measure different yet related aspects of workplace safety in hospitals. Healthcare organizations and researchers can use these items in conjunction with the SOPS Hospital Survey to assess how well the organizational culture in hospitals supports workplace safety and to identify strengths and areas for improvement. Given the importance of both workplace safety and patient safety in ensuring a comprehensive safety culture in healthcare [24], future work should more fully explore the relationship between patient safety culture and workplace safety culture.
Supplementary Materials: The following supporting information can be downloaded at: https: //www.mdpi.com/article/10.3390/ijerph19116815/s1, Table S1: Pilot survey items dropped from final item set based on Technical Expert Panel (TEP) review.
Author Contributions: All authors participated in the development and testing of the supplemental items. K.Z., N.Y. and J.S. conceptualized the current study and drafted the manuscript. K.Z. conducted the analyses. All authors reviewed and revised the manuscript for submission to the journal. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement:
The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board at Westat (FWA 00005551). The study protocol was carried out in accordance with relevant guidelines and regulations.
Informed Consent Statement: Informed consent language was included at the beginning of the web survey, along with Westat's IRB contact information, but we received a waiver of written informed consent for the web survey. The informed consent language was "The survey is voluntary, but your feedback will help your hospital identify areas for patient safety and workplace safety improvement. If you do not wish to answer a question, you may leave it blank. Westat will keep your individual responses to this survey confidential. Only group results will be reported." Data Availability Statement: Some of the de-identified Workplace Safety Supplemental Item data are available upon request for research purposes.