Building a Resilience Ecosystem to Improve Employee Mental Health and Wellbeing in Canadian High-Stress Low-Control Occupations

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The study represents a significant contribution to the field of mental health among public safety personnel. The authors reflect on the three-year implementation of the Canadian National Standard  across five organizations. I consider the shift from an individualistic understanding of resilience toward an 'ecological model' - which incorporates family and organizational culture - to be a key contribution. Methodologically, the Integrated Knowledge Translation (approach and the effort to link psychometric data with economic indicators (such as health benefit costs) are particularly valuable.

The study is subject to limitations stemming from the small sample size of organizations (n=5) and inconsistencies in administrative sick leave data, which failed to correlate with psychological improvements. Furthermore, the finding that the positive effects of resilience training dissipated after four months is particularly noteworthy, as it raises concerns regarding the long-term sustainability of such interventions.

To enhance the quality and interpretative value of the article, I recommend that the authors address the following points:

• Expanding the discussion on 'Sick Leave': Given that no correlation was found between mental health and the utilization of sick leave, I recommend a more in-depth analysis of the 'culture of presenteeism' or 'shift-swapping' phenomena. In the PSP community, it is common for employees to manage health issues informally, which masks administrative sick leave data. Including this explanation as a legitimate confounding factor would significantly benefit the authors' analysis.
• The authors state that the improvements in resilience were only short-term. I recommend including specific practical suggestions in the text regarding how organizations should modify their training programs to ensure sustained effects (e.g., the introduction of 'booster' sessions or continuous mentoring).
• Anomalies appear within the text (e.g., the negative correlation between depression and benefit costs at T1). I recommend that the authors provide at least a theoretical interpretation of these phenomena in the discussion (e.g., the stigma associated with utilizing benefits for depression) rather than merely stating them as findings.
• For better clarity, it would be appropriate to include a graphical representation of the correlation between organizational stress and actual financial costs (Benefit Costs), as this constitutes one of the study's strongest arguments for corporate management.
• Recommend unifying the format for reporting p-values and correlation coefficients throughout the tables to maintain a high formal standard of the manuscript.

Author Response

Reviewer 1

 

The study represents a significant contribution to the field of mental health among public safety personnel. The authors reflect on the three-year implementation of the Canadian National Standard across five organizations. I consider the shift from an individualistic understanding of resilience toward an 'ecological model' - which incorporates family and organizational culture - to be a key contribution. Methodologically, the Integrated Knowledge Translation (approach and the effort to link psychometric data with economic indicators (such as health benefit costs) are particularly valuable.

Response: Thank you for the kind words

 

The study is subject to limitations stemming from the small sample size of organizations (n=5) and inconsistencies in administrative sick leave data, which failed to correlate with psychological improvements. Furthermore, the finding that the positive effects of resilience training dissipated after four months is particularly noteworthy, as it raises concerns regarding the long-term sustainability of such interventions... The authors state that the improvements in resilience were only short-term. I recommend including specific practical suggestions in the text regarding how organizations should modify their training programs to ensure sustained effects (e.g., the introduction of 'booster' sessions or continuous mentoring).

Response: We have previously studied resilience skill decay and have added reference to this work. We have also added some discussion to the conclusion.

 

Expanding the discussion on 'Sick Leave': Given that no correlation was found between mental health and the utilization of sick leave, I recommend a more in-depth analysis of the 'culture of presenteeism' or 'shift-swapping' phenomena. In the PSP community, it is common for employees to manage health issues informally, which masks administrative sick leave data. Including this explanation as a legitimate confounding factor would significantly benefit the authors' analysis.

Response: This has been added to the limitations of the study. This was most pronounced in fire service organizations and less so in police services where contracts allowed for sick days with pay.

 

Anomalies appear within the text (e.g., the negative correlation between depression and benefit costs at T1). I recommend that the authors provide at least a theoretical interpretation of these phenomena in the discussion (e.g., the stigma associated with utilizing benefits for depression) rather than merely stating them as findings.

Response: Thank you for the suggestion. You will find Line 574 discusses the potential impacts of stigma within PSP, while comments on potential reasons for the anomalies were added at lines 647.

 

For better clarity, it would be appropriate to include a graphical representation of the correlation between organizational stress and actual financial costs (Benefit Costs), as this constitutes one of the study's strongest arguments for corporate management.

Response: We thank the reviewer for the suggestion to provide a graphical representation. We have added Figure 2 to the manuscript, which plots Organizational Stress against Extended Health for the four organizations. While five organizations participated in the broader study, one organization lacked data for the 'Extended Health Cost' variable. We have updated the manuscript text to explicitly state this data constraint.

To clarify the relationships observed, we have included non-parametric trend lines (dashed for T2, long-dashed for T1) to illustrate the directional patterns. While the sample size is n=4, the plot demonstrates that the data points follow a consistent positive trend. This supports our use of Kendall’s tau, which is specifically suited for small-N datasets as it relies on the rank concordance of pairs.

 

Recommend unifying the format for reporting p-values and correlation coefficients throughout the tables to maintain a high formal standard of the manuscript.

Response: For the PHW Survey Responses (Table 4), we have added the p-value in the table. Results are now reported to three decimal places with leading zeros removed, and statistical significance is clearly denoted with asterisks (*p < .05, **p < .01).

Regarding the pre-post course survey comparisons, we have maintained these as a descriptive analysis of response scores. Given the significant discrepancy in sample sizes between the pre-course and post-course (e.g., n=131 vs. n=15) groups, we have intentionally omitted p-values for this specific comparison to avoid misleading interpretations of statistical significance in an underpowered context.

We would like to clarify a typographical error in the Abstract: the correlation was mistakenly labeled as r instead of Kendall’s tau . We have corrected this in the revised manuscript to ensure consistency with the Results section and Table 6. Regarding our choice of statistical method, we utilized Kendall’s tau rather than the Pearson correlation coefficient for the following reasons. Given our small sample size (n = 4/5 for these specific indicators), Kendall’s tau is more robust and less sensitive to outliers than Pearson’s r. Kendall’s tau, being a non-parametric measure based on rank concordance, provides a more reliable estimate of the relationship between organizational survey measures and administrative indicators in this context.

 

 

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors,

Thank you for the opportunity to review your manuscript entitled “Building a resilience ecosystem to improve employee mental health and wellbeing in high-stress low-control occupations.” The topic is highly relevant, and your work offers an important contribution to understanding how multilevel resilience initiatives can be implemented within public safety organizations. I appreciate the considerable effort invested in collecting longitudinal data, engaging multiple organizations, and integrating the National Standard and the Knowledge-to-Action Cycle into your framework.

Below, I provide detailed comments intended to support the strengthening of your manuscript. My aim is to help enhance clarity, methodological transparency, and analytical rigor so that the contribution of your work can be fully appreciated by readers.

General Comments

Your manuscript presents a rich and ambitious implementation study with strong practical relevance. The ecological resilience model and the multi‑method approach are notable strengths. However, several aspects of the manuscript would benefit from clarification, restructuring, and more cautious interpretation of results. In particular, methodological details, statistical procedures, and the organization of the results and discussion sections require further refinement.

Major Comments

1. Methodological Clarity

The description of the study design as a “longitudinal multiple‑cohort implementation science” approach is interesting but requires clearer justification. Please elaborate on the rationale for this design, its limitations, and how it aligns with your research aims.

Data cleaning procedures (e.g., removal of the fastest 2% of responses) should be supported by references or methodological justification. Similarly, the rationale for aggregating data at the organizational level needs to be explained more explicitly, as this substantially reduces statistical power.

2. Statistical Analyses

The use of Kendall’s tau with only five organizations raises concerns regarding the robustness of the correlations reported. Some values (e.g., τ = 1.0) appear statistically implausible and should be interpreted with caution. Please provide a more detailed explanation of the analytical choices, including limitations associated with small sample sizes.

3. Results Presentation

The results section is comprehensive but would benefit from greater synthesis. Consider reorganizing the section into clearer subsections (surveys, training outcomes, organizational indicators, correlational analyses) and providing summary tables with effect sizes, confidence intervals, and significance levels.

Interpretations of increases in social phobia and decreases in individual resilience should be framed more cautiously, given the exploratory nature of the study and the small sample sizes at later time points.

4. Discussion and Interpretation

The discussion is informative but overly long and occasionally repetitive. It would benefit from a more focused structure, clearer linkage to the empirical findings, and a more balanced consideration of methodological limitations (e.g., self‑selection bias, attrition, absence of a control group).

Please also expand the discussion of how your findings align with or diverge from recent literature on psychological health and safety implementation in PSP contexts.

5. Manuscript Structure and Conciseness

Some theoretical sections, particularly those providing historical background on resilience, could be condensed to improve focus and readability. Streamlining these sections would allow the empirical contribution to stand out more clearly.

 

Minor Comments

• The abstract should include key quantitative indicators and be more concise.
• The title could be strengthened by specifying the Canadian context and ecological model.
• Clarify how incomplete responses were handled.
• Ensure consistent terminology when referring to PSP subgroups (e.g., police vs. fire services).
• Consider integrating the KTA Cycle more explicitly into the interpretation of results.

Questions for Clarification

1. How do you justify the validity of correlations based on n = 5 organizations?
2. How were differences between police and fire services addressed analytically?
3. What is the empirical basis for excluding fast responses?
4. How do you interpret the decrease in individual resilience at T2?
5. Why were more robust T2/T3 data not collected for the training component?
6. How do you rule out external organizational changes as alternative explanations for improvements?

 

Conclusion

Your study addresses an important and timely topic and has the potential to make a meaningful contribution to the literature on psychological health and safety in high‑stress occupations. With substantial revisions to improve methodological transparency, analytical rigor, and narrative focus, the manuscript could be significantly strengthened. I hope these comments are helpful as you revise your work.

Author Response

Reviewer 2

Thank you for what we perceive as fair and thoughtful comments which we have tried to address in the following manner:

 

Major Comments

1. Methodological Clarity

The description of the study design as a “longitudinal multiple‑cohort implementation science” approach is interesting but requires clearer justification. Please elaborate on the rationale for this design, its limitations, and how it aligns with your research aims.

RESPONSE: As we were required to guarantee absolute anonymity for participants, paired data was not an option in the analysis. Further, targeted improvements included items of comparison that were collected at the department level (like sick time, long term leave, overtime, call volume, health benefits costs among others) making a case study the most logical way to deal with organizational data over time from multiple organizations. Following processes described by the Canadian Institutes for Health Research for implementation science and knowledge to action cycles, we believe the description is accurate. Limitations to the methods are described in the limitations, with the major one being the small organization n limiting statistical comparison and power.

 

Data cleaning procedures (e.g., removal of the fastest 2% of responses) should be supported by references or methodological justification. Similarly, the rationale for aggregating data at the organizational level needs to be explained more explicitly, as this substantially reduces statistical power.

Response: Thank you for this important observation. The removal of the fastest 2% of responses is based on response time per item (sec/item), a well-established index for detecting careless or insufficient effort (C/IE) responding in survey data. The theoretical rationale is that a minimum amount of time must elapse for a respondent to read, cognitively process, and meaningfully respond to each item — respondents completing items below this threshold are unlikely to have engaged with item content (Curran, 2016; Huang et al., 2012). We used a data-driven, distributional approach by identifying respondents falling below the 2nd percentile of the observed sec/item distribution within each survey (around 3.5 sec/item). This approach is supported by Matjašič et al. (2018), whose systematic review of web survey paradata documents the use of percentile-based cutoffs as a standard statistical method for identifying response time outliers. The resulting threshold of approximately 3.5 sec/item in our surveys is above the conventional 2 sec/item minimum benchmark (Huang et al., 2012) to accommodate the length of our survey questions. This threshold was also internally verified within our study context to ensure optimal data filtering. The second-stage cleaning criteria (top 5% completion time combined with additional quality flags) follows the multi-indicator approach recommended by Curran (2016), who emphasized that combining response time with content-based indicators improves detection accuracy and reduces false exclusions.

1. Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19.
2. Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99-114.
3. Matjašič, M., Vehovar, V., & Manfreda, K. L. (2018). Web survey paradata on response time outliers: A systematic literature review. Advances in Methodology and Statistics, 15(1), 23-41.

 

1. Statistical Analyses

The use of Kendall’s tau with only five organizations raises concerns regarding the robustness of the correlations reported. Some values (e.g., τ = 1.0) appear statistically implausible and should be interpreted with caution. Please provide a more detailed explanation of the analytical choices, including limitations associated with small sample sizes.

Response:  We acknowledge the reviewer’s concern regarding the small sample size. We have expanded our explanation of the analytical choices in Section 2.6 and limitations in Section 4.1.

We utilized Kendall’s tau because it is a non-parametric, rank-based coefficient specifically recommended for small datasets where parametric assumptions (normality and linearity) cannot be met. Unlike Pearson’s r, Kendall’s tau is based on the concordance of pairs.  τ = 1.0 indicates that all pairs are concordant (i.e., the ranks match perfectly), which is a common and mathematically plausible outcome in small-N rank correlation. We have clarified in the text that these values are intended to describe the strength of the directional trend within this specific sample rather than to provide generalized population-level inferences.

 

1. Results Presentation

The results section is comprehensive but would benefit from greater synthesis. Consider reorganizing the section into clearer subsections (surveys, training outcomes, organizational indicators, correlational analyses) and providing summary tables with effect sizes, confidence intervals, and significance levels.

Response: The results were formatted into sections with some movement of material for better flow. Significance levels were added to Table 4.

 

Interpretations of increases in social phobia and decreases in individual resilience should be framed more cautiously, given the exploratory nature of the study and the small sample sizes at later time points.

Response: Language was toned down in the plausible factors.

 

1. Discussion and Interpretation

The discussion is informative but overly long and occasionally repetitive.

Response: some repetitive material was removed, such as repetition of T1 and T2 definitions, but the additional information requested did make it longer.

 

It would benefit from a more focused structure, clearer linkage to the empirical findings… Please also expand the discussion of how your findings align with or diverge from recent literature on psychological health and safety implementation in PSP contexts.

Response: We added a paragraph (line 603) to try and align our results to current literature reporting on longitudinal efforts in PSP and PSP families

 

and a more balanced consideration of methodological limitations (e.g., self‑selection bias, attrition, absence of a control group).

Response: We have expanded our explanation of the analytical choices in Section 2.6 and limitations in Section 4.1.

 

1. Manuscript Structure and Conciseness

Some theoretical sections, particularly those providing historical background on resilience, could be condensed to improve focus and readability. Streamlining these sections would allow the empirical contribution to stand out more clearly.

 Response: While reviewers differed in this regard significant changes were not made to the introduction.

 

Minor Comments

• The abstract should include key quantitative indicators and be more concise.

Response: We would like to clarify a typographical error in the Abstract: the correlation was mistakenly labeled as r instead of Kendall’s tau . We have corrected this in the revised manuscript. We have also condensed the abstract.

• The title could be strengthened by specifying the Canadian context and ecological model.

Response: we have added “Canadian” to the title

• Clarify how incomplete responses were handled.

Response: We have clarified the handling of incomplete data in the "Data Cleaning" section. The survey was designed with a forced-response format, preventing respondents from skipping individual items. However, participants retained the right to withdraw or exit the survey at any time. Submissions with no data beyond the consent page were excluded from the dataset. For participants who exited the survey prematurely, we utilized all completed data for the specific instruments they had finished. We have updated the manuscript to explicitly state that only fully completed instruments were included in the corresponding analyses.

• Ensure consistent terminology when referring to PSP subgroups (e.g., police vs. fire services).

Response: thank you, we have tried to pick up where differences occurred

• Consider integrating the KTA Cycle more explicitly into the interpretation of results.

Response: We added a line about the benefit of using a knowledge to action cycle at line 614 and linked it to process at line 631. Many of the outputs of the KTA Cycle were not reported in this paper with outputs reported in other papers (such as our systematic literature review and meta analysis on return on investment). The best example of use of KTAC in this paper is the development of material for the training courses.

Questions for Clarification

1. How do you justify the validity of correlations based on n = 5 organizations?

Response:  We have expanded our explanation of the analytical choices in Section 2.6 and limitations in Section 4.1.

The validity of these correlations is grounded in the choice of Kendall’s tau, a non-parametric method specifically recommended for very small datasets. While the small sample size limits the ability to generalize these findings to a broader population (low statistical power), the method provides a valid and robust measure of rank-order consistency within the studied organizations. We report these as descriptive, exploratory trends that indicate a strong directional relationship between organizational survey responses and administrative indicators within this specific cohort.  We have clarified in the text that these values are intended to describe the strength of the directional trend within this specific sample rather than to provide generalized population-level inferences.

 

1. How were differences between police and fire services addressed analytically?

Response: as the unit of comparison was a department, and each had differing activities and work-related events, we did not factor in police and fire organizations for this analysis. The only police/fire distinctions were made in survey results (line 499).

1. What is the empirical basis for excluding fast responses?

Response: Same as the response to the question of Reviewer 2’s major comments (1. Methodological Clarity)

1. How do you interpret the decrease in individual resilience at T2?

Response: We have added a reference to support the skill decay timeline that can be expected.

1. Why were more robust T2/T3 data not collected for the training component?

Response: We tried several times to increase our n, but were not successful even with labour and management support. The champions in each organization were also promoting participation. The completion of activities was poor even when started. Departments offered several explanations, but we have no way of determining the lower participation.

1. How do you rule out external organizational changes as alternative explanations for improvements? (any ideas?)

To be blunt, we cannot. As with any longitudinal study the significant events impacting each department were different and outside of our control (for example, one police service had a leadership change, while one fire service changed their shift pattern) which is why we had chosen a case series design. All agencies did however follow the general framework for improving mental health and wellbeing, created a mental health and wellbeing strategic plan the project funded, had access to the same funded material and events developed by the research group, attended monthly meetings the year prior to implantation all the way through to final reporting, and had the same structured self-studies of progress annually. We hope to publish a paper with clear examples from each case study at a later date.

 

Conclusion

Your study addresses an important and timely topic and has the potential to make a meaningful contribution to the literature on psychological health and safety in high‑stress occupations. With substantial revisions to improve methodological transparency, analytical rigor, and narrative focus, the manuscript could be significantly strengthened. I hope these comments are helpful as you revise your work.

Response: Thank you. We do appreciate your feedback and assistance in improving the paper significantly.