1.1. The Logic Model and the Theory Behind the Intervention Implemented in the RCT Study—The Bases for this Process Evaluation
1.2. Theoretical Frameworks in Process Evaluation
- (A) To explore the implementation of an organizational intervention, ProMES, as a stress-preventive intervention at a Swedish primary health care (i.e., recruitment, reach, dose). (B) To assess the extent to which the intervention components were implemented in accordance with the original guidelines (fidelity and adaptations);
- To examine the influence of contextual factors on the implementation process (hindrances and facilitators);
- To examine the participants’ experience of working with the method (satisfaction, relevance, acceptability, feasibility).
2. Materials and Methods
2.1. Study Design
2.2. Implementation Object (ProMES)
2.3. Measurement and Data Collection
2.3.2. Mechanisms of Impact: Participant Experience
2.4. Data Analysis
3.1. Implementation Outcomes
3.1.3. Dose Delivered
3.1.4. Fidelity and Adaptation
3.2. Mechanisms of Impact: Participants Experience
3.3. Context Facilitators and Hindrancies
3.3.1. Intervention Characteristics
“I have been exposed to many such methods over the years, but this is the first one that looks at the big picture… Before (prior to ProMES), the doctors would come up with things that gave more job for nurses or for laboratory technicians or secretaries, and especially the secretaries. This is the first time we have looked at the big picture”.
“We may have answered more phone calls, increased the number of visits…. But if it is because we have implemented ProMES, or if it is because we have strengthened the staffing or just a natural result of our own usual development work, I cannot say one or the other”.
“If the consultant is not here to help us anymore…I think of these tables when I pass the board, I think there is a little too much to read, one cannot take everything in. I think that if we are in it without the consultant then you want it collected in a simpler way so that we can quickly see what we have done”.
3.3.2. Inner Setting
“We have met the primary health care management at higher level and the person responsible for IT in primary health care, and the data output group at county council level, and they do not think that the outputs we are interested in are interesting enough to invest in…I think it is wrong to think that we can only go on measuring what the politicians or the leaders of the county council want to measure, wrong not to invest in the fact that the working people have thoughts and ideas”.
“Because when we get below the 50 percent (of usual workforce), then it starts to be… Not good. Then, it does not matter which system you use. There are just too few hands”.
“Such a large staff loss was not even in our wildest imagination. In addition, new nurses and physicians under specialist training required extra supervision”.
“Personnel turnover is a disadvantage because those who have participated in the process disappear, and those who come are not on the train, they do not know what it is about”.
“I can feel that we are a fairly large group, there are 15 of us and there were 4 who were working with the consultant. I think we others did not have the same participation as those who sat with him, that there was not so much time to address everything that was expressed in the group. Then the consultant sent out what had been said (in the design team) and how it improved and so, so it is good to have taken part … but not everyone in our group was so involved…”.
“It is still the case that it is not the solution to the staff shortages, but I think that with the model (i.e., ProMES) and especially if we were to have more meetings between different professional categories, then I think that we would gain more for the health care unit, than just working with each individual professional group”.
“When we have three care teams, we will also have a greater opportunity to compare different teams with each other based on different working methods. To test different stuff. We have all learned a model that we can build on. We hope that the work will continue”.
4.1. Methodological Considerations
4.2. Implications for Future Research
Conflicts of Interest
- Stansfeld, S.A.; Candy, B. Psychosocial work environment and mental health—A meta-analytic review. Scand. J. Work Environ. Health 2006, 32, 443–462. [Google Scholar] [CrossRef] [PubMed]
- Ruotsalainen, J.H.; Verbeek, J.; Mariné, A.; Serra, C. Preventing occupational stress in healthcare workers. Cochrane Database Syst. Rev. 2015, 2015, CD002892. [Google Scholar] [CrossRef] [PubMed]
- Giga, S.; Fletcher, I.; Sgourakis, G.; Vrkljan, B.H.; A Mulvaney, C. Organisational level interventions for reducing occupational stress in healthcare workers. Cochrane Database Syst. Rev. 2018, 2018. [Google Scholar] [CrossRef]
- Nielsen, K.; Abildgaard, J.S. Organizational interventions: A research-based framework for the evaluation of both process and effects. Work Stress 2013, 27, 278–297. [Google Scholar] [CrossRef]
- Moore, G.F.; Audrey, S.; Barker, M.; Bond, L.; Bonell, C.; Hardeman, W.; Moore, L.; O’Cathain, A.; Tinati, T.; Wight, D.; et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015, 350, h1258. [Google Scholar] [CrossRef] [PubMed]
- Semmer, N.K. Job stress interventions and the organization of work. Scand. J. Work Environ. Health 2006, 32, 515–527. [Google Scholar] [CrossRef] [PubMed]
- Biron, C.; Karanika-Murray, M. Process evaluation for organizational stress and well-being interventions: Implications for theory, method, and practice. Int. J. Stress Manag. 2014, 21, 85–111. [Google Scholar] [CrossRef]
- Havermans, B.M.; Mc Schlevis, R.; Boot, C.R.; Brouwers, E.P.; Anema, J.R.; Van Der Beek, A.J. Process variables in organizational stress management intervention evaluation research: A systematic review. Scand. J. Work Environ. Health 2016, 42, 371–381. [Google Scholar] [CrossRef]
- Dewa, C.S.; Loong, D.; Bonato, S.; Joosen, M.C.W. The effectiveness of return-to-work interventions that incorporate work-focused problem-solving skills for workers with sickness absences related to mental disorders: A systematic literature review. BMJ Open 2015, 5, e007122. [Google Scholar] [CrossRef]
- Arapovic-Johansson, B.; Wåhlin, C.; Hagberg, J.; Kwak, L.; Björklund, C.; Jensen, I. Participatory work place intervention for stress prevention in primary health care. A randomized controlled trial. Eur. J. Work Organ. Psychol. 2018, 27, 219–234. [Google Scholar] [CrossRef]
- Harvey, S.B.; Modini, M.; Joyce, S.; Milligan-Saville, J.S.; Tan, L.; Mykletun, A.; A Bryant, R.; Christensen, H.; Mitchell, P.B. Can work make you mentally ill? A systematic meta-review of work-related risk factors for common mental health problems. Occup. Environ. Med. 2017, 74, 301–310. [Google Scholar] [CrossRef] [PubMed]
- Nieuwenhuijsen, K.; Bruinvels, D.; Frings-Dresen, M. Psychosocial work environment and stress-related disorders, a systematic review. Occup. Med. 2010, 60, 277–286. [Google Scholar] [CrossRef] [PubMed]
- Karasek, R.; Theorell, T. Healthy Work; Basic Books: New York, NY, USA, 1990. [Google Scholar]
- Egan, M.; Bambra, C.; Thomas, S.; Petticrew, M.; Whitehead, M.; Thomson, H. The psychosocial and health effects of workplace reorganisation. 1. A systematic review of organisational-level interventions that aim to increase employee control. J. Epidemiol. Community Health 2007, 61, 945–954. [Google Scholar] [CrossRef] [PubMed]
- Holman, D.; Axtell, C. Can job redesign interventions influence a broad range of employee outcomes by changing multiple job characteristics? A quasi-experimental study. J. Occup. Health Psychol. 2016, 21, 284–295. [Google Scholar] [CrossRef]
- Siegrist, J. Adverse health effects of high effort-low reward conditions. J. Occup. Health Psychol. 1996, 1, 27–41. [Google Scholar] [CrossRef]
- Siegrist, J.; Li, J. Associations of Extrinsic and Intrinsic Components of Work Stress with Health: A Systematic Review of Evidence on the Effort-Reward Imbalance Model. Int. J. Environ. Res. Public Health 2016, 13, 432. [Google Scholar] [CrossRef]
- Pritchard, R.D.; Harrell, M.M.; DiazGranados, D.; Guzman, M.J. The productivity measurement and enhancement system: A meta-analysis. J. Appl. Psychol. 2008, 93, 540–567. [Google Scholar] [CrossRef]
- Pritchard, R.D.; Weaver, S.; Ashwood, E. Evidence-Based Productivity Improvement; Informa UK Limited: London, UK, 2012. [Google Scholar]
- Gaglio, B.; E Glasgow, R.; Brownson, R.C.; Colditz, G.A.; Proctor, E. Evaluation Approaches for Dissemination and Implementation Research. In Dissemination and Implementation Research in Health Translating Science to Practice; Oxford University Press (OUP): Oxford, UK, 2012; pp. 327–356. [Google Scholar]
- Damschroder, L.J.; Aron, D.C.; E Keith, R.; Kirsh, S.R.; A Alexander, J.; Lowery, J.C. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement. Sci. 2009, 4, 50. [Google Scholar] [CrossRef]
- Linnan, L.; Steckler, A. A. A process evaluation for public health interventions and research. In An Overview, in Process Evaluation for Public Health Interventions and Research; Linnan, L., Steckler, A., Eds.; Jossey-Bass: San Francisco, CA, USA, 2002. [Google Scholar]
- Proctor, E.; Silmere, H.; Raghavan, R.; Hovmand, P.; Aarons, G.; Bunger, A.; Griffey, R.; Hensley, M. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm. Policy Ment. Health Ment. Health Serv. Res. 2010, 38, 65–76. [Google Scholar] [CrossRef]
- Creswell, J.; Klassen, A.; Plano Clark, V.; Smith, K. Best Practices for Mixed Methods Research in the Health Sciences; National Institutes of Health; Office of Behavioral and Social Sciences Research: Bethesda, MD, USA, 2011.
- A Curry, L.; Krumholz, H.M.; O’Cathain, A.; Clark, V.L.P.; Cherlin, E.; Bradley, E.H. Mixed methods in biomedical and health services research. Circ. Cardiovasc. Qual. Outcomes 2013, 6, 119–123. [Google Scholar] [CrossRef]
- Palinkas, L.A.; Horwitz, S.M.; Green, C.A.; Wisdom, J.P.; Duan, N.; Hoagwood, K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm. Policy Ment. Health Ment. Health Serv. Res. 2013, 42, 533–544. [Google Scholar] [CrossRef] [PubMed]
- Graneheim, U.H.; Lindgren, B.-M.; Lundman, B. Methodological challenges in qualitative content analysis: A discussion paper. Nurse Educ. Today 2017, 56, 29–34. [Google Scholar] [CrossRef] [PubMed]
- Graneheim, U.; Lundman, B. Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Educ. Today 2004, 24, 105–112. [Google Scholar] [CrossRef] [PubMed]
- Mayring, P. Qualitative Content Analysis (28 Paragraphs). Forum Qualitative Sozialforschung/Forum: Qualitative Social Research. Available online: https://www.qualitative-research.net/index.php/fqs/article/view/1089/2385 (accessed on 2 September 2020).
- CFIR Memo Template. Available online: https://cfirguide.org/tools/tools-and-templates/ (accessed on 15 May 2019).
- CFIR Rating Rules. Available online: https://cfirguide.org/wp-content/uploads/2017/04/ratingrules10-29-14.pdf (accessed on 16 April 2020).
- LaMontagne, A.D.; Noblet, A.J.; Landsbergis, P.A. Intervention development and implementation: Understanding and addressing barriers to organizational-level interventions. In Improving Organizational Interventions for Stress and Well-Being. Addressing Process and Context; Biron, C., Karrinka-Murray, M., Cooper, C., Eds.; Routledge: Hove, UK, 2012; pp. 21–38. [Google Scholar]
- Woltmann, E.M.; Whitley, R.; McHugo, G.J.; Brunette, M.; Torrey, W.C.; Coots, L.; Lynde, D.; Drake, R.E. The Role of Staff Turnover in the Implementation of Evidence-Based Practices in Mental Health Care. Psychiatr. Serv. 2008, 59, 732–737. [Google Scholar] [CrossRef]
- Yazdani, A.; Wells, R. Barriers for implementation of successful change to prevent musculoskeletal disorders and how to systematically address them. Appl. Ergon. 2018, 73, 122–140. [Google Scholar] [CrossRef]
- Havermans, B.M.; Boot, C.R.; Brouwers, E.P.; Houtman, I.L.; Anema, J.R.; Van Der Beek, A.J. Process Evaluation of a Digital Platform-Based Implementation Strategy Aimed at Work Stress Prevention in a Health Care Organization. J. Occup. Environ. Med. 2018, 60, e484–e491. [Google Scholar] [CrossRef]
- Nielsen, K.; Nielsen, M.B.; Ogbonnaya, C.; Känsälä, M.; Saari, E.; Isaksson, K. Workplace resources to improve both employee well-being and performance: A systematic review and meta-analysis. Work Stress 2017, 31, 101–120. [Google Scholar] [CrossRef]
- Kivimaki, M.; Vahtera, J.; Pentti, J.; Thomson, L.; Griffiths, A.; Cox, T. Downsizing, changes in work, and self-rated health of employees: A 7-year 3-wave panel study. Anxiety Stress Coping 2001, 14, 59–73. [Google Scholar] [CrossRef]
- Dellve, L.; Strömgren, M.; Williamsson, A.; Holden, R.; Eriksson, A. Health care clinicians’ engagement in organizational redesign of care processes: The importance of work and organizational conditions. Appl. Ergon. 2018, 68, 249–257. [Google Scholar] [CrossRef]
- Ståhl, A.-C.F.; Ståhl, C.; Smith, P.M. Longitudinal association between psychological demands and burnout for employees experiencing a high versus a low degree of job resources. BMC Public Health 2018, 18, 915. [Google Scholar] [CrossRef]
- Quinlan, M. Organisational restructuring/downsizing, OHS regulation and worker health and wellbeing. Int. J. Law Psychiatry 2007, 30, 385–399. [Google Scholar] [CrossRef] [PubMed]
- Daniels, K.; Gedikli, C.; Watson, D.; Semkina, A.; Vaughn, O. Job design, employment practices and well-being: A systematic review of intervention studies. Ergonomics 2017, 60, 1177–1196. [Google Scholar] [CrossRef] [PubMed]
- Bal, P.M.; Dóci, E. Neoliberal ideology in work and organizational psychology. Eur. J. Work Organ. Psychol. 2018, 27, 536–548. [Google Scholar] [CrossRef]
- Kaltenbrunner, M.; Bengtsson, L.; Mathiassen, S.E.; Engström, M. A questionnaire measuring staff perceptions of Lean adoption in healthcare: Development and psychometric testing. BMC Health Serv. Res. 2017, 17, 235. [Google Scholar] [CrossRef]
- Semmer, N.K.; Jacobshagen, N.; Meier, L.L.; Elfering, A.; Beehr, T.A.; Kälin, W.; Tschan, F. Illegitimate tasks as a source of work stress. Work Stress 2015, 29, 32–56. [Google Scholar] [CrossRef]
- Anell, A.L.; Glenngård, A.H.; Merkur, S. Sweden health system review. Health Syst. Transit. 2012, 14, 1–159. [Google Scholar]
- Gogievski, M.J.; Hobfoll, S.E. Work can burn us out or fire us up: Conservation of resources in burnout and engagement. In Handbook of Stress and Burnout in Health Care; Halbesleben, J.R.B., Ed.; Nova Science Publishers, Incorporated: New York, NY, USA, 2008; pp. 7–21. [Google Scholar]
- Carayon, P.; Bass, E.J.; Bellandi, T.; Gurses, A.P.; Hallbeck, M.S.; Mollo, V. Sociotechnical systems analysis in health care: A research agenda. IIE Trans. Health Syst. Eng. 2011, 1, 145–160. [Google Scholar] [CrossRef]
- Connell, R.; Fawcett, B.; Meagher, G. Neoliberalism, New Public Management and the human service professions. J. Sociol. 2009, 45, 331–338. [Google Scholar] [CrossRef]
- Havermans, B.M.; Brouwers, E.; Hoek, R.J.A.; Anema, J.R.; Van Der Beek, A.J.; Boot, C.R. Work stress prevention needs of employees and supervisors. BMC Public Health 2018, 18, 642. [Google Scholar] [CrossRef]
- Lesener, T.; Gusy, B.; Wolter, C. The job demands-resources model: A meta-analytic review of longitudinal studies. Work Stress 2018, 33, 76–103. [Google Scholar] [CrossRef]
- Russell, J.; Berney, L.; Stansfeld, S.A.; Lanz, D.; Kerry, S.M.; Chandola, T.; Bhui, K. The role of qualitative research in adding value to a randomised controlled trial: Lessons from a pilot study of a guided e-learning intervention for managers to improve employee wellbeing and reduce sickness absence. Trials 2016, 17, 396. [Google Scholar] [CrossRef] [PubMed]
- Kirk, M.A.; Kelley, C.; Yankey, N.; Birken, S.A.; Abadie, B.; Damschroder, L.J. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement. Sci. 2016, 11, 72. [Google Scholar] [CrossRef] [PubMed]
- Jensen, I.; Brämberg, E.B.; Wåhlin, C.; Björklund, C.; Hermansson, U.; Karlson, M.L.; Elinder, L.S.; Rosenschöld, P.M.A.; Nevala, T.; Carter, N.; et al. Promoting Evidence-Based Practice for Improved Occupational Safety and Health at Workplaces in Sweden. Report on a Practice-Based Research Network Approach. Int. J. Environ. Res. Public Health 2020, 17, 5283. [Google Scholar] [CrossRef] [PubMed]
|Problem and Evidence Base||Resources||Activities/Strategies/Core Components of ProMES||Method||Short-Term Outcomes||Medium-Term Outcomes||Long-Term Outcomes|
|Preconditions||Management||0. Activities before starting ProMES||Checklists|
|Some work-related factors are known to be risk factors for ill-health: absence of influence and control, insufficient interaction with co-workers, unclear and conflicting tasks, insufficient participation in decision-making, low esteem reward, and insufficient feedback||Experienced ProMES fascilitator|
Multi-disciplinary research team
|1. Initial meetings|
2. Form design team
3. Identify objectives = identify results
(Input from the whole unit. Decision making process is discussion to consensus)
4. Develop indicators=operationalize results
5. Approval from management
6. Develop contingencies = operationalize relation between result and evaluation
7. Approval by management
8. Gather indicator data
9. Develop feedback reports
10. Conduct feedback meetings = information on results and evaluations of results for the determined period of time and if needed improvement/development of new strategies
11. Monitor project over time
|(1) Participative decision making with regard to demands, i.e. which work results need to be improved and how much;|
(2) involvement in the development of the evaluation system, thereby increasing the likelihood of perceiving it as accurate and fair;
(3) timely feedback about the results of one’s efforts that can be thought of as a positive reinforcement;
(4) information sharing; and
(5) discussion of problem solving and work strategies
|Controllable and acceptable demands.|
Higher control through higher decision authority.
Higher reward through
extrinsic recognition (from pears and management).
Better social support (coworkers and supervisors).
Other possible mechanisms:
Justice (measurement system perceived as fair).
Self-efficacy: Confidence in the group’s ability to take action and overcome barriers.
|Primary outcome: Lower levels of job strain.|
Lower effort-reward imbalance.
Better sleep and recovery.
|Lower occurrence of common mental disorders such as: |
Depression and exhaustion (secondary outcomes).
|Process Evaluation Items||Examples of Specific Questions||Methods of Measurement|
|Recruitment||Who recruited the participating units and how?|
What were the reasons for participation and declining participation?
What characterizes units that declined to participate?
|Administrative data, project logbook.|
|Dose delivered||Frequency and duration of meetings.||Administrative data, poject logbook.|
|Reach||What percent of employees in the participating unit participated in ProMES? |
How would you describe your participation with the ProMES?
Have you participated in ny design team? Yes/No
|Administrative data, project logbook, 2 questions from questionnaire.|
|Fidelity and adaptation||To what extent was ProMES implemented as intended (according to manual)?||Project logbook, checklists, information from interviews.|
|Mechanism of impact|
|Participant responses||How satisfied are the employees with the content, work procedures, delivery. Their opinion of ProMES’s appropriateness/usefulness, acceptability, feasibility, sustainability.||Questionnaire.|
|Context||Which factors/circumstances have either facilitated or hindered working with ProMES?||Semi-structured interviews.|
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).