Attending a Blended In-Service Management Training in a Public Health System: Constraints and Opportunities for Pharmacists and Health Services

Management and public health are important domains of competency for pharmacists. In about 90% of Brazilian health departments, pharmacists manage the selection and purchase of medicines. The Pharmaceutical Services and Access to Medicines Management Course (PSAMM) was offered to pharmacists working in the public health system. The aim of this study is to analyze the impacts of the course as perceived by the students (pharmacists). Two thousand five hundred pharmacists completed the course. It is a mixed-methods study, including subscribing forms (n = 2500), evaluation questionnaire (n = 1500), focus groups (n = 10), and semi-structured interviews (n = 31). Participants reported a high level of satisfaction with the course; they considered to have developed competencies related to leadership and management, competencies needed to enhance and sustain their practices in health services. Data analyses showed important barriers to complete the course: high course workload, poor quality of Internet access, lack of support from the health services. Participants highlighted crucial features of the course that helped them develop key competencies: practical in-service activities, useful and realistic contents, tutoring. These features helped participants overcome some important constraints described by them. The educational model described in this study was perceived as having a long-term impact on their behaviors and management practices in health services.


Introduction
Access to medicines is an essential component to achieve universal health coverage [1]. However, access to medicines and their rational use are persistent global concerns. These issues have a significant impact on the health system's quality and, consequently, on health [2]. The expansion of access to medicines was listed as one of the ten most prominent problems that demand attention from the World Health Organization [3]. It is necessary to address this issue from a health system perspective, considering the complex relationships between medicines and health financing, human resources, health information, and the broader issue of access to health services and interventions [1,4].
In Brazil, access to medicines is a responsibility defined by public policy. The National Pharmaceutical Policy defines services and responsibilities at all levels of governance of the unified health system (Sistema Único de Saúde-SUS) to guarantee access to medicines and pharmaceutical services.
With the decentralization of the Brazilian health system, the federal government is responsible for the general policies and guidance strategies. The 5560 municipalities, for their part, undertake a series of responsibilities that demand the mobilization of knowledge The PSAMM targeted pharmacists (government employees), and it was offered free of charge through a blended approach composed of mostly e-learning and four faceto-face meetings. Besides financing pharmacist training during the same period, the Brazilian Ministry of Health was supporting other actions, such as infrastructure and workforce expansion to develop pharmaceutical services and access to medicines in diverse municipalities [19].
The PSAMM course had a total number of 4244 students enrolled. By the end of the course, 2500 students completed the course between 2012 and 2016 (1445 pharmacists in class A and 1055 pharmacists in class B); 303 gave up, and 1441 students failed the course. It is important to highlight that the results presented here include only students that completed the course. These pharmacists work in 1068 of the 5570 Brazilian municipalities, located in all the 27 federal states.
The PSAMM curriculum was prepared by 50 experts from 20 institutions from all over the country. The course didactic material was produced in printed and online versions for the Moodle platform [20]. Asynchronous activities such as reading texts, cartoons, video documentaries, and forums for each topic were offered. Every student needed to develop a training activity in service called the operative plan, based on Carlos Matus' principles of strategic situational planning [21]. The aim of the operative plan was to develop management skills in real and current health services where the pharmacists work. These operative plans were available for the health centers where they were developed, and, in many cases, they were put into practice as reported by Portela et al. [22].
The PSAMM total workload was 375 h, distributed over three main themes, and carried out over 15 months (64 weeks). The curriculum is described in Table 1 [16]. Students were divided into 31 regional centers located in 17 Brazilian states, where the evaluations and four face-to-face meetings were held. The tutors (also pharmacists) were experienced professionals from all regions of the country and linked to the regional course centers. The course had 132 peer tutors (25 students per group). By the end of the course, each student was expected to acquire: knowledge about management and pharmaceutical policy, self-confidence in developing management functions, motivation to work, and improved skills and behaviors that increase the managerial capacity of pharmaceutical departments [23,24].

Study Design
Considering a variety of data sources related to the students' perceptions of the PSAMM course's features and possible impacts, the study design took into account welldocumented and extensively used models for evaluating continuing education in many educational contexts. The Kirkpatrick model [25] proposes four distinct levels as a sequence of ways to evaluate an educational program: (1) reactions (satisfaction with the learning intervention), (2) learning (the extent to which professionals change or improve attitudes, knowledge, skills, and/or self-efficacy as a result of attending the educational interventions), (3) behavior (the extent to which professionals' learning was translated into their post-learning behavior or their professional performance), (4) results (can be seen as patients' health outcomes resulting from the influence of learning interventions on professionals' behavior changes). In the present study, results are considered the changes in management practices in place in the health services in which the students work.
The evaluation of educational interventions to effectively change professional behavior can be seen within the behavior change theories. The theoretical domains framework can be mapped in three domains: capability (the individual's psychological and physical capacity to engage in the activity concerned-it includes having the necessary basic knowledge and skills); motivation (all those brain processes that energize and direct behavior, not just goals and conscious decision-making); opportunity (all the factors that lie outside the individual that make the behavior possible or prompt it) [26]. This model forms part of the behavior change wheel proposed by Michie et al. [27].
This study was designed with the assumption of the Brazilian Policy for health professionals' permanent education in the SUS [28], i.e., that students, educational intervention, and workplace are all parts of a continuing education strategy. To perform the evaluation, the Kirkpatrick model helped define categories related to the satisfaction with the educational intervention; the perceived impact on students (related to their gain in knowledge and changes in professional behaviors); and the perceived impacts on health services management. The theoretical domains framework brought up the category of the students' characteristics, particularly those related to the domains of capability, motivation, and the evaluation of the course associated with the domain of opportunity.
Taking such assumptions into account, the study groups the data into three categories: (a) students' sociodemographic, educational, and professional characteristics; (b) evaluation of the infrastructure, contents, and teaching methods; (c) perceived impacts on the students' behaviors and the management practices in health services.

Data Collection and Analysis
A mixed methods research agenda was followed to facilitate the inclusion of multiple students' perspectives about the possible impacts of the course for them and for the health services in which they worked, including four data sources and data collection methods at different times, as described in Table 2. The partial studies were conducted separately, and the integration was performed at the interpretation phase only.

Subscription Form
The analysis of the students' registration forms on the PSAMM Course used the following data: age group, gender, distribution of graduates by geographic region, Type of hiring and government level. The data were processed using Microsoft Excel ® 2010 software and analyzed using the dynamic table tool. This study considered data from 2500 graduates.

Likert-Scale Questionnaire
A questionnaire with closed questions was prepared based on the model proposed by Ruggeri et al. [29] for e-learning programs for health professionals. The adaptation and usefulness to the didactic resources of the Moodle platform, the technical and personal facilities and difficulties perceived for the incorporation of online activities, and the perception of learning management concepts and their applicability in the work process were evaluated. For the analysis, a 4-point Likert-scale was employed (I totally agree/I agree/I do not agree, nor disagree/I disagree). One thousand five hundred students from class B responded to the questionnaire during the last face-to-face meeting in 2014. They received printed copies of the questionnaire and answered it anonymously. The questionnaire was previously validated by the tutors.
The analyses were performed using descriptive statistics in the STATA program-SPSS-Version 2014 (College Station, TX, USA) Association analyses were performed. These were expressed through frequencies and central tendency measurement tests by Brazil's geographic regions, through exact Fisher's test. Differences of p < 0.05 were considered statistically significant.

Focus Groups and Semi-Structured Interviews
Qualitative data collection happened through focus groups and individual interviews. It happened between August and October of 2018 (on average, three years after the students had finished the course). This last qualitative data collection intended to identify whether the results found at the end of the course were sustained. It is also meant to determine the context in which the students perceive the course's late results.
It was carried out in 10 states across the country. The criterion for the selection of the states for the data collection was the density of resident students. The interviews took place in the state capitals and, whenever possible, in 1 smaller municipality in each state.
Ten students were selected in each state for the focus groups, and three to be individually interviewed. The students were randomly selected from the list of students from each state. They were assigned to the interviews or to the focus groups according to their availability to participate in the given activity. The students were contacted by phone and email. Whenever necessary, new students were randomly selected and contacted until the desired sample was achieved.
Guiding questions for the focus groups were used to address topics like the students' experience attending the continuing education course and working at SUS, and the impact of the course for them and for the health services. Ten focus groups were carried out in the capital cities and lasted for around 1 h and 30 min. Each focus group had an average attendance of 6 participants.
The interviews addressed topics such as the choice of attending the course, the course structure and activities, the identification of changes in professional practices and work processes due to the course. Thirty-one interviews were conducted in 14 municipalities.
All focus groups and interviews were recorded and transcribed.
To safeguard participants' identities, they were identified in the individual interviews by the letter E-for "interview", in Portuguese-followed by the acronym for their geographical region (EN-interview conducted in the North region, ENE-Northeast, ECO-Center-West, ES-South, ESE-Southeast). Focus groups were identified as GFfor "focus group", in Portuguese-followed by a number to differentiate the ten groups, e.g., GF1.
Two researchers (AAL and FM) reviewed all notes from the field, transcribed audiorecordings, and read these materials repeatedly before generating an initial codebook. This process allowed the researchers to gain a holistic understanding of the data and increase familiarity with the content, both manifest and latent. An inductive approach was used to conduct content and thematic analyses: identification of all relevant concepts and examples expressed; indexation of highlighted excerpts by theme and correlation with others; rearrangement of the highlighted excerpts, considering their contexts and specificities, in a coherent and understandable flow; mapping of the concepts, scope, and nature of the studied phenomenon. A deductive approach was then applied to identify manifest content related to the impacts on students' behaviors and changes in service management practices, connecting the themes to a theoretical framework [30,31].
NVIVO 10 (Mairena del Aljarafe, Sevilla, Spain) was used to manage the data during the coding process and help organize themes. The highlighted excerpts per category were indexed and, considering their contexts and specificities, selected to exemplify each category.
Finally, the results of each partial study, including quantitative and qualitative finds, were analyzed together to identify possible convergent and divergent results and complementary finds. The discussion and conclusions consider the whole set of results.

Ethical Considerations
This study obtained permission from the Ethics Committee on Research with Human Beings (CAAE: 46912815.0.0000.0121). All participants were asked to provide written consent. The information was anonymized, and no personal identifiers were used during data analysis or publication.

Students' Sociodemographic, Educational, and Professional Characteristics
The profile of students who completed the course is shown in Table 3. Most students are women (77%) and are between 25 and 39 years old (68%). More than 60% had already completed some postgraduate studies before taking the course, predominantly specialization courses, and 55.4% reported having no previous experience with the use of distance education in continuing education. A total of 75.2% point out flaws in their undergraduate courses for not focusing on SUS.
The Southeast region has the highest concentration of students who completed the course (36.7%), followed by the Northeast region (26.3%). Among the students, working at the municipal level (76.3%) and being hired through public examinations (73.5%) predominate. Most students had one job (51.1%), but 10.4% had three or more job contracts. This is related to the fact that 38.6% work more than 40 h a week.
The majority considered the workplace structure to be inadequate (64%) and considered the quality of Internet access at work to be good (51.6%). However, 35.3% consider access to be precarious or state they do not have access to the Internet.
The variables age group, gender, distribution of students by region, type and government level of employment refer to the 2500 students who completed the course. The educational level variables, perceived level of previous training focused on SUS, the number of work contracts, weekly workload, quality of Internet access in the workplace, the structure of the workplace, and previous use of distance education refers to 1500 students in class B.

Evaluation of the Infrastructure, Contents, and Teaching Methods
In total, 966 questionnaires of 1500 administered were considered valid, 44 (4.5%) in the North Region, 364 (37.8%) in the Northeast Region, 122 (12.6%) in the Midwest Region, 309 (32.0%) in the Southeast Region, and 127 (13.1%) in the South Region. The results are presented in Table 4. Students from all regions rated the course very positively. The operative plan (an in-1500 service hands-on activity that was mandatory for all the students) and the tutoring in small groups of students (the same tutor following the group throughout the course) stood out as the most useful and important resources for learning. In all the country regions, the rate was high, but for the operative plan and the tutoring, there were significant differences in rates between regions.
The results of the assessment of infrastructure, contents, and teaching methods from the perspective of PSAMM Course students in focus groups and interviews are presented in Table 5. There are 2215 min of audio recordings. Lack of work support "I was allowed to participate in the face-to-face meeting, but I had a discount on the wages of the double hours. You have a policy that encourages you, you have the Ministry of Health that offer, but they have internal legislation that makes it difficult" [GF08] Course configuration to overcome the barriers encountered Course organization adapted to the regional needs "And I thought the most interesting thing was that you had this act of care with the students [providing the materials in a DVD/flash drive], understanding the situation we live in with the difficulty to access the internet." [GF01] Decentralized coordination in regional centers "We had very good support here in our town. The tutors and regional coordinators would help us when we had difficulties " [EN5] Tutoring "So, the tutoring part was very important, it directed us a lot and consolidated the thoughts and actions." [ENE7] Face-to-face meetings "The face-to-face meetings were important to clear up doubts, ask questions. There were teachers from the region and from outside. There were those exchanges." [GF8]

Analytical Categories Themes Selected Quotes
Quality of teaching material "I liked the teaching material very much. It is a material that I use to this day, to study, to teach classes. I've used it many times. That's a point that I consider extremely positive in the course." [GF09] "I had everything I needed available on the platform, so it met my needs." [ES4] In-service activities-operative plan "It pushed us to build a way, make a diagnosis, discuss with all stakeholders, design proposals, revise proposals, listen. Apply the management tools." [ENE6] "If it were just theoretical content, it would not achieve the objective as it did, at least for me. Because theoretical content you see in any postgraduate course." [ENE10] Flexibility of activities "My concern when I started was in relation to texts, I imagined that because it is e-learning, classes could be monotonous, with only texts to read. I have already taken courses like this, and I was very positively surprised by the layout of (Moodle), it made me want to continue studying." [PL6] The themes identified among the categories of analysis highlight important motivators to attend the course, as well as constraints that made it difficult to complete it, including characteristics of the course, but also of the workplaces and the environment. The themes that emerged in the category "configuration of the course to overcome the barriers encountered" describe the course's characteristics that contributed in a crucial way so that students could complete the course.

Perceived Impacts of the PSAMM Course on the Students' Behaviors and on Management Practices in Health Services
The closed questionnaire results are presented in Table 6, with 966 responses considered valid (the same as item 3.2).
Students from all regions assessed that the course provided behavioral changes for them and positive changes in management practices in their workplaces. There was a significant difference between the regions and an average disagreement of 5.4% about the conditions to develop leadership in their workplace.
The results of the evaluation of the impacts of the course from the perspectives of PSAMM Course's students obtained in the focus groups and in the interviews are presented in Table 7.   Table 7. Analysis of focus groups and interviews on perceived impacts of the PSAMM course on the students' behaviors and on management practices in health services.

Perceived Impacts Theme Quotes
Changes in student attitudes/practices due to the course Increase in Confidence "I am confident, I have more autonomy. The knowledge acquired was very important to consolidate the actions within the pharmacy, for nursing and for the direction of the service. I have more knowledge and the actions I propose here are more strengthened" [ENE6] Communication capacity "We begin to see other possibilities, to listen to others. In management, it is too easy to make a decision disconnected from reality, but when we go to the service and look at the factual reality, we notice the need to put ourselves, at least a little, in other people's shoes. Leadership capacity "I went from being the pharmacy girl to effectively being the pharmacy service manager." [GF08] Change in the understanding of the role of the manager and of management "We stopped putting out fires on a daily bases to effectively become a work of management, of planning, in which you, according to the scenario in front of you, starts to program, plan, and look ahead in that scenario." [ESE5] Benefits for health services Negotiation with hierarchical superiors "To better negotiate with the suppliers, I needed access to the City Hall's financial system. So, I negotiate the permission to access with the health secretary and the management secretary, and got good results." [ECO3] Interprofessional collaboration "We are pharmacists, and we work with nutrition, and we had never consider looking for a nutritionist to write the proper description of a product in an auction notice. So, we established a partnership with the nutritionists of the companies that already took part in the process to help us with the reference terms." [ECO3] Actions integrated with the popular councils of health "Participating in public audiences, participating in the Municipal Health Plan, participating in committees, all of this came after the course." [ES3] Recognition and integration of the Pharmaceutical Department in the Health Secretariat "We used to go to the administrative meetings but when there was a more technical meeting sometimes the pharmacy was forgotten. So, we make them remember us and started to be more informed of the routines of the unit, of how things worked the interaction with the other professionals improved very much and this was also a benefit to us. To the service."

Discussion
The evaluation of the course from the perspective of the students brings insights on strategic issues for the offer of continuing education (CE) for health professionals who are responsible for the practical implementation of an important public policy in the country.
Although the evaluation of the course was generally positive, students described some difficulties in attending it. The profile of students presents the same characteristics of the professional category in the country [32,33] and helps to understand the constraints found. The perceived precarity in public health contents during the undergraduate course and the rates of students who have already taken some kind of specialization or post-graduation courses to demonstrate some capacity, as well as motivation, to seek training and professional improvement. Regarding the opportunity to change professional behavior, considering the CE as a key element of this process, Tables 3 and 5 bring essential elements that indicate substantial limitations for students related to workload and workplaces with no or precarious Internet access and support for CE.
Even though e-learning has provided a very important opportunity for these professionals to be able to access CE while staying in their own cities, access to good quality Internet (and sometimes also electricity) is still a challenge in remote regions [34]; as described, for instance, by students from the Amazon region. It is worth highlighting that CE attendance barriers for pharmacists worldwide are lack of time [35], cost, workplace, personal factors, and significant constraints related to geographical access [36].
In addition to the country's structural difficulties, students also realized that there is a low incentive for education on the part of health institutions for pharmacists, which constrains opportunities for behavioral changes. This finding is corroborated by data found in the literature, which suggests that training for skills development, such as leadership, focuses mainly on doctors, nurses, and public health specialists [15].
In this sense, the pharmacists' path in the search for better training goes through obstacles. As shown in the students' profile, several pharmacists face a high workload, with two or three jobs as a barrier to the search for professional development. Data available in the literature [15] suggest that health professionals do not have protected time for professional training, which requires the use of resources such as distance learning. As a result, students take this responsibility individually; professional development invades their personal lives, which results in work overload.
The evaluation of the infrastructure, contents, and teaching methods of the PSAMM Course indicates that some characteristics of the course provided opportunities to overcome the constraints found in attending the course and also to sustain the results even three years later (Tables 4, 5 and 7). They highlighted the importance of close tutoring with peer feedback, face-to-face meetings, and hands-on in-service activities to promote learning opportunities. Among the results in Table 5, it is possible to identify the importance of strategies to bring the course closer to the reality experienced by students and their needs, such as regional centers, contents related to the particularities of each region, and flexibilities related to accessing the platform and completing the proposed activities.
Forums were the teaching methods with the lowest evaluation rates (Table 4) and were not mentioned by students in the qualitative research years later. Some authors suggest that the distance modality may be more suitable for some students than for others, pointing to the blended method (online and face-to-face) as a better choice [29,37,38]. The results reported here show that the use of e-learning proved to be adequate, considering the particularities already mentioned. However, they point out the importance of face-to-face meetings during the course, suggesting that the blended model, with decentralized centers and regional tutoring, provides better conditions for learning. Even in the North (Amazon) and Northeast regions, where the infrastructure of health services and Internet access are of lower quality, students positively evaluated the characteristics of the course. Such positive characteristics are revealed in their impact on students and services.
Students perceived the impacts of the course at all levels defined by Kirkpatrick: from the satisfaction with the educational intervention to the impacts on their levels of knowledge, going through behavioral changes and the reflexes in the management practices of the services where they work (Tables 6 and 7). It is crucial to highlight that the students noticed positive results when they were still in the last stage of the course and that these results persisted and became clearer after about three years of the course's completion. This result allows us to conclude that the PSAMM course brought a good and sustained return for the investment made by the Ministry of Health.
In this study, it is possible to find evidence of consolidation of competencies, such as leadership and communication, for example, indicated by the FIP as necessary competencies for the pharmaceutical professional. These are fundamental to overcome the barriers found in the health system and to strengthen the role of the pharmacist in public health. The perception that the course had brought students better conditions to exercise leadership and conduct management tasks at the end of the course was, years later, consolidated into the concrete performance of these important competencies for a health services manager.
The changes reported by students in their behavior are in line with their reports about changes in service management practices, particularly in the use of informationbased negotiation, with greater confidence and a broader view of the health system and how to overcome the barriers of hierarchy. The reports reveal the adoption of desirable and important management practices for the sustainability of health services, such as interprofessional collaboration and close relationship with popular local health councils, as well as strategic planning and agreement on actions related to access and rational drug use. These changes result in greater visibility and authority for the pharmaceutical sector within the healthcare system and for the pharmacist.
These impacts, however, happened and were reported at different intensities among the participants in this study. Contextual and policy issues related to local health services constrained the full development of behaviors and practices, as evidenced by the reported difference between the regions of the country in the ability to exercise leadership in their workplace ( Table 7). The development of the operative plan was also constrained in some of the workplaces, particularly in regions North, Northeast and Midwest. Consequently, some students found the operative plan was not useful for them (Tables 4 and 6). Hence, the intensity with which students report the achievements after the course must be weighted considering the context in which they attended the course and work.
As its main limitation, this study has an evaluation based on the perception reported only by pharmacists who completed the course, not including other points of view, such as managers and patients. Even so, the evidence presented here indicates that the provision of CE for pharmacists working in the public health system produces important and positive impacts for them and for health services.
The study also reveals issues that need to be considered when offering CE for health professionals. Students, educational intervention, and the workplace are drivers of the impacts of the CE. The Brazilian Policy for health professionals' permanent education in the SUS [28] defines that educational interventions for health professionals must be understood and developed as a health system's investment: therefore, educational activities must be part of health workers' work, with the aim of achieving the health system's benefits. Considering this assumption, the workplace support to develop educational intervention is an essential constituent of the CE strategy.
CE proposals must be attentive to the context of the target audience, proposing flexible strategies that make sense with the students and the local health system's reality. New investments in continuing education should focus primarily on hands-on educational strategies with activities in the workplace, with compatible theoretical load, using the available resources [15], and meeting the needs of students; in a way that enables the development of the course at the service location and applying guidelines such as those reported by Ramani et al. [39] for the planning, implementation, and evaluation of the educational interventions in the local context.

Final Remarks
The set of results demonstrate the intrinsic relationship between the characteristics of students, the characteristics of educational intervention, and the impacts perceived by professionals/students in the practical reality of health services. Based on students' perception, future educational proposals should rest primarily on strategies with a practical focus related to local services and pharmacists' needs. Educational interventions' design and health management must support pharmacists in overcoming the various constraints to attend CE.
The educational model described in this study had a long-term impact on students and services. As part of the Brazilian Ministry of Health investments to develop pharmaceutical services and access to medicines in diverse municipalities, continuing pharmaceutical education promotes important benefits in terms of pharmacists' professional behaviors and management practices in health services. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.