Distributing Publicly-Funded Influenza Vaccine—Community Pharmacies’ Perspectives on Acquiring Vaccines from Public Health and from Private Distributors in Ontario, Canada

Objectives: To explore community pharmacies’ experience with two models of distribution for publicly-funded influenza vaccines in Ontario, Canada—one being publicly-managed (2015–2016 influenza season) and one involving private pharmaceutical distributors (2016–2017 season). Methods: Online surveys were distributed to community pharmacies across Ontario during the 2015–2016 and 2016–2017 influenza seasons with sampling proportional to Ontario Public Health Unit catchment populations. Quantitative data were analyzed descriptively and inferentially and qualitative data were summarized for additional context. Results: Order fulfillment appeared more responsive with the addition of private distributors in 2016–2017, as more pharmacies reported shorter order fulfillment times (p < 0.01); however, pharmacies reported significantly more days with zero on-hand inventory in 2016–2017 (p < 0.01), as well as more instances of patients being turned away due to vaccine unavailability (p < 0.05). In both seasons, a similar proportion of pharmacies reported slower order fulfillment and limited order quantities early in the season. Improved availability early in the season when patient demand is highest, more vaccines in a pre-filled syringe format, and better communication from distributors on product availability dates were recommended in qualitative responses. Conclusions: Introducing private distributors for the management and fulfillment of pharmacies’ orders for the publicly funded influenza vaccine appeared to have mixed results. While key concerns surrounding the frequency, responsiveness, and method of delivery were addressed by this change, challenges remain—in particular, acquiring sufficient vaccine early in the season to meet patient demand. As pharmacies become more prominent as vaccination sites, there are several opportunities to ensure that patient demand is met in this setting.


Introduction
Historically, publicly funded vaccinations in Canada were completely managed within the public sector, from ordering and distribution through administration by public health departments or physician offices. In recent years, multiple Canadian jurisdictions have outsourced components of their publicly funded vaccination strategies to the private sector, including parts of vaccine distribution and administration. For example, most jurisdictions have introduced regulations that grant pharmacists vaccine administration authority, and while there is considerable heterogeneity, there is a trend towards public payers remunerating pharmacists for their administration of publicly funded vaccinations [1][2][3]. The earliest and the most widespread public-private partnership (PPP) taking advantage of pharmacists as immunizers in Canada has involved integrating their services with annual influenza vaccination programs [4]. The first Canadian province to build such a relationship with pharmacists was Alberta in 2007, with several other provinces making pharmacists part of their publicly-funded influenza programs in subsequent years [4]. As a result of these relationships, pharmacies are now the primary site for influenza vaccinations across Canada, and in Ontario-the most populous jurisdiction-the involvement of pharmacists as influenza immunizers since 2012 has led to improved influenza vaccination rates and a predicted societal net positive financial impact [4][5][6].
When pharmacists were introduced into Ontario's Universal Influenza Immunization Program (UIIP) in 2012, the decision was made to use the existing government-managed influenza vaccine supply chain-coordinated and executed by the Ontario Government Pharmaceutical and Medical Supply Services (OGPMSS). From 2012 to the end of the 2015-2016 influenza season, all healthcare providers with Toronto postal codes (Ontario's largest center, population~3 million) had their vaccines delivered by the OGPMSS; in areas outside of the specified Toronto postal codes, vaccines were transferred to regional Public Health Units (PHU) throughout the province, and healthcare providers coordinated pick-up or delivery of their vaccine orders from their PHU. This system was revised prior to the 2016-2017 influenza season, with major changes affecting pharmacies outside of Toronto. Under the new system, all healthcare providers, including pharmacists, within Toronto continue to receive their vaccines directly from the OGPMSS, but for community pharmacies located outside of specified Toronto postal codes, distribution of influenza vaccines is now coordinated through approved pharmacy distributors. Private pharmaceutical distributors delivering publicly-funded vaccines to community pharmacists represents the PPP of interest, as there appears to be limited work in this space, particularly from the perspective of an end-user (i.e., immunizers, including physicians, pharmacists, etc.). It appears that most work exploring the outsourcing of vaccine delivery logistics has thus far focused on optimizing national health systems; within this work, there appears to be no Canadian or North American accounts of outsourcing in this way [7][8][9][10][11]. Further, the work that has explored immunizers' experience with the logistics of vaccine delivery has not had an opportunity to explore the differences between public or privately-run delivery strategies-instead, largely focusing on satisfaction with an existing system or challenges faced in a particular season [12,13]. Finally, there appears to have been no published work aimed at understanding pharmacists' experience with vaccine delivery strategies to date. The work presented here has an opportunity to contribute novel insight given its unique timing and perspective.
This work aims to explore the impact of introducing private pharmaceutical distributors as third-party logistics providers to community pharmacies in the Ontario influenza vaccine supply chain. Of particular interest were community pharmacies' perspectives on: whether the supply of influenza vaccines was adequate to meet patient demand; how the mechanics of vaccine distribution may impact patient access to vaccinations; and opportunities for continued improvement in vaccine distribution for this setting. This data are particularly pertinent today, as there remain important knowledge gaps that could inform the redesign of vaccine supply chains-particularly those that involve PPPs. Previous work has acknowledged that clinician input could enhance the structure and function of PPPs [14], and understanding front-line pharmacists' experience with these systems could inform the roll-out of pandemic vaccines-including COVID vaccines-that pharmacists may provide to the public.

Study Design
This project involved two cross-sectional online surveys, delivered via Qualtrics survey software (Qualtrics, Provo, UT, USA). The first survey took place before the changes to Ontario's influenza vaccine distribution strategy and captured the 2015-2016 influenza season; this survey was distributed in February 2016 and received responses through the end of March 2016. The second survey took place after the aforementioned changes to distribution were implemented for the 2016-2017 influenza season; this iteration was distributed in March 2017 and continued until May 2017 (Data collection took place later in the 2016-2017 season as there was a longer influenza season [15,16] and the intention was for respondents to provide a complete picture of their 2016-2017 experience). Study protocol and methods received ethics approval from the University of Waterloo Office of Research Ethics (ORE #30749).

Data Collection
Survey questions were developed by pharmacy practice researchers, with input from practicing community pharmacists and pharmacy advocacy organizations. Questions focused on the logistics of ordering and receiving the seasonal influenza vaccine, including differences among the various vaccine formats (e.g., pre-filled syringes, multi-dose vials, and intranasal formulations), and the performance of the vaccine distribution system being used in that season. Open-ended survey questions were included to provide a more in-depth understanding of the quantitative data, particularly in the areas of pharmacists' overall perceptions of the distribution system, areas for improvement and distributor performance, such as why order fulfillment times were delayed or partial orders were received.

Sampling
Participant recruitment was based on the Ontario College of Pharmacists' publicly available register of community pharmacies and their staff. Sites that have accreditation as community pharmacies but operate in another setting (e.g., hospital pharmacies or Canadian Forces' pharmacies) were excluded. Pharmacies were categorized by Ontario PHU (n = 36) using the University of Waterloo Geospatial Centre resources. Sampling of pharmacies was then performed by first selecting the number of pharmacies to sample within each PHU proportional to its catchment (population per PHU/provincial population * number of community pharmacies in Ontario), and then selecting pharmacies randomly within each PHU. The aims of this approach were to ensure proportional representation of pharmacist respondents relative to the Ontario population, and to ensure representation from each PHU in the sample (i.e., Toronto is served by one PHU and represents 20% of the population, and thus a proportionate sample of pharmacists were asked to complete the survey). Selected pharmacies were screened using phone calls to determine if the pharmacy administered influenza vaccines as part of the UIIP and to identify the individual responsible for ordering the vaccine for their pharmacy; these individuals were asked to complete the survey through a Qualtrics link emailed directly to them. The same pharmacies who were sent the survey in the 2015/2016 influenza survey were re-surveyed in the 2016/2017 influenza season; however, data was not matched year-over-year as no identifying data were collected to ensure anonymity. In addition, the authors were unaware that a change would be made to the vaccine distribution system when they designed the 2015-2016 survey (thus data that could be used to match participants in future iterations were not collected in the 2015-2016 survey). In 2016-2017, it became clear that geographic information was important for understanding how changes to the distribution system affected the experience of pharmacies outside of Toronto postal codes; therefore, the first 3 digits of participants' postal codes (i.e., Forward Sortation Area (FSA)) were collected in the 2016-2017 iteration of the survey. The use of internet protocol (IP) addresses captured by Qualtrics to match or geolocate respondents was explored; however, would require several assumptions, including that participants responded from a static IP address, rather than the more common dynamic IP addresses, using the same device at the same location in each year. Moreover, a protocol for excluding participants using a virtual private network (VPN) was not used, because survey links were emailed directly to registered Ontario pharmacists who had been screened over the phone. The procedures used to manage this unpaired data are described below.

Data Analysis
Data were not paired by participants or pharmacies across the two cross-sectional surveys. As mentioned, pharmacies with specific Toronto-area postal codes received their vaccines from OGPMSS in both 2015-2016 and 2016-2017. For the purpose of this analysis, all participants with these Toronto-area postal codes were removed from the 2016-2017 dataset (n = 9, Toronto participants removed from 2016 to 2017), because their distributor model had not changed from the year prior, but remained in the 2015-2016 dataset, because they had a shared experience with all other pharmacies in that season.
Quantitative data were analyzed using SAS Release 3.8 (SAS Institute Inc., Cary, NC, USA). Almost all quantitative data were collected as categorical counts; the exception to this was a continuous numerical response to the question, "On average, how long did it take between when an order was placed and when it was available for pick-up or delivered?", where data were binned as part of cleaning. Therefore, chi-squared tests of significance were used for all conclusions provided in this paper.
Qualitative data from open-ended questions were relatively concise and limited to <50 words per question. A summary of participants' remarks was prepared; the interpretation of these remarks was verified by a second reader and the two readers reached an agreement on any discrepancies in the summary of remarks. Sentiments that were shared among several participants and that provide context for quantitative data or further understanding of pharmacists' experiences are presented and discussed.

Participating Pharmacies and Vaccination Workflow
Characteristics of the participating pharmacies are shown in Table 1 Table 2 presents key indicators of participating pharmacies' experience with ordering and fulfilment. There was a statistically significant difference in ordering method between years (p < 0.01); ordering by fax dominated (91.0%) in 2015-2016, and online orders grew from 9.0% to 91.4% year-over-year. There was a statistically significant difference (p < 0.01) in the reported average time to order fulfilment between years, with a higher proportion of respondents in 2016-2017 stating that their orders were ready for pick-up or were delivered within 1-3 days compared to 2015-2016. However, in both seasons, >30% of respondents indicated that order fulfilment was slower earlier in the season compared to the remainder of the season. In the 2016-2017 season, a higher percentage of respondents reported that the distributor limited their order quantities both in early and late season compared to 2015-2016.  Table 3 presents key indicators about the impact of vaccine ordering and fulfilment on patient care. Respondents in the 2016-2017 season reported significantly more instances where they were unable to provide patients with an influenza vaccination due to the vaccine being unavailable (p < 0.05). These 2016-2017 respondents also reported significantly more days with zero influenza vaccine inventory (p < 0.01). In both seasons, approximately 40% of participating pharmacies indicated that they ran out of the specific vaccine formats requested by patients at one point. Table 3. Impact of vaccine ordering and fulfilment on patient access to vaccines.

2016-2017 n (%)
Were there instances where you could not give influenza vaccine to patients who wanted it due to vaccine not being available?

Qualitative Results
The results of key qualitative responses are presented in Table 4. Identifiable themes within each year are noted, and a year-over-year comparison is provided. The desire for more frequent order fulfillment, faster turnaround times upon ordering, and delivery appear to be resolved in the 2016-2017 season, without a single comment related to these themes.

•
Requests for more specific dates of availability, and the option to order in quantities that were not multiples of pack sizes or to return un-used vaccine doses from an opened pack were new themes in 2016-2017.

Discussion
Several significant improvements were seen following the introduction of pharmacy distributor-managed ordering and delivery of the publicly funded influenza vaccine; however, there are remaining challenges with important implications for when, and if, patients are vaccinated. For example, despite improvements in order fulfillment time with distributor-based vaccine management, limits on order quantities were more pronounced, and there were more pharmacies that reported instances of having no vaccine on-hand when patients requested it than under the public health distribution model. With the aim of vaccinating as many people as possible under a universal immunization program, efforts should be made to facilitate consistent product availability and stable pharmacy inventories, particularly at the start of the influenza season when public demand is highest, and pharmacies are running influenza clinics. To our knowledge, this is the first paper to examine healthcare providers' experiences across two models of vaccine distribution-one being publicly-led and one being privately-managed.
To ensure representativeness, a novel sampling approach was utilized to identify the number of pharmacies from each PHU to be sampled in proportion to that region's population and number of pharmacies. In both seasons, the majority of participating pharmacies were chains, banners, or franchises (62.9% and 59.3% in 2015-2016 and 2016-2017, respectively), which aligns with earlier surveys of Ontario community pharmacists on topics related to vaccines and immunization [17,18]. Differences in business models are not expected to have had a significant impact on pharmacies' experiences with vaccine distribution from PHUs; however, in 2016-2017, there is the potential that corporate (chain, banner, franchise) relationships with private distributors could have influenced the distribution experience. Geographic differences were not analyzed, although they may also affect distribution experience.
In the 2015-2016 public-health managed distribution system, pharmacies felt they were spending a disproportionate amount of time on influenza vaccine inventory management compared to their other pharmaceutical inventory; this system relied heavily on faxed orders, that-in most cases-were fulfilled once per week, and required pharmacies to coordinate pick-up of vaccines from depots. In 2015-2016, pharmacies felt that more frequent order fulfillment, faster turnaround times on orders placed, and vaccine delivery were opportunities for improvement. The introduction of pharmacy distributor-managed ordering and delivery of the influenza vaccine appears to have brought considerable efficiencies from pharmacies' perspective.
The statistically significant shift in ordering method between seasons-where online ordering effectively replaced fax ordering as the primary method-can be attributed to pharmacy distributors already having online ordering infrastructure that, in many cases, was integrated with pharmacy management software(s). This paperless ordering also appears likely to have contributed to pharmacies' perception of increased system "efficiency", as it aligned with how pharmacies were placing orders for all other products. It is not surprising that pharmacy distributors could offer more responsive order processing and fulfillment for pharmacies-seen with the increase in pharmacies reporting 1 to 3-day lead times-as delivering to pharmacies was previously not a core task of PHUs.
Despite improvement in overall order processing and fulfillment, respondents in both seasons indicated that orders placed early in a season were typically slower to arrive than those placed later, and that quantities on these early orders were often limited by the distributor, whether that be PHUs or pharmacy distributors. Pharmacies believed that quantity limits were based on prior year vaccine ordering patterns and usage, as well as preferential allocation of vaccines to physician offices. We were unable to identify evidence that these strategies are mandated at the provincial level; however, we recognize these may be part of internal communications between the Ministry of Health and PHUs/distributors. The past and current vaccine order forms used by Ontario PHUs require a count of on-hand vaccine inventory, ostensibly to limit stockpiling; this may offer an alternative explanation for ordering limits experienced beyond the initial order [19]. Similar barriers related to orders being limited or late were identified by physicians in a 2005-2006 survey in the United States, although other studies of immunizers' experience with the vaccine supply chain remain very limited [12].
Importantly, pharmacies in both seasons indicated that these bottlenecks had a negative impact on pharmacy workflow, but perhaps more importantly, had both direct and indirect implications for patients' access to vaccination. For example, in both seasons, pharmacies discussed that significant time was spent resolving order issues with the distributor and explaining shortages to patients-both of which presented challenges to the pharmacy workflow. As mentioned previously, more pharmacies reported stock-outs when receiving vaccines from pharmacy distributors, and this likely contributed to more reports of patients being turned away. Some respondents indicated that they coordinated patient wait lists for those who wanted to receive the vaccine when the pharmacy was re-stocked, but this was another time-intensive activity for the pharmacy. The management of wait lists was also complicated by pharmacies not having estimated fulfillment dates for their orders. These challenges are not new nor unique to the pharmacy profession, as parallel frustrations have been expressed among physicians more than a decade ago, including the need to spend more time and effort obtaining vaccines, and a lack of communication from distributors surrounding the status of the supply chain [12].

Addressing Remaining Challenges
While resolving early-season supply issues is complicated, communication between distributors and pharmacies can be improved by informing pharmacies of their initial allotment of vaccine, as opposed to asking pharmacies to submit order quantities which are then restricted. Anticipated delivery dates should be provided to assist pharmacies with managing wait lists and patient queries. Anticipated delivery dates may also allow pharmacies to avoid stock-outs while minimizing on-hand inventory, as it clearly indicates when just-in-time ordering (i.e., with next day delivery) is not possible. Allowing for a greater supply of vaccine doses early in the season or allowing the distribution of partial packages may also help meet patient demand while reducing wastage. It is notable that pharmacies explained that they were forced to over-order late in the influenza season as demand for the vaccine waned, because distributors only sold vaccines in multiples of the pack size (e.g., 1 pack = 8 vaccine doses). It is unknown how many unused doses were ultimately returned at the end of the influenza season, that could have been offered to individuals impacted by shortages elsewhere. Finally, a more drastic redesign of the current system could involve a shift towards vendor-managed inventory (VMI)-where distributors receive on-hand inventory data from healthcare facilities and determine replenishing shipment quantities based on this data instead of from purchase orders. VMI systems have proven beneficial in some countries that also outsourced logistics of publicly-funded vaccine distribution activities [20].

Conclusions
Moving forward, vaccine supply chain redesigns are opportunities for research at the intersections of economics, epidemiology, and social sciences. There remains a lack of real-world accounts surrounding these redesigns, and even fewer that consider the impact on immunizers as end consumers in these systems. With pharmacists' growing role as immunizers in Canada and globally, the processes and frameworks for designing these types of outsourcing arrangements and best practices for implementing them appear to be underdeveloped and may lack considerations for pharmacies as sites of immunization. Finally, these findings and recommendations are particularly relevant in an age where pandemic responsiveness within vaccine supply chains and immunization capacity are at the fore. Existing pharmaceutical distribution infrastructure will need to be leveraged in creative ways to achieve rapid and widespread rollout of any pandemic-relevant vaccine or medication and appreciating the challenges that pharmacies face in acquiring and managing their inventories, depending on the distributor and systems in place, can inform how we design pandemic response strategies.