How Do Users Respond to Mass Vaccination Centers? A Cross-Sectional Study Using Natural Language Processing on Online Reviews to Explore User Experience and Satisfaction with COVID-19 Vaccination Centers

To reach large groups of vaccine recipients, several high-income countries introduced mass vaccination centers for COVID-19. Understanding user experiences of these novel structures can help optimize their design and increase patient satisfaction and vaccine uptake. This study drew on user online reviews of vaccination centers to assess user experience and identify its key determinants over time, by sentiment, and by interaction. Machine learning methods were used to analyze Google reviews of six COVID-19 mass vaccination centers in Berlin from December 2020 to December 2021. 3647 user online reviews were included in the analysis. Of these, 89% (3261/3647) were positive according to user rating (four to five of five stars). A total of 85% (2740/3647) of all reviews contained text. Topic modeling of the reviews containing text identified five optimally latent topics, and keyword extraction identified 47 salient keywords. The most important themes were organization, friendliness/responsiveness, and patient flow/wait time. Key interactions for users of vaccination centers included waiting, scheduling, transit, and the vaccination itself. Keywords connected to scheduling and efficiency, such as “appointment” and “wait”, were most prominent in negative reviews. Over time, the average rating score decreased from 4.7 to 4.1, and waiting and duration became more salient keywords. Overall, mass vaccination centers appear to be positively perceived, yet users became more critical over the one-year period of the pandemic vaccination campaign observed. The study shows that online reviews can provide real-time insights into newly set-up infrastructures, and policymakers should consider their use to monitor the population’s response over time.


Introduction
In the wake of large-scale COVID-19 vaccination campaigns starting in December 2020, numerous countries across the globe opted to use mass vaccination centers as a key pillar of their vaccine roll-outs [1]. A vaccination center is "a location, normally used for non-healthcare activities, set up for high-volume and high-speed vaccinations during infectious disease emergencies" [2]. In Germany, as in many other high-income countries, mass vaccination centers were a novel construct, as vaccinations are usually administered in outpatient practices.
Understanding how users experience these novel structures can provide valuable insights. Positive user experience of healthcare is not just an inherent goal by itself [3], well-designed vaccination services can also contribute to higher vaccine uptake [4]: they

Data Collection
The German capital Berlin was chosen as a study site since online reviews were available for all its six vaccination centers. This provided a large sample and allowed the team to study an entire federal state. Furthermore, Berlin hosts a diverse population, promising heterogeneous reviewers. Vaccination centers accounted for approximately 40% of all COVID vaccinations administered in Berlin between December 2020 and December 2021 [31]. The Appendix A ( [32][33][34][35][36][37][38][39][40][41]) provides more details .
Google Maps reviews for all six COVID vaccination centers were retrieved for 12 consecutive months, from 27 December 2020 (start of the vaccination campaign) to 26 December 2021. A dataset containing 3797 online reviews was extracted from publicly available Google Maps reviews sites (Appendix B). Online reviews are provided voluntarily and contain, at a minimum, a star rating ranging from one star to five stars, where one star is the lowest score and five stars is the highest score. In addition, reviews can contain a free text comment. Among other items, the dataset extracted included the following: a review rating ranging from one star to five stars; a timestamp detailing date and time when the review was submitted; a review text. Reviews dating before the opening and after the closing of a site were excluded. For one site (a festival arena) that did not have a dedicated review sub-site, reviews that clearly related to the event location rather than to the vaccination center were removed. Non-English reviews were translated to English using Google Translate [42].

Key Themes
An analysis of key themes and sentiments was conducted in Python 3.9. 9 [43] on all reviews containing text using validated machine learning algorithms: topic modeling with Latent Dirichlet Allocation (LDA) and keyword extraction with Bidirectional Encoder Representations from Transformers (BERT).
LDA is a probabilistic topic modeling algorithm. It extracts word clusters called "topics" that are distributed across texts and connected as latent constructs based on their relative frequency and location [44]. Review texts were pre-processed for LDA using the NLTK Python library [45]. The list of stop words, i.e., words to exclude, was amended to also exclude words such as "COVID" and "vaccine" that did not add meaning ( Figure S1). LDA was then run on the entire text using the Gensim Python library [46]. To refine the model parameters and ensure the topics are interpretable and meaningful, the model was evaluated based on topic coherence (C_v measure) and the intertopic distance map. The C_v coherence score was calculated for 2 to 15 topics to determine the optimal number of topics. The Dirichlet hyperparameters α and η were set to default [46]. For topic labeling, each topic's salient terms were examined based on term frequency (λ = 1.0) and relevance (λ = 0.6) as suggested by Sievert and Shirley [47], and sample reviews were inspected. The weight parameter λ ranged from 0 to 1. A smaller λ indicates rare but exclusive terms for the topic, whereas a larger λ features frequent but not necessarily exclusive words [25].
For validation and a more granular look into the reviews overall and sub-groups, automated keyword extraction was used. Keyword extraction identifies single salient words based on semantics. The recent NLP model keyBERT [48][49][50] was used, which can identify more than one keyword in each text. For example, in "Super organized and very friendly staff.", keyBERT identified "staff", "organized", and "friendly" as keywords. Text pre-processing was not needed, but as with topic modeling, additional stop words were added. Synonyms were then aggregated through lemmatization and manual review (e.g., "employees", "staff", and "helper" were grouped to "staff"; "appointment" and "appointments" to "appointment"; Table S1). To ensure the relevance of the results, only words with high frequency were considered in the analysis. Keywords identified in at least 1% (28/2740) of reviews analyzed were then grouped and sorted to a user experience framework (see Section 2.2.2). Sub-set analyses were conducted by sentiment and time period. All extractions were validated by one human investigator on a random sub-sample of 30 reviews.

Key Interactions
A vaccination center user experience framework was developed to summarize the findings and draw actionable conclusions (Appendix C). User experience was defined as the combination of a vaccine recipient's user journey through the vaccination center and the key determinants of their experience at the vaccination center. User journey phases were defined according to the German Ministry of Health's handbook for vaccination centers [37]. Key determinants of user experience, such as accessibility, hygiene, and duration, were identified through policy documents on vaccination center planning [37] and case studies of vaccination centers [38,39], as well as existing Patient Reported Experience Measures (PREMs) frameworks [40,41]. These determinants were clustered into 3 themes: staff, process/management, and location. Cost was omitted from the framework, as COVID-19 vaccinations were freely available in Germany.

Timeline
To understand user responses during different phases of the vaccination campaign, seven time periods were defined based on population eligibility and the setup of alternative vaccination structures over time ( Figure 1). In the beginning, vaccinations were solely offered in vaccination centers based on appointments and priority groups. As the campaign progressed, eligible groups were gradually expanded, and other structures (e.g., outpatient practices) could offer vaccines. From June 2021 (period 4) onwards, the prioritization of certain population groups was abandoned, and vaccinations were offered to anyone above the age of 12 in vaccination centers. From July 2021 (period 5) onwards, walk-ins without appointment were introduced [36], and the first vaccination centers closed. September 2021 (period 6) marked the start of the booster campaign for elderly, and November 2021 (period 7) marked the start of the booster campaign for all. campaign progressed, eligible groups were gradually expanded, and other structures (e.g., outpatient practices) could offer vaccines. From June 2021 (period 4) onwards, the prioritization of certain population groups was abandoned, and vaccinations were offered to anyone above the age of 12 in vaccination centers. From July 2021 (period 5) onwards, walk-ins without appointment were introduced [36], and the first vaccination centers closed. September 2021 (period 6) marked the start of the booster campaign for elderly, and November 2021 (period 7) marked the start of the booster campaign for all.

Satisfaction
To understand which themes elicit satisfaction or dissatisfaction, keyword analysis was conducted separately on positive and negative reviews. Users directly expressed their sentiment through review ratings ranging from one to five stars. To ensure that the review rating corresponded to the review content, a Pearson correlation coefficient between the sentiment of the review text and the star rating was calculated using the NLTK and SciPy Python libraries [45,51]. Since the Pearson coefficient showed a statistically significant correlation, the rating score was used for further analysis. A review was deemed positive if it received four or five stars, neutral if it received three stars, and negative if it received one or two stars. Users leaving a positive review were assumed to be satisfied, and those leaving a negative review were considered dissatisfied [14].

Results
After exclusion, the final dataset contained 3647 reviews. Of these, 907 (25%) solely contained a rating, and 2740 (75%) contained rating and text ( Figure 2A). The average length of a review text was 33 words in English, with a minimum of 1 and a maximum of 544 words.

Satisfaction
To understand which themes elicit satisfaction or dissatisfaction, keyword analysis was conducted separately on positive and negative reviews. Users directly expressed their sentiment through review ratings ranging from one to five stars. To ensure that the review rating corresponded to the review content, a Pearson correlation coefficient between the sentiment of the review text and the star rating was calculated using the NLTK and SciPy Python libraries [45,51]. Since the Pearson coefficient showed a statistically significant correlation, the rating score was used for further analysis. A review was deemed positive if it received four or five stars, neutral if it received three stars, and negative if it received one or two stars. Users leaving a positive review were assumed to be satisfied, and those leaving a negative review were considered dissatisfied [14].

Results
After exclusion, the final dataset contained 3647 reviews. Of these, 907 (25%) solely contained a rating, and 2740 (75%) contained rating and text ( Figure 2A). The average length of a review text was 33 words in English, with a minimum of 1 and a maximum of 544 words.

Key Themes
Topic modeling with LDA identified 5 key topics. Keyword extraction identified 47 keywords.
For topic modeling, five topics received the highest coherence score (0.486). The five topics ( Figure 3, Table S2) covered the vaccination process, location, and staff. Topics 1 and 4 contained comparatively narrow terms. Topics 2, 3, and 5 were more convoluted. Topic 1 was related to scheduling and wait time, and Topic 4 was related to staff friendliness and overall organization. Topic 2 contained terms related to efficiency and duration ("quick", "start", and "finish"), vaccination effects, and personal protective equipment ("mask" and "ffp"). An inspection showed that some users reported the effects they experienced shortly after their vaccination, e.g., "[...] I tolerated the vaccine without side effects, only the shoulder hurt a little, like sore muscles. [...]". Reviews mentioning masks discussed subjects ranging from complaints about security staff not wearing masks to the provision of masks at the centers. Topic 3 broadly dealt with the vaccination center location, specifically its accessibility ("parking", "welcome", and "entrance"), uniqueness ("effort", "event", and "concert"), and the intuitiveness of the process exemplified through the term "uncomplicated". Several users commented on the vaccination being a special event, but also on the fact that a location housed events before becoming a vaccination center. Topic 5 again encompassed site access ("bus", "shuttle", and "entrance"), but also registration and documentation ("certificate", "code", and "digital"). Based on the marginal topic distribution ( Figure 3A), the two most salient topics were 1 and 4. Topic 5, regarding accessibility and documentation, was also quite prominent.

Key Themes
Topic modeling with LDA identified 5 key topics. Keyword extraction identified 47 keywords.
For topic modeling, five topics received the highest coherence score (0.486). The five topics ( Figure 3, Table S2) covered the vaccination process, location, and staff. Topics 1  In the intertropic distance map, each circle represents one topic. The circle size represents the relative number of terms that belong to the topic. The distance between circles represents the relative similarity and connectedness of topics. Topic circles that are closer to each other have more terms in common. For the top 10 terms per topic, the most frequent terms within a topic are shown at λ-value 1, and the top terms combining frequency and relevance are shown at λ-value 0.6. Relevance reflects the level at which a term exclusively belongs to a single topic. The λ-values 1 and 0.6 are suggested by the prior literature to analyze topics (see Section 2.2.1).
Keyword extraction identified 47 keywords overall that featured in at least 1% (28/2740) of reviews containing text. The top 30 keywords by frequency are displayed in Figure 4. Four terms were featured in more than 10% (274/2740) of reviews: "organization" (45%, 1227/2740), "friendliness" (34%, 940/2740), "staff" (34%, 940/2740), and  In the intertropic distance map, each circle represents one topic. The circle size represents the relative number of terms that belong to the topic. The distance between circles represents the relative similarity and connectedness of topics. Topic circles that are closer to each other have more terms in common. For the top 10 terms per topic, the most frequent terms within a topic are shown at λ-value 1, and the top terms combining frequency and relevance are shown at λ-value 0.6. Relevance reflects the level at which a term exclusively belongs to a single topic. The λ-values 1 and 0.6 are suggested by the prior literature to analyze topics (see Section 2.2.1).
Keyword extraction identified 47 keywords overall that featured in at least 1% (28/2740) of reviews containing text. The top 30 keywords by frequency are displayed in Figure 4.

Key Interactions
The 47 keywords were mapped against the user experience framework. Overall, more keywords were related to key determinants of the experience rather than moments in the vaccination center user journey ( Figure 5). All three key determinant themes (staff, process/management, and location) were mentioned in reviews. Terms related to overall organization (52%, 1426/2740) and friendliness/responsiveness (49%, 1331/2740) were more frequent than those related to patient flow/wait time (25%, 688/2740), duration/efficiency (18%, 497/2740), and accessibility (12%, 321/2740). Intuitiveness, information/education, and hygiene/infection prevention and control were seldom or not featured.

Key Interactions
The 47 keywords were mapped against the user experience framework. Overall, more keywords were related to key determinants of the experience rather than moments in the vaccination center user journey ( Figure 5). All three key determinant themes (staff, process/management, and location) were mentioned in reviews. Terms related to overall organization (52%, 1426/2740) and friendliness/responsiveness (49%, 1331/2740) were more frequent than those related to patient flow/wait time (25%, 688/2740), duration/efficiency (18%, 497/2740), and accessibility (12%, 321/2740). Intuitiveness, information/education, and hygiene/infection prevention and control were seldom or not featured.
There were five key moments in the user journey that were frequently mentioned: scheduling, transit, arrival, waiting, and the vaccination itself. Most keywords were related to the visit itself, while only a few were concerning experiences outside of the vaccination center pre-or post-visit. Aside from scheduling, other administrative processes like registration and documentation were not salient. Neither were adverse events or continuous monitoring of vaccination effects.

Timeline
The number of reviews varied over time (Figure 2A). In period 2 (when more-also younger-individuals gained access to vaccinations), the number of reviews steeply increased. There was a noticeable drop in reviews after period 5. The top keywords by time period largely overlapped with overall keywords ( Figure 2C). Organization and staff were salient in all periods. The three terms "organization", "staff", and "friendliness" consistently dominated the top three keywords across all periods until period 7. Other words related to friendliness, e.g., "helpful", "nice", and "thanks", were salient in all periods but became less prevalent over time. Terms related to waiting and duration, such as "wait", "quick", "queue", and particularly the term "appointment", became gradually more salient from period 2 onwards. "Taxi", "accompany", and "support" are only featured among the top 10 keywords in period 1.
Vaccines 2023, 11, x FOR PEER REVIEW 9 of 19 Figure 5. Keywords featured in at least 1% of reviews sorted to the user experience framework. Keywords identified through the keyword extraction were grouped and then mapped against the framework to highlight key interactions and identify recurring themes across reviews. A single review could contain keywords related to several dimensions or phases, e.g., both "organization" and "staff". A single keyword may also simultaneously be related to an enabler and the journey, e.g., "wait", which is related to "waiting" in the journey and to "patient flow/wait time" under enabler dimensions. The frequency count of the enabler dimensions or journey phases hence do not add up to the total number of reviews.
There were five key moments in the user journey that were frequently mentioned: scheduling, transit, arrival, waiting, and the vaccination itself. Most keywords were related to the visit itself, while only a few were concerning experiences outside of the vaccination center pre-or post-visit. Aside from scheduling, other administrative processes like registration and documentation were not salient. Neither were adverse events or continuous monitoring of vaccination effects.

Timeline
The number of reviews varied over time (Figure 2A). In period 2 (when more-also younger-individuals gained access to vaccinations), the number of reviews steeply increased. There was a noticeable drop in reviews after period 5. The top keywords by time period largely overlapped with overall keywords ( Figure 2C). Organization and staff were salient in all periods. The three terms "organization", "staff", and "friendliness" consistently dominated the top three keywords across all periods until period 7. Other words related to friendliness, e.g., "helpful", "nice", and "thanks", were salient in all periods but became less prevalent over time. Terms related to waiting and duration, such as "wait", "quick", "queue", (1) only featuring keywords contained in at least 1% of reviews; (2) Reviews with keywords related to "transit" e.g., "bus", "taxi", "shuttle" are counted twice: under pre-visit transit to vaccination center and under post-visit transit from vaccination center.

Figure 5.
Keywords featured in at least 1% of reviews sorted to the user experience framework. Keywords identified through the keyword extraction were grouped and then mapped against the framework to highlight key interactions and identify recurring themes across reviews. A single review could contain keywords related to several dimensions or phases, e.g., both "organization" and "staff". A single keyword may also simultaneously be related to an enabler and the journey, e.g., "wait", which is related to "waiting" in the journey and to "patient flow/wait time" under enabler dimensions. The frequency count of the enabler dimensions or journey phases hence do not add up to the total number of reviews.
Over time, ratings stayed relatively constant with little variance until period 3. In period 4, average ratings started to decrease. Similar to the overall keywords, the keywords identified in positive (N = 2426) and negative (N = 223) reviews containing text were mostly linked to organization, staff, and appointments ( Figure 4B).

Principal Results
This study had three principal findings. First, the overall reception of vaccination centers was strongly positive. Second, the most important themes identified in the online reviews were wait time, overall organization, and friendliness, while the most important moments in the user journey were scheduling, transit, arrival, waiting, and vaccination. Third, efficiency/duration and wait time/patient flow were leading drivers for dissatisfaction, and their prevalence increased over the vaccination campaign as satisfaction overall decreased.
Many online review-based studies in healthcare [11,24,28,52] and consumer research [53-55] observed a positive rating tendency. The vaccination center online reviews were extremely positively skewed compared to those of studies in other settings however [11]. Two survey-based studies of COVID-19 vaccination experience in Saudi Arabia and Saxony (Germany) [56,57] found similarly high overall satisfaction scores of above 90% for vaccination centers. Aside from actual experiences or self-selection, this positive skew could be linked to the gratitude and hope associated with the COVID vaccination [58], specifically in the beginning of the vaccination campaign. These feelings could also explain the satisfaction decrease over time; at the start, COVID vaccines were rare, and users were likely more appreciative of the opportunity to get vaccinated, especially early target populations that were more exposed and vulnerable to the virus. As the campaign lasted, the vaccine became more of a commodity. User expectations with regards to the vaccination process probably increased, while recipients became less eager and dependent on vaccines. This seems to fit with the timeline analysis: once the prioritization was abandoned in period 4 and vaccines were offered to anyone aged 12 and above, the rating scores dropped more noticeably ( Figure 2B). Additionally, after period 5, only two centers stayed open, and, in period 7, the time between initial and booster vaccination was suddenly decreased, leading to "chaos" [59] around Christmas 2021. Both put pressure on scheduling and impacted accessibility.
Two studies previously evaluated English language tweets on COVID-19 vaccination after the vaccine rollout, irrespective of geographical location, and also found that scheduling and appointments were prevalent themes [25,28]. A systematic review found that important themes in patient online reviews generally included physicians' demeanor, staff friendliness, time spent with patients, ease of scheduling, wait time, and cost [11]. While staff interaction and appointment management were reflected in the vaccination center reviews, costs were irrelevant in the context of our study. With regards to time spent with patients, vaccination center users seemed more focused on efficiency (i.e., little time spent at the vaccination site) and the competence and friendliness of staff rather than ample exposure time to doctors. A survey-based study comparing vaccination centers and GP offices in Saxony, a more rural German region, found that wait times were lower at mass vaccination centers than at GP offices [56].
Topic modeling provided five topics with broader terms that could not always be distinctly sorted to a single determinant, whereas keyword extraction provided keywords more focused on distinct determinants. Both methods identified similar principal themes overall, which increases confidence in the results. Furthermore, the spheres indicating the topics in the intertopic distance map for LDA topic modeling (Figure 3) do not overlap, which can be understood as a characteristic of good quality. Yet, it is noteworthy that keyword extraction did not determine terms related to side effects or documentation (e.g., "certificate" and "code") as salient. They were featured, however, in Topics 2 and 5 of topic modeling. "Side effects" were also a prevalent topic identified in analyses of tweets related to COVID-19 vaccine rollout [25,28].
Studies cite a range of critical themes in negative online reviews: discordant expectations (education, support, and promises) and sub-optimal communication and quality of care (management, organization, staff, and equipment) [12,60,61]. These were only partly reflected in our findings. As vaccination centers are single-purpose facilities under special circumstances, user priorities appear different from those of traditional healthcare delivery. Huangfu and colleagues [28] also found that appointments played a major role in negative tweets about COVID-19 vaccine rollout. In our study, scheduling, wait time, patient flow, and duration became more salient as the vaccination campaign progressed. In 2020, Volpp and colleagues highlighted the need to reduce these "hassle factors" as crucial for driving COVID vaccine uptake [10]. Dysfunctional scheduling in particular has been a common criticism from the beginning of the German vaccine roll-out [62]. Scheduling, which influences wait time, and choice of location, which influences accessibility, lie outside of the vaccination site operators' control, however. Both are managed by the public administration. To address these critical user experience themes, cross-stakeholder collaboration is necessary.

Limitations
Our study has three key limitations. First, it is important to note that online reviews are voluntary and predominantly anonymous. Online reviews hence suffer from selfselection bias, rely on user sincerity and certain groups appear more prone to providing them, i.e., this limits representativeness. A survey of patients in Germany showed that younger, female, more educated, and chronically ill people were more likely to use patientreview websites [63]. The reviews used in this study also lacked information on reviewer characteristics and motives. Furthermore, some reviews may not be left by actual service users [64]. A manual examination showed that some reviews were written by companions, e.g., "I went to the vaccination with my 90-year-old mother today. [ . . . ]". As the objective was to understand general population response, this does not tarnish the results.
Second, although this study used a large sample covering an entire federal state in Germany over different phases of the COVID-19 vaccination campaign, it studied only one geographical area. Berlin may be particular as vaccination centers were central to the state's vaccine roll-out and some reviews mirror local specificities, e.g., the existence of language barriers. Furthermore, experiences from a well-communicated city-state with a balanced distribution of centers may not translate to rural areas. A study showed that the share of people able to use public transport was higher and the median travel time to vaccination centers in Berlin was lower than they were in other German federal states [65]. It would hence be interesting to see future studies apply the methodology to other geographical areas.
Third, the text quality, specifically related to emojis and translations, was challenging. A total of 258 reviews used emojis. An inclusion was attempted using demoji [66] but produced misleading results in thematic analysis (e.g., light, face, skin, and syringe). Colloquial language, spelling mistakes, and wordplay were not always perfectly translated to English. Some, but not all, translation errors, e.g., "snake" for the German word "Schlange", which means both queue and snake, could be corrected at the stage of keyword merging. Despite machine learning techniques having made tremendous progress, this highlights some of the currently persisting shortcomings of computer-aided analysis. Improved algorithms for the translation and the analysis of non-English texts and emojis are needed.
To our knowledge, this is the first study to systematically assess vaccination centers from a user's perspective in Germany using online reviews and the first to assess pandemic response structures using NLP. It lays the foundation for future research in this area and contributes to pandemic response planning. Aside from addressing the limitations above, future research should include a comparison to other vaccination structures, e.g., physicians' offices, and other single-purpose healthcare structures, e.g., test centers, and vaccination experiences outside of a global pandemic. It should also contrast online reviews with traditional survey results to further gauge their potential.

Conclusions
Positive user experience seems a worthwhile investment for decision-makers and operators during a pandemic, where adherence to services and societal cohesion are essential. Overall, the results of the study with regards to vaccination centers as a pandemic mass vaccination structure are encouraging: User receptions of vaccination centers were very positive, and smooth processes and friendly staff were highly valued. As the mass vaccination campaign lasted, however, efficiency and time management needed to be monitored to ensure long-term satisfaction. Online reviews provided useful, free, and real-time feedback. This makes them an attractive, early recognition "armchair epidemiology" tool [67] for evaluating novel structures during crises, where circumstances continuously change. Their use in pandemic response should therefore be strengthened-encouraging user reviews, establishing units in public administration capable of their analysis, and promoting an iterative design mindset.

Supplementary Materials:
The following supporting information can be downloaded at https: //www.mdpi.com/article/10.3390/vaccines11010144/s1, Figure S1: Data pre-processing procedure for LDA topic modeling. Figure S2: Keywords contained in at least 1% of reviews sorted by key determinant. Table S1: Word grouping for keyword analysis.

Acknowledgments:
We acknowledge the financial support from the Open Access Publication Fund of Charité-Universitätsmedizin Berlin and the German Research Foundation (DFG) to support the publication of our research paper.

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix A
A total of six mass vaccination centers were set up in Berlin and opened between 27 December 2020, and 8 March 2021. The centers were set up by or on behalf of the federal states and financed through public funds [32]. In Berlin, the centers were operated by several Berlin-based humanitarian aid organizations coordinated through DRK SWB, a nonprofit company established by the German Red Cross Berlin, the Association of Statutory Health Insurance Physicians Berlin and the Senate Department for Health, Nursing and Equality. The vaccination centers were spread across the city to ensure reachability from all districts. The vaccination centers were open daily usually between 9AM and 5PM with a few exceptional closures (e.g., two sites closed temporarily due to a suspension of AstraZeneca while side effects were being reviewed [1,2]). A single vaccination center could perform up to 4.000 vaccinations per day [35]. The effective daily volume depended on the number of vaccines available, and the number of appointments booked. Three different vaccines were available at the vaccination centers. In the beginning vaccinations were offered based on appointments and priority group, from July 2021 walk-ins were introduced [36]. As the vaccination campaign progressed and other structures (e.g., outpatient practices and company physicians) got involved, vaccination centers were gradually closed. At the time the study was conducted two vaccination centers were still running.
Vaccines 2023, 11, x FOR PEER REVIEW 13 of Figure A1. Overview of COVID-19 vaccination centers in Berlin.
A total of six mass vaccination centers were set up in Berlin and opened betwe December 27, 2020, and March 8, 2021. The centers were set up by or on behalf of federal states and financed through public funds [32]. In Berlin, the centers were opera by several Berlin-based humanitarian aid organizations coordinated through DRK SW a non-profit company established by the German Red Cross Berlin, the Association Statutory Health Insurance Physicians Berlin and the Senate Department for Hea Nursing and Equality. The vaccination centers were spread across the city to ensure rea ability from all districts. The vaccination centers were open daily usually between 9A and 5PM with a few exceptional closures (e.g., two sites closed temporarily due to a s pension of AstraZeneca while side effects were being reviewed [33,34]). A single vacci tion center could perform up to 4.000 vaccinations per day [35]. The effective daily volu depended on the number of vaccines available, and the number of appointments book Three different vaccines were available at the vaccination centers. In the beginning v cinations were offered based on appointments and priority group, from July 2021 wa ins were introduced [36]. As the vaccination campaign progressed and other structu (e.g., outpatient practices and company physicians) got involved, vaccination cent were gradually closed. At the time the study was conducted two vaccination centers w still running.

Appendix B
The raw data was obtained on 27 dec 2021 from Google Maps Reviews public revi websites (Table A1) using Outscraper (https://outscraper.com). Online reviews for all vaccination centers in Berlin were obtained for the period 27 December 2020 00:00:00 27 December 2021 00:00:00. The dataset used for the analysis can be found on github https://github.com/stellaroxanne/User-Reviews-Vaccination-Centers-Berlin (last access 27 December 2022).

Appendix B
The raw data was obtained on 27 dec 2021 from Google Maps Reviews public review websites (Table A1) using Outscraper (https://outscraper.com). Online reviews for all six vaccination centers in Berlin were obtained for the period 27 December 2020 00:00:00 to 27 December 2021 00:00:00. The dataset used for the analysis can be found on github at: https://github.com/stellaroxanne/User-Reviews-Vaccination-Centers-Berlin (last accessed 27 December 2022).

Appendix C
The figure below ( Figure A2) shows the final framework for vaccination center user experience composed of user journey and key determinants.
The user journey follows the path of a vaccination center user from their invitation to get vaccinated up to their follow-up appointment. It is composed of three phases: pre-visit, visit and post-visit. The user journey can also be applied to vaccinations in other settings as it follows a similar overarching path.
The table below (Table A2) shows the different publications on vaccination centers and patient reported outcome and experience measures that were referenced to determine the relevant key determinants for a positive vaccination experience. The key determinants that form part of the final framework can be clustered into three groups: staff, process/management and location.  Figure A2. Framework for vaccination center user experience combining key determinants of user experience and the vaccination center user journey. Figure A2. Framework for vaccination center user experience combining key determinants of user experience and the vaccination center user journey.