# Population-Level Effectiveness of COVID-19 Vaccination Program in the United States: Causal Analysis Based on Structural Nested Mean Model

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Data

#### 2.2. Treatment and Outcome Variable

#### 2.3. Causal Parameters of Interest

#### 2.4. Confounders

#### 2.5. Data Analysis

#### 2.5.1. Structural Nested Mean Model

#### 2.5.2. Generalized Estimating Equation Models and Fixed Effects Models

#### 2.5.3. Scenarios Analysis

- No-vaccination scenario: No people are vaccinated since week 1, which means ${\overline{a}}_{i,13}^{*}={\overline{0}}_{13}$.
- Twice speed scenario: The number of people vaccinated for the first time each week is twice the actual number in each state, which means ${\overline{a}}_{i,13}^{*}=\left(2{a}_{i,1}^{\u271d},2{a}_{i,2}^{\u271d},\dots ,2{a}_{i,13}^{\u271d}\right)$.
- Half speed scenario: The number of people vaccinated for the first time each week is half of the actual number in each state, which means ${\overline{a}}_{i,13}^{*}=\left(\frac{{a}_{i,1}^{\u271d}}{2},\frac{{a}_{i,2}^{\u271d}}{2},\dots ,\frac{{a}_{i,13}^{\u271d}}{2}\right)$.
- 1% constant speed scenario: 1% of the population receive their first dose in each week in each state, which means ${\overline{a}}_{i,13}^{*}=\left(100,100,\dots ,100\right)$.
- 4% constant speed scenario: 4% of the population receive their first dose each week in each state, which means ${\overline{a}}_{i,13}^{*}=\left(400,400,\dots ,400\right)$.
- Speed up scenario: For the first six weeks, 1% of the population receive their first dose in each week in each state, while for the remaining seven weeks, 4% of the population receive their first dose in each week in each state, which means ${\overline{a}}_{i,13}^{*}=\left(100,100,\dots ,100,400,\dots ,400\right)$.
- Speed down scenario: For the first seven weeks, 4% of the population receive their first dose in each week in each state, while for the remaining six weeks, 1% of the population receive their first dose in each week in each state, which means ${\overline{a}}_{i,13}^{*}=\left(400,400,\dots ,400,100,\dots ,100\right)$.

#### 2.5.4. Additional Analysis and Extension

## 3. Results

#### 3.1. Baseline Characteristics

#### 3.2. Impact of COVD-I9 Vaccine Program on Weekly Growth Rate of COVID-19 New Cases

#### 3.3. Population-Level Effectiveness of COVID-19 Vaccination and Averted Disease Burden

#### 3.4. Scenarios Analysis

## 4. Discussion

## 5. Conclusions

## Supplementary Materials

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Appendix A

## Appendix B. Details about the Estimation of SNMM

## Appendix C. Specification of GEE Models and Fixed Effects Models

## References

- Dong, E.; Du, H.; Gardner, L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect. Dis.
**2020**, 20, 533–534. [Google Scholar] [CrossRef] - Mathieu, E.; Ritchie, H.; Ortiz-Ospina, E.; Roser, M.; Hasell, J.; Appel, C.; Giattino, C.; Rodés-Guirao, L. A global database of COVID-19 vaccinations. Nat. Hum. Behav.
**2021**, 5, 956–959. [Google Scholar] [CrossRef] [PubMed] - COVID-19 Vaccinations in the United States, Jurisdiction. Available online: https://data.cdc.gov/Vaccinations/COVID-19-Vaccinations-in-the-United-States-Jurisdi/unsk-b7fc (accessed on 9 December 2021).
- Polack, F.P.; Thomas, S.J.; Kitchin, N.; Absalon, J.; Gurtman, A.; Lockhart, S.; Perez, J.L.; Marc, G.P.; Moreira, E.D.; Zerbini, C.; et al. Safety and efficacy of the BNT162b2 mRNA COVID-19 vaccine. N. Engl. J. Med.
**2020**, 383, 2603–2615. [Google Scholar] [CrossRef] [PubMed] - Baden, L.R.; El Sahly, H.M.; Essink, B.; Kotloff, K.; Frey, S.; Novak, R.; Diemert, D.; Spector, S.A.; Rouphael, N.; Creech, C.B.; et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N. Engl. J. Med.
**2021**, 384, 403–416. [Google Scholar] [CrossRef] - Sadoff, J.; Gray, G.; Vandebosch, A.; Cárdenas, V.; Shukarev, G.; Grinsztejn, B.; Goepfert, P.A.; Truyers, C.; Fennema, H.; Spiessens, B.; et al. Safety and efficacy of single-dose Ad26. COV2. S vaccine against COVID-19. N. Engl. J. Med.
**2021**, 384, 2187–2201. [Google Scholar] [CrossRef] - Bjork, J.; Inghammar, M.; Moghaddassi, M.; Rasmussen, M.; Malmqvist, U.; Kahn, F. Effectiveness of the BNT162b2 vaccine in preventing COVID-19 in the working age population-first results from a cohort study in Southern Sweden. medRxiv
**2021**. [Google Scholar] [CrossRef] - Thompson, M.G.; Burgess, J.L.; Naleway, A.L.; Tyner, H.L.; Yoon, S.K.; Meece, J.; Olsho, L.E.W.; Caban-Martinez, A.J.; Fowlkes, A.; Lutrick, K.; et al. Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers—Eight US locations, December 2020–March 2021. Morb. Mortal. Wkly. Rep.
**2021**, 70, 495–500. [Google Scholar] [CrossRef] - Vasileiou, E.; Simpson, C.R.; Shi, T.; Kerr, S.; Agrawal, U.; Akbari, A.; Bedston, S.; Beggs, J.; Bradley, D.; Chuter, A.; et al. Interim findings from first-dose mass COVID-19 vaccination roll-out and COVID-19 hospital admissions in Scotland: A national prospective cohort study. Lancet
**2021**, 397, 1646–1657. [Google Scholar] [CrossRef] - Pawlowski, C.; Lenehan, P.; Puranik, A.; Agarwal, V.; Venkatakrishnan, A.J.; Niesen, M.J.M.; O’Horo, J.C.; Virk, A.; Swift, M.D.; Badley, A.D.; et al. FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system. Med
**2021**, 2, 979–992. [Google Scholar] [CrossRef] - Bernal, J.L.; Andrews, N.; Gower, C.; Robertson, C.; Stowe, J.; Tessier, E.; Simmons, R.; Cottrell, S.; Roberts, R.; O’Doherty, M.; et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on COVID-19 related symptoms, hospital admissions, and mortality in older adults in England: Test negative case-control study. BMJ
**2021**, 373. [Google Scholar] [CrossRef] - Kim, J.H.; Marks, F.; Clemens, J.D. Looking beyond COVID-19 vaccine phase 3 trials. Nat. Med.
**2021**, 27, 205–211. [Google Scholar] [CrossRef] [PubMed] - Gallagher, M.E.; Sieben, A.J.; Nelson, K.N.; Kraay, A.N.M.; Orenstein, W.A.; Lopman, B.; Handel, A.; Koelle, K. Indirect benefits are a crucial consideration when evaluating SARS-CoV-2 vaccine candidates. Nat. Med.
**2021**, 27, 4–5. [Google Scholar] [CrossRef] [PubMed] - Hanquet, G.; Valenciano, M.; Simondon, F.; Moren, A. Vaccine effects and impact of vaccination programmes in post-licensure studies. Vaccine
**2013**, 31, 5634–5642. [Google Scholar] [CrossRef] [PubMed] - Chernozhukov, V.; Kasahara, H.; Schrimpf, P. Causal impact of masks, policies, behavior on early COVID-19 pandemic in the U.S. J. Econom.
**2021**, 220, 23–62. [Google Scholar] [CrossRef] - Courtemanche, C.; Garuccio, J.; Le, A.; Pinkston, J.; Yelowitz, A. Strong Social Distancing Measures In The United States Reduced The COVID-19 Growth Rate: Study evaluates the impact of social distancing measures on the growth rate of confirmed COVID-19 cases across the United States. Health Aff.
**2020**, 39, 1237–1246. [Google Scholar] [CrossRef] [PubMed] - Hsiang, S.; Allen, D.; Annan-Phan, S.; Bell, K.; Bolliger, I.; Chong, T.; Druckenmiller, H.; Huang, L.Y.; Hultgren, A.; Krasovich, E.; et al. The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature
**2020**, 584, 262–267. [Google Scholar] [CrossRef] - Pozo-Martin, F.; Weishaar, H.; Cristea, F.; Hanefeld, J.; Bahr, T.; Schaade, L.; El Bcheraoui, C. The impact of non-pharmaceutical interventions on COVID-19 epidemic growth in the 37 OECD member states. Eur. J. Epidemiol.
**2021**, 36, 629–640. [Google Scholar] [CrossRef] - Kaufman, B.G.; Whitaker, R.; Mahendraratnam, N.; Hurewitz, S.; Yi, J.; Smith, V.A.; McClellan, M. State variation in effects of state social distancing policies on COVID-19 cases. BMC Public Health
**2021**, 21, 1239. [Google Scholar] [CrossRef] - Moghadas, S.M.; Vilches, T.N.; Zhang, K.; Wells, C.R.; Shoukat, A.; Singer, B.H.; Meyers, L.A.; Neuzil, K.M.; Langley, J.M.; Fitzpatrick, M.C.; et al. The impact of vaccination on coronavirus disease 2019 (COVID-19) outbreaks in the United States. Clin. Infect. Dis.
**2021**, 73, 2257–2264. [Google Scholar] [CrossRef] - Makhoul, M.; Ayoub, H.H.; Chemaitelly, H.; Seedat, S.; Mumtaz, G.R.; Al-Omari, S.; Abu-Raddad, L.J. Epidemiological impact of SARS-CoV-2 vaccination: Mathematical modeling analyses. Vaccines
**2020**, 8, 668. [Google Scholar] [CrossRef] - Makhoul, M.; Chemaitelly, H.; Ayoub, H.H.; Seedat, S.; Abu-Raddad, L.J. Epidemiological differences in the impact of COVID-19 vaccination in the United States and China. Vaccines
**2021**, 9, 223. [Google Scholar] [CrossRef] [PubMed] - Foy, B.H.; Wahl, B.; Mehta, K.; Shet, A.; Menon, G.I.; Britto, C. Comparing COVID-19 vaccine allocation strategies in India: A mathematical modelling study. Int. J. Infect. Dis.
**2021**, 103, 431–438. [Google Scholar] [CrossRef] [PubMed] - Romero-Brufau, S.; Chopra, A.; Ryu, A.J.; Gel, E.; Raskar, R.; Kremers, W.; Anderson, K.S.; Subramanian, J.; Krishnamurthy, B.; Singh, A.; et al. Public health impact of delaying second dose of BNT162b2 or mRNA-1273 COVID-19 vaccine: Simulation agent based modeling study. BMJ
**2021**, 373, n1087. [Google Scholar] [CrossRef] [PubMed] - Wheelock, A.; Thomson, A.; Sevdalis, N. Social and psychological factors underlying adult vaccination behavior: Lessons from seasonal influenza vaccination in the US and the UK. Expert Rev. Vaccines
**2013**, 12, 893–901. [Google Scholar] [CrossRef] [PubMed] - Soares, P.; Rocha, J.V.; Moniz, M.; Gama, A.; Laires, P.A.; Pedro, A.R.; Dias, S.; Leite, A.; Nunes, C. Factors associated with COVID-19 vaccine hesitancy. Vaccines
**2021**, 9, 300. [Google Scholar] [CrossRef] [PubMed] - Bish, A.; Yardley, L.; Nicoll, A.; Michie, S. Factors associated with uptake of vaccination against pandemic influenza: A systematic review. Vaccine
**2011**, 29, 6472–6484. [Google Scholar] [CrossRef] - Kelly, C.; Arnold, R.; Galloway, Y.; O’Hallahan, J. A prospective study of the effectiveness of the New Zealand meningococcal B vaccine. Am. J. Epidemiol.
**2007**, 166, 817–823. [Google Scholar] [CrossRef] - Fu, W.; Ho, P.-C.; Liu, C.-L.; Tzeng, K.-T.; Nayeem, N.; Moore, J.S.; Wang, L.-S.; Chou, S.-Y. Reconcile the debate over protective effects of BCG vaccine against COVID-19. Sci. Rep.
**2021**, 11, 8356. [Google Scholar] [CrossRef] - Imai, K.; Kim, I.S. When should we use unit fixed effects regression models for causal inference with longitudinal data? Am. J. Political Sci.
**2019**, 63, 467–490. [Google Scholar] [CrossRef] - Blackwell, M.; Glynn, A.N. How to make causal inferences with time-series cross-sectional data under selection on observables. Am. Political Sci. Rev.
**2018**, 112, 1067–1082. [Google Scholar] [CrossRef] - Rosenbaum, P.R. The consequences of adjustment for a concomitant variable that has been affected by the treatment. J. R. Stat. Soc. Ser. A
**1984**, 147, 656–666. [Google Scholar] [CrossRef] - Hernán, M.A.; Robins, J.M. Causal Inference: What If; Chapman & Hall/CRC: Boca Raton, FL, USA, 2020. [Google Scholar]
- Bruhn, C.A.W.; Hetterich, S.; Schuck-Paim, C.; Kürüm, E.; Taylor, R.J.; Lustig, R.; Shapiro, E.D.; Warren, J.L.; Simonsen, L.; Weinberger, D.M. Estimating the population-level impact of vaccines using synthetic controls. Proc. Natl. Acad. Sci. USA
**2017**, 114, 1524–1529. [Google Scholar] [CrossRef] - Lau, W.C.Y.; Murray, M.; El-Turki, A.; Saxena, S.; Ladhani, S.; Long, P.; Sharland, M.; Wong, I.C.K.; Hsia, Y. Impact of pneumococcal conjugate vaccines on childhood otitis media in the United Kingdom. Vaccine
**2015**, 33, 5072–5079. [Google Scholar] [CrossRef] [PubMed] - Robins, J.M. Causal inference from complex longitudinal data. In Latent Variable Modeling and Applications to Causality; Springer: New York, NY, USA, 1997; pp. 69–117. [Google Scholar]
- Robins, J.M. Correcting for non-compliance in randomized trials using structural nested mean models. Commun. Stat.-Theory Methods
**1994**, 23, 2379–2412. [Google Scholar] [CrossRef] - Vansteelandt, S.; Joffe, M. Structural nested models and G-estimation: The partially realized promise. Stat. Sci.
**2014**, 29, 707–731. [Google Scholar] [CrossRef] - Rubin, D.B. Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol.
**1974**, 66, 688–701. [Google Scholar] [CrossRef] - Rubin, D.B. Causal inference using potential outcomes: Design, modeling, decisions. J. Am. Stat. Assoc.
**2005**, 100, 322–331. [Google Scholar] [CrossRef] - Hale, T.; Angrist, N.; Goldszmidt, R.; Kira, B.; Petherick, A.; Phillips, T.; Webster, S.; Cameron-Blake, E.; Hallas, L.; Majumdar, S.; et al. A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker). Nat. Hum. Behav.
**2021**, 5, 529–538. [Google Scholar] [CrossRef] - Robins, J. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. Math. Model.
**1986**, 7, 1393–1512. [Google Scholar] [CrossRef] - Haber, M. Estimation of the population effectiveness of vaccination. Stat. Med.
**1997**, 16, 601–610. [Google Scholar] [CrossRef] - Tanaka, S.; Matsuyama, Y.; Shiraki, M.; Ohashi, Y. Estimating the effects of time-varying treatments: Incidence of fractures among postmenopausal Japanese women. Epidemiology
**2007**, 18, 529–536. [Google Scholar] [CrossRef] [PubMed] - Aryaie, M.; Sharifi, H.; Saber, A.; Nazemipour, M.; Mansournia, M.A. Longitudinal Causal Effects of Normalized Protein Catabolic Rate on All-Cause Mortality in Patients With End-Stage Renal Disease: Adjusting for Time-Varying Confounders Using the G-Estimation Method. Am. J. Epidemiol.
**2021**, 190, 1133–1141. [Google Scholar] [CrossRef] [PubMed] - Robins, J.M.; Hernan, M.A.; Brumback, B. Marginal structural models and causal inference in epidemiology. Epidemiology
**2000**, 11, 550–560. [Google Scholar] [CrossRef] [PubMed] - Vansteelandt, S.; Sjolander, A. Revisiting g-estimation of the effect of a time-varying exposure subject to time-varying confounding. Epidemiol. Methods
**2016**, 5, 37–56. [Google Scholar] [CrossRef] - Zeger, S.L.; Liang, K.-Y.; Albert, P.S. Models for longitudinal data: A generalized estimating equation approach. Biometrics
**1988**, 44, 1049–1060. [Google Scholar] [CrossRef] [PubMed] - Liang, K.-Y.; Zeger, S.L. Longitudinal data analysis using generalized linear models. Biometrika
**1986**, 73, 13–22. [Google Scholar] [CrossRef] - Cole, S.R.; Platt, R.W.; Schisterman, E.F.; Chu, H.; Westreich, D.; Richardson, D.; Poole, C. Illustrating bias due to conditioning on a collider. Int. J. Epidemiol.
**2010**, 39, 417–420. [Google Scholar] [CrossRef] - Ainslie, K.E.C.; Haber, M.; Orenstein, W.A. Challenges in estimating influenza vaccine effectiveness. Expert Rev. Vaccines
**2019**, 18, 615–628. [Google Scholar] [CrossRef] - Imbens, G.W.; Rubin, D.B. Causal Inference in Statistics, Social, and Biomedical Sciences; Cambridge University Press: Cambridge, UK, 2015. [Google Scholar]
- Sah, P.; Vilches, T.N.; Moghadas, S.M.; Fitzpatrick, M.C.; Singer, B.H.; Hotez, P.J.; Galvani, A.P. Accelerated vaccine rollout is imperative to mitigate highly transmissible COVID-19 variants. EClinicalMedicine
**2021**, 35, 100865. [Google Scholar] [CrossRef] - U.S. Reaches 70% COVID Vaccine Milestone for Adults about a Month behind Biden’s Goal. Available online: https://www.cnbc.com/2021/08/02/covid-vaccine-us-reaches-bidens-70percent-goal-for-adults-a-month-behind.html (accessed on 5 April 2022).
- South Korea Coronavirus: PM Aims for ‘Herd Immunity by Autumn’. Available online: https://www.bbc.com/news/world-asia-56156234 (accessed on 5 April 2022).
- Africa Faces Steepest COVID-19 Surge yet. Available online: https://afrocoms.newsweaver.com/1rz5qc4oi8/imj9z0y2kyw?lang=en (accessed on 9 December 2021).

**Figure 2.**Predicted number of new cases in each week under different scenarios. (

**a**) shows the comparison of predicted number of new cases under no vaccination scenario and status quo. (

**b**) shows the comparison of predicted number of new cases under half speed scenario and twice speed scenario. (

**c**) shows the comparison of predicted number of new cases under 1% constant speed scenario and 4% constant speed scenario. (

**d**) shows the comparison of predicted number of new cases under speed up scenario and speed down scenario.

Variables | Mean | SD |
---|---|---|

Number of physicians (per capita) | 475.2 | 192.6 |

GDP (millions of chained 2012 dollars) | 375,880 | 490,590 |

Population | 6,551,748 | 7,415,328 |

Race composition (proportion of black people) | 0.132 | 0.109 |

Proportion of old people (aged 65 or above) | 0.164 | 0.020 |

Red or Blue state in 2016 election (Red = 1) | 0.6 | 0.495 |

Unemployment Rate (in March, 2021) | 5.56 | 1.72 |

Proportion of people with advanced degrees | 0.126 | 0.042 |

Sex ratio | 0.977 | 0.033 |

Cumulative cases at baseline | 569,301 | 664,218 |

Vaccination coverage at baseline (per 10,000 people) | 1572 | 238 |

Decline of Growth Rate | |||
---|---|---|---|

Estimate | SE | 95% CI | |

Main analysis | |||

SNMM with g-estimation | 1.02% | 0.0037 | (1.69%, 0.26%) |

GEE analysis | |||

GEE (adjust baseline covariates) | 0.754% | 0.00076 | (0.974%, 0.533%) |

GEE (adjust baseline and time-varying covariates) | 1.74% | 0.0035 | (2.42%, 1.05%) |

Fixed effects model | |||

Two-way fixed effects model | 1.52% | 0.0029 | (2.09%, 0.96%) |

Two-way fixed effects model (adjust time-varying covariates) | 1.87% | 0.0029 | (2.43%, 1.30%) |

Scenarios | Cumulated New Cases (Million) | Vaccination Effectiveness | ||
---|---|---|---|---|

Estimate (95% CI) | Difference (%) ^{a} | Estimate (95% CI) | Difference (%) ^{b} | |

Base case | ||||

Status quo | 4.55 ^{c} | / | 63.9% (18.0%, 87.5%) | / |

Scenario analysis | ||||

No-Vaccination | 12.60 (5.55, 36.51) | 8.05 (177%) | 0% | −63.9% |

vaccination speed: two times the status-quo speed | 2.84 (2.34, 3.93) | −1.71 (−37.6%) | 77.5% (29.2%, 93.6%) | 13.6% |

vaccination speed: half of the status-quo speed | 7.10 (5.03, 10.88) | 2.55 (56.0%) | 43.7% (9.34%, 70.2%) | −20.2% |

vaccination speed: 4% population per week | 3.99 (3.71, 4.40) | −0.56 (−12.3%) | 68.4% (20.7%, 89.8%) | 4.5% |

vaccination speed: 1% population per week | 8.66 (5.21, 16.06) | 4.11 (90.3%) | 31.3% (6.07%, 56.02%) | −32.6% |

Speed-down: 4% for first 7 weeks and 1% for last 6 weeks | 4.13 (3.91, 4.44) | −0.42 (−9.2%) | 67.3% (19.9%, 89.3%) | 3.4% |

Speed-up: 1% for first 6 weeks and 4% for last 7 weeks | 7.52 (5.10, 11.47) | 2.97 (65.3%) | 40.3% (8.10%, 68.6%) | −23.6% |

^{a.}The difference is the estimate of cases under each scenario minus the cases under the status quo. The percentage in the bracket is the ratio of difference over cases under the status quo.

^{b.}The difference refers to estimates of vaccine effectiveness in each scenario minus the effectiveness under the status quo.

^{c.}The observed cases under the status quo.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wang, R.; Wang, J.; Hu, T.; Zhou, X.-H. Population-Level Effectiveness of COVID-19 Vaccination Program in the United States: Causal Analysis Based on Structural Nested Mean Model. *Vaccines* **2022**, *10*, 726.
https://doi.org/10.3390/vaccines10050726

**AMA Style**

Wang R, Wang J, Hu T, Zhou X-H. Population-Level Effectiveness of COVID-19 Vaccination Program in the United States: Causal Analysis Based on Structural Nested Mean Model. *Vaccines*. 2022; 10(5):726.
https://doi.org/10.3390/vaccines10050726

**Chicago/Turabian Style**

Wang, Rui, Jiahao Wang, Taojun Hu, and Xiao-Hua Zhou. 2022. "Population-Level Effectiveness of COVID-19 Vaccination Program in the United States: Causal Analysis Based on Structural Nested Mean Model" *Vaccines* 10, no. 5: 726.
https://doi.org/10.3390/vaccines10050726