Measuring the Effectiveness of COVID-19 Vaccines Used during a Surge of the Delta Variant of SARS-CoV-2 in Bangladesh: A Test-Negative Design Evaluation
, Mallick Masum Billah 2, Monalisa 2, Shah Ali Akbar Ashrafi 3, Mohammed Ziaur Rahman 1, Omar Hamza Bin Manjur 4, Mokibul Hassan Afrad 1, S M Shamsuzzaman 5, Ahmed Abu Saleh 6, Mostafa Aziz Sumon 7, Tahmina Shirin 2, add
Show full author list
Round 1
Reviewer 1 Report
The work will be helpful for the future researchers who are working in the same direction but it has main limitations on the modeling aspects. The authors should focus on the modeling. How the research will be helpful for the community who are working on the mathematical modeling, statistical approach etc.
The COVID-19 pandemic has already spread throughout the world and the people are aware about the diseases and they are using precautions about the pandemic. But, still the covid-19 is spreading very quickly. There are major comments before considering the second round revision.
---- The abstract is a little thin and does not quite convey the vibrancy of the findings and the depth of the main conclusions. The authors should please extend this somewhat for a better first impression.
---- The manuscript lacks motivation. Author needs to include the motivation of the study.
-----Authors should write keywords in professional way.
----- There is already an abundance of modeling studies on COVID-19, vaccinations, and the months or years to come. However, apart from Ferguson's (now classic) work, Moore and Giordano, very little is said about similar modeling works. This is an issue for three reasons. First, the intended audience for such pieces is made of policy-makers and the general public: they are already facing an abundance of (occasionally conflicting) findings from models. If there is no attempt to contextualize the findings from this piece among others, then we're more likely to be adding noise to a crowded space, instead of providing valuable guidance. Second, several of the modeling assumptions made here may be in line with other pieces (which may provide some strength to the methods) or may be rather unique (which may need more discussion). Finally, as a piece of scientific literature, the contributions should be situated based on what already exists. In sum, the authors should explain how each of their assumptions and modeling choices compares to the literature; how their findings compare to the literature; and hence what is their specific contribution. Related models include, but are not limited to:
DOI: 10.1126/science.abe6959
https://doi.org/10.1140/epjs/s11734-022-00648-w
- https://doi.org/10.1371/journal.pone.0261869
https://doi.org/10.1007/s11071-022-07235-7
https://doi.org/10.1016/j.vaccine.2021.04.002
https://doi.org/10.1007/s12190-021-01507-y
https://doi.org/10.1140/epjp/s13360-021-01997-6
https://doi.org/10.1016/ S1473-3099(22)00320-6
----Some references contain errors and inconsistent formatting. It is difficult to give credit to research if even elementary aspects of the work are not error free. This should be corrected with care and love to detail.
----The manuscript is comprehensive, and I have enjoyed learning about the presented results. I find that the manuscript is written with very poor english and the presentation is not good, and I am in principal in favor of publication, although the following comments should nevertheless be accommodated in one major revision.
Author Response
The work will be helpful for the future researchers who are working in the same direction but it has main limitations on the modeling aspects. The authors should focus on the modeling. How the research will be helpful for the community who are working on the mathematical modeling, statistical approach etc.
The COVID-19 pandemic has already spread throughout the world and the people are aware about the diseases and they are using precautions about the pandemic. But, still the covid-19 is spreading very quickly. There are major comments before considering the second round revision.
---- The abstract is a little thin and does not quite convey the vibrancy of the findings and the depth of the main conclusions. The authors should please extend this somewhat for a better first impression.
Response:
We agree with the reviewer that the paper will be helpful to scientists working on COVID vaccines. We now state (page 17) that our findings will be useful for future modelling of vaccine deployments. Because of the word limits for Abstracts, we do not believe that it will be possible to broaden the Abstract while still conveying essential information on methods and results.
---- The manuscript lacks motivation. Author needs to include the motivation of the study.
Response:
We disagree with the observation that the manuscript lacks motivation. The last paragraph of the Introduction clearly states:
“Effectiveness studies, done after deployment, have been critical to monitoring protection by vaccines as the pandemic has progressed, especially because vaccines have been challenged by many of the new variants that have arisen.7 However, such studies have been done considerably more commonly in affluent than in poor countries, leaving the latter with gaps in critical information about real-world impact of the different constellation of vaccines used in these settings and among populations that have distinctive features that may impact vaccine protection in poorer settings.8 Here, we report on an evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test negative design, a variant of case control design, which has been successfully used in multiple vaccine evaluations in other countries 9-11”.
-----Authors should write keywords in professional way.
Response: Keywords have been updated.
----- There is already an abundance of modeling studies on COVID-19, vaccinations, and the months or years to come. However, apart from Ferguson's (now classic) work, Moore and Giordano, very little is said about similar modeling works. This is an issue for three reasons. First, the intended audience for such pieces is made of policy-makers and the general public: they are already facing an abundance of (occasionally conflicting) findings from models. If there is no attempt to contextualize the findings from this piece among others, then we're more likely to be adding noise to a crowded space, instead of providing valuable guidance. Second, several of the modeling assumptions made here may be in line with other pieces (which may provide some strength to the methods) or may be rather unique (which may need more discussion). Finally, as a piece of scientific literature, the contributions should be situated based on what already exists. In sum, the authors should explain how each of their assumptions and modeling choices compares to the literature; how their findings compare to the literature; and hence what is their specific contribution. Related models include, but are not limited to:
DOI: 10.1126/science.abe6959
https://doi.org/10.1140/epjs/s11734-022-00648-w
- https://doi.org/10.1371/journal.pone.0261869
https://doi.org/10.1007/s11071-022-07235-7
https://doi.org/10.1016/j.vaccine.2021.04.002
https://doi.org/10.1007/s12190-021-01507-y
https://doi.org/10.1140/epjp/s13360-021-01997-6
https://doi.org/10.1016/ S1473-3099(22)00320-6
Response:
We thank the reviewer for comments and references on mathematical models of COVID. However, our paper reports on an empiric, field evaluation of COVID vaccine protection and does not report on mathematical modelling of vaccine protection or COVID disease epidemiology. Furthermore, the Discussion of the paper contextualizes our findings in relation to the findings on the protection conferred by the same vaccines used in other settings (page 16, second and third complete paragraphs).
----Some references contain errors and inconsistent formatting. It is difficult to give credit to research if even elementary aspects of the work are not error free. This should be corrected with care and love to detail.
Response: The references have been updated.
----The manuscript is comprehensive, and I have enjoyed learning about the presented results. I find that the manuscript is written with very poor english and the presentation is not good, and I am in principal in favor of publication, although the following comments should nevertheless be accommodated in one major revision.
Response:
Professor John Clemens, a native English speaker with and an extensive track record in publishing scientific papers, has edited the English text. However, we are very open to specific suggestions on wording, such as that provided by Reviewer 3.
Reviewer 2 Report
Farhana Khanam et al reported an evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test negative design, a variant of case control design. Their finding suggested that only Moderna (mRNA-1273) provided significant protection against COVID-19 disease of any severity caused by the Delta variant, however Moderna (mRNA-1273) and Sinopharm (Vero Cell-inactivated) as well as Serum Institute of India (ChAdOx1 nCoV-19) all provided protection against severe disease. This is a well-done study, and the manuscript is technically well done. I only have a minor suggestion to the title of manuscript.
Since only the disease caused by Delta variant was evaluated, the title should clearly show that the effectiveness of COVID-19 vaccines is against Delta variant. It may be changed to: Effectiveness of COVID-19 vaccines during Delta wave in Bangladesh: a test-negative design evaluation
Author Response
Farhana Khanam et al reported an evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test negative design, a variant of case control design. Their finding suggested that only Moderna (mRNA-1273) provided significant protection against COVID-19 disease of any severity caused by the Delta variant, however Moderna (mRNA-1273) and Sinopharm (Vero Cell-inactivated) as well as Serum Institute of India (ChAdOx1 nCoV-19) all provided protection against severe disease. This is a well-done study, and the manuscript is technically well done. I only have a minor suggestion to the title of manuscript.
Since only the disease caused by Delta variant was evaluated, the title should clearly show that the effectiveness of COVID-19 vaccines is against Delta variant. It may be changed to: Effectiveness of COVID-19 vaccines during Delta wave in Bangladesh: a test-negative design evaluation
Response: We thank the Reviewer for positive comments on the paper. The title has been updated as per suggestion.
Reviewer 3 Report
Khanam et al. reported their statistical studies of the effectiveness of COVID-19 vaccines in Bangladesh. In these studies, they selected good numbers of test-negative controls and drew a clear conclusion that only one of the three evaluated vaccines has protection against COVID-19 delta variant infection, while all three vaccines have protection against severe disease. The authors also discussed the limitation of this study in the discussion section. However, the manuscript contains certain weaknesses: first, the topic is outdated since the Omicron variant has become the dominant variant now, which lowered the significance and novelty of this manuscript; second, some concerns about the statistical data should be addressed, including high p values. Please see my detailed comments below:
(1) in Table 2-4 and supplementary tables, some of the P values are high; please provide an explanation or discussion in the main text.
(2) The keywords selected in the keyword section are too vague. Other keywords may be added to better attract readers and give them an overview of the manuscript.
(3) In the sentence “The first approval for use of a
COVID-19 vaccine following large-scale trials was given by the Medicines and Healthcare
Products Regulatory Agency (MHRA) within 326 days after sequencing of the virus”, Please specify whether “the first approval” refers to the first approval in the world or a certain region.
(4) For references from websites (such as references 1, 2, 3, 6, and 9), please provide the web links.
(5) Please pay attention to inconsistent font size (Moderna) and formats of dates in the introduction section and fix them.
(6) For the sentence “Here, we report on an
evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test negative
design, a variant of case control design, which has been successfully used in multiple
vaccine evaluations in other countries”, references should be provided.
(7) It seems like the following sentence is missing a few words. Please fix it. “Though similar proportions 3 (1%) of the 313 cases and 10 (1%) of the 1196 controls had died by the (ultimately severe or not) showed significant associations only for being female and a non-smoker”.
Author Response
Khanam et al. reported their statistical studies of the effectiveness of COVID-19 vaccines in Bangladesh. In these studies, they selected good numbers of test-negative controls and drew a clear conclusion that only one of the three evaluated vaccines has protection against COVID-19 delta variant infection, while all three vaccines have protection against severe disease. The authors also discussed the limitation of this study in the discussion section. However, the manuscript contains certain weaknesses: first, the topic is outdated since the Omicron variant has become the dominant variant now, which lowered the significance and novelty of this manuscript;
Response: Although the current dominant variant is Omicron, not Delta, our paper is important because of the dearth of COVID vaccine evaluations done in developing countries. Indeed, the paper reports on the first such evaluation to have been done in Bangladesh. As well, the study reported on in the paper illustrates how findings on vaccine protection from affluent countries do not necessarily apply to developing countries.
second, some concerns about the statistical data should be addressed, including high p values. Please see my detailed comments below:
Response: We address the detailed comments below.
(1) in Table 2-4 and supplementary tables, some of the P values are high; please provide an explanation or discussion in the main text.
Response: The Discussion of the paper (page 16, first full paragraph) emphasizes the limited statistical power, due to inadequate sample sizes with resulting high P values, for many of the secondary and exploratory analyses.
(2) The keywords selected in the keyword section are too vague. Other keywords may be added to better attract readers and give them an overview of the manuscript.
Response: The updated keywords are Bangladesh; COVID-19 vaccines; Delta variant; Effectiveness; Test-negative design (TND).
(3) In the sentence “The first approval for use of a
COVID-19 vaccine following large-scale trials was given by the Medicines and Healthcare
Products Regulatory Agency (MHRA) within 326 days after sequencing of the virus”, Please specify whether “the first approval” refers to the first approval in the world or a certain region.
Response: The first approval refers to the first approval in the UK. We have mentioned it in the manuscript.
(4) For references from websites (such as references 1, 2, 3, 6, and 9), please provide the web links.
Response: Web links of the references have been provided.
(5) Please pay attention to inconsistent font size (Moderna) and formats of dates in the introduction section and fix them.
Response: The font size and format of dates have been updated.
(6) For the sentence “Here, we report on an
evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test negative
design, a variant of case control design, which has been successfully used in multiple
vaccine evaluations in other countries”, references should be provided.
Response: References have been provided (ref. numbers are 9-11).
(7) It seems like the following sentence is missing a few words. Please fix it. “Though similar proportions 3 (1%) of the 313 cases and 10 (1%) of the 1196 controls had died by the (ultimately severe or not) showed significant associations only for being female and a non-smoker”.
Response: Thanks for this observation. The sentence has been updated on page 7 as follows:
“though similar proportions 3 (1%) of the 313 cases and 10 (1%) of the 1196 controls had died by the 30 day follow-up. Comparisons of ultimately severe cases and their matched controls (ultimately severe or not) showed significant associations only for being female and a non-smoker.”
Round 2
Reviewer 1 Report
Manuscript Title: Effectiveness of COVID-19 vaccines in Bangladesh: a test-negative design evaluation
Manuscript Number: vaccines-2022106
Recommendation: Major Revision
The results listed in the paper in the form of formulas, figures, and analysis seems true and correct. The paper is well written and it is written in a truly sporty manner. English is generally good, I think it needs to be polished further and some typos need to be revised. Further punctuation marks should be checking through the paper, especially after the equations and at the end of the statements.
* Remark, comments, and questions:
----- Title of paper is not clear. Try to clear meaning of the title.
------ The abstract is a little thin and does not quite convey the vibrancy of the findings and the depth of the main conclusions. The authors should please extend this somewhat for a better first impression.
------ The manuscript lacks motivation. Author needs to include the motivation of the study.
------Authors should write keywords in professional way.
----- There is already an abundance of modeling studies on COVID-19, vaccinations, and the months or years to come. However, apart from Ferguson's (now classic) work, Moore and Giordano, very little is said about similar modeling works. This is an issue for three reasons. First, the intended audience for such pieces is made of policy-makers and the general public: they are already facing an abundance of (occasionally conflicting) findings from models. If there is no attempt to contextualize the findings from this piece among others, then we're more likely to be adding noise to a crowded space, instead of providing valuable guidance. Second, several of the modeling assumptions made here may be in line with other pieces (which may provide some strength to the methods) or may be rather unique (which may need more discussion). Finally, as a piece of scientific literature, the contributions should be situated based on what already exists. In sum, the authors should explain how each of their assumptions and modeling choices compares to the literature; how their findings compare to the literature; and hence what is their specific contribution. Related models include, but are not limited to:
--https://doi.org/10.1016/j.chaos.2020.110173
--https://doi.org/10.1016/j.chaos.2020.110049
-- https://doi.org/10.1007/s11071-022-07235-7
------Authors should insert all figures in appropriate places.
------Conclusion should be written in a more clear way. So try to short it and write in a professional way.
------Analysis is missing in paper so add it.
------Authors should improve the English of paper.
------Authors should correct grammatical error at few stage of paper.
-------Presentation of paper should be improved.
-------Try to reduce similarity of work.
-------References list are not appropriate.
Author Response
REVIEWER 1:
The results listed in the paper in the form of formulas, figures, and analysis seems true and correct.
Response: Many thanks for the positive comment from the Reviewer.
The paper is well written and it is written in a truly sporty manner. English is generally good, I think it needs to be polished further and some typos need to be revised. Further punctuation marks should be checking through the paper, especially after the equations and at the end of the statements.
Response: We thank the Reviewer for pointing this out. We have gone through the paper carefully to correct typographical errors and other infelicities.
* Remark, comments, and questions:
----- Title of paper is not clear. Try to clear meaning of the title.
Response: We have revised the title to better describe the study reported in the paper as follows: Measuring the effectiveness of COVID-19 vaccines used during a surge of the Delta variant of SARS-CoV-2 in Bangladesh: a test-negative design evaluation.
------ The abstract is a little thin and does not quite convey the vibrancy of the findings and the depth of the main conclusions. The authors should please extend this somewhat for a better first impression.
Response: We have expanded the conclusion of the abstract to convey more fully how our paper adds to existing studies on COVID-19 vaccines. We have added the following sentence to the abstract:
“Because our findings are not in complete accord with evaluations of the same vaccines in more affluent settings, our study underscores the need for country-level COVID-19 vaccine evaluations in developing countries.”
------ The manuscript lacks motivation. Author needs to include the motivation of the study.
Response: We appreciate the Reviewer’s comment, but respectfully disagree. As the last paragraph of the Introduction clearly states:
“Effectiveness studies, done after deployment, have been critical to monitoring protection by vaccines as the pandemic has progressed, especially because vaccines have been challenged by many of the new variants that have arisen.7 However, such studies have been done considerably more commonly in affluent than in poor countries, leaving the latter with gaps in critical information about real-world impact of the different constellation of vaccines used in these settings and among populations that have distinctive features that may impact vaccine protection in poorer settings.8 Here, we report on an evaluation of the effectiveness of COVID-19 vaccines in Bangladesh, using the test-negative design, a variant of case control design, which has been successfully used in multiple vaccine evaluations in other countries 9-11”.
------Authors should write keywords in professional way.
Response: Keywords have been updated as follows: Bangladesh; COVID-19 vaccines; Delta variant; Vaccine effectiveness; Test-negative design (TND)
----- There is already an abundance of modeling studies on COVID-19, vaccinations, and the months or years to come. However, apart from Ferguson's (now classic) work, Moore and Giordano, very little is said about similar modeling works. This is an issue for three reasons. First, the intended audience for such pieces is made of policy-makers and the general public: they are already facing an abundance of (occasionally conflicting) findings from models. If there is no attempt to contextualize the findings from this piece among others, then we're more likely to be adding noise to a crowded space, instead of providing valuable guidance. Second, several of the modeling assumptions made here may be in line with other pieces (which may provide some strength to the methods) or may be rather unique (which may need more discussion). Finally, as a piece of scientific literature, the contributions should be situated based on what already exists. In sum, the authors should explain how each of their assumptions and modeling choices compares to the literature; how their findings compare to the literature; and hence what is their specific contribution. Related models include, but are not limited to:
--https://doi.org/10.1016/j.chaos.2020.110173
--https://doi.org/10.1016/j.chaos.2020.110049
-- https://doi.org/10.1007/s11071-022-07235-7
Response: We agree with the Reviewer that contextualization of our findings in relationship with existing literature is critical, and we also fully agree that differences in methods and assumptions of the study can account for differences in findings. Accordingly, we now stress in the Discussion (pages 16, 17) that our study design was very similar in assumptions and approach to other evaluations of the protection conferred by the same COVID-19 vaccines in studies done in other countries. This point should assure the reader that differences in our study findings were not likely the result merely of differences in our study assumptions and design. We also devote two full paragraphs in the Discussion to describing how our study findings on vaccine protection against overall and severe COVID-19 disease did or did not differ from the findings of analogous studies done in other countries (pages 16, 17). Because our study was not a modelling analysis, but rather a direct field evaluation of vaccine clinical effectiveness, we believe that these elements of the paper now sufficiently contextualize the results of our study.
------Authors should insert all figures in appropriate places.
Response: We submitted the main paper and figures separately and organization of the figures was done by the editor office. However, we will make any needed further changes in locations when we receive the galleys.
------Conclusion should be written in a more clear way. So try to short it and write in a professional way.
Response: Our paper does not have separate heading for Conclusion. However, we believe that the final paragraph of the Discussion, now revised (pages 16, 17) adequately serves this purpose. Moreover, as indicated in the earlier response to the Reviewer we have expanded the conclusion of the abstract.
------Analysis is missing in paper so add it.
Response: We present the details of the analysis in the section on “Sample size estimation and statistical analysis” (pages 4, 5).
------Authors should improve the English of paper.
Response: As we have noted in our earlier comment that we have gone over the paper and attempted to correct grammar and punctuation, and to rectify typographical errors.
------Authors should correct grammatical error at few stage of paper.
Response: Please see our previous responses.
-------Presentation of paper should be improved.
Response: We have modified our paper in multiple ways, as noted in earlier responses.
-------Try to reduce similarity of work.
Response: As indicated in our previous responses, we have modified the text to indicate more clearly how our study conforms to and differs from the broad literature of real world of COVID-19 vaccines.
-------References list are not appropriate.
Response: We have checked the references list and replaced the website reference with a research article (reference no. 8).
Reviewer 3 Report
The authors have addressed my comments appropriately. It is worth noting that, although the authors have talked about the limitations in the statistical power of this project in the discussion section, I would still recommend commenting on high P values somewhere in the result sections, to remove confusion from the readers. Nevertheless, I have no big concerns about this manuscript at this time. Although the Delta variant is no longer the dominant one, the findings in this manuscript would still provide insights into the effectiveness of different types of vaccines in developing countries and future vaccine deployments.
Author Response
The authors have addressed my comments appropriately. It is worth noting that, although the authors have talked about the limitations in the statistical power of this project in the discussion section, I would still recommend commenting on high P values somewhere in the result sections, to remove confusion from the readers. Nevertheless, I have no big concerns about this manuscript at this time. Although the Delta variant is no longer the dominant one, the findings in this manuscript would still provide insights into the effectiveness of different types of vaccines in developing countries and future vaccine deployments.
Response: We thank the Reviewer for this comment. We have added one comment regarding high P values in the Discussion on page 16.
