DARE-SAFE: Denominator-Adjusted Rate Estimates of Substance Adverse Events Frequency Evaluation in Pharmaceuticals and Vaccines

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

A review report of the manuscript entitled “DARE-SAFE: Denominator-Adjusted Rate Estimates of Substance Adverse Events Frequency Evaluation in Pharmaceuticals and Vaccines”

• (page 1, line 9) In the beginning of the abstract part, what is known or contextual background of this study should be provided.
• (page 1, abstract) Authors need to confirm that all abbreviations are defined before being used for the first time. For instance, CDC, US FDA, COVID-19.
• (page 2, lines 54-55) the authors stated that “This promotion by medical professional belies significant financial incentives for physicians to promote vaccines, which includes bonuses up to tens of thousands of dollars.” However, in my opinion, this statement might be misleading as it claims that medical professionals promote vaccines merely for their benefits, in this case financial gain, which misrepresents the incentive structure. There are indeed incentives for doctors to get vaccines, but they are usually based on quality-of-care metrics and patient coverage thresholds rather than direct financial bonuses for each vaccine. In addition, the claim of “bonuses up to tens of thousands of dollars” is not universal and should be revised by giving more context. Consider revising this sentence to sound more neutral. For example, "Rather than direct payments for each vaccine, medical professionals may get financial incentives based on how many people get vaccinated. This is usually part of quality-of-care programs." These rewards are meant to get people to follow public health advice.”
• (page 1, lines 32-33) I suggest that the authors first introduce the acronyms of “US FDA Adverse Event Reporting System” and “Vaccine Adverse Events Reporting System” on their first mention in the introduction so that the writers can use the acronyms later on in their manuscript. Repetition of such long terms may not be as effective, and a late introduction of those acronyms could potentially cause confusion among readers. Thanks!
• (page 2, lines 50-52) this sentence “…vaccines are understood to have side effects, ranging from site pain and cold and flu symptoms to severe impairments and death…” will be more significant if it’s also supported by another relevant study. For example, https://pmc.ncbi.nlm.nih.gov/articles/PMC10914054/
• (page 2, lines 60 and 64) the term for “COVID-19” should be uniformly written throughout this manuscript. Please change the current “Covid-19” to “COVID-19” for consistency.
• Authors are suggested to proofread the manuscript after addressing all comments to avoid any typological, grammatical, and lingual mistakes and errors. For example, “If the number of doses is known, than it is possible to provide” on page 2 lines 67-68.
• (page 3, line 101) the research motivation appears somewhat superficial. Please clarify the significance of understanding adverse events for drugs and vaccines, and explicitly state how your study contributes to the expanding body of knowledge in this field.
• (page 3, methods) the methods section lacks information on how the data was analyzed. Please specify how reporting rates were normalized or if any statistical tests were performed (for example, confidence intervals, correlation analysis).
• (page 3, lines 138-139) authors reported that "Using a fixed intercept model, and weighting by the number of VAERS reports, the ratio of total reports to deaths is 70 (R²=0.966)." However, they did not explain why this model was chosen in the methods section. And the R² value of 0.966 is high, which indicates an almost perfect correlation. This strong association might be due to data limitations, and I suggest that the authors cross-check the results one more time to make sure that they are correctly reported.
• (page 6, lines 177-179) authors mentioned that "The analysis also revealed a consistent ratio of approximately 70:1 for total VAERS reports to deaths for vaccines. This ratio was determined using a fixed intercept model weighted by the number of VAERS reports, with an R² value of 0.966, indicating a strong correlation." Nevertheless, the correlation (R²=0.966) does not imply causation. A fixed intercept model may oversimplify the relationship between VAERS reports and deaths, as other factors (age, comorbidities, underreporting of non-fatal events) could influence the ratio. I found no statistical validation (e.g., confidence intervals, sensitivity analyses) mentioned to support the reliability of this model.
• (Page 6, Discussion) This section lacks a detailed explanation and comparison to similar studies. Additionally, the authors have not provided an analysis of the study’s strengths and limitations. Furthermore, the conclusion section is missing.
• (page 2, lines 48-50) this sentence “Pharmaceutical drugs are commonly understood to have side effects associated with These side effects are measured against their benefits to determine if it is in the patient’s best interest to take a prescription for a drug.” also shares the same idea with a similar study by https://www.narrax.org/main/article/view/80/48. Kindly include this reference to make the sentence stronger.

Author Response

My apologies, the previous comments were meant for the other reviewer. I have attached the correct response below.

A review report of the manuscript entitled “DARE-SAFE: Denominator-Adjusted Rate Estimates of Substance Adverse Events Frequency Evaluation in Pharmaceuticals and Vaccines”

Reviewer: (page 1, line 9) In the beginning of the abstract part, what is known or contextual background of this study should be provided.

Response: We have added a background section to the abstract, communicating the background and our motivation to embark on this study.

Response: We thank the reviewer for this helpful suggestion, as it has stimulated our own thinking. We have added the following to 'Pharmacovigilance reporting for pharmaceutical drugs'

 

Establishing the VAERS and FAERS reporting rates for vaccines and pharmaceuticals is useful for performing a risk assessment when recommending vaccines or drugs. This may also enable estimations of healthcare usage associated with drugs and pharmaceuticals. This also serves as a potential benchmark for risk prediction using in silico (6), in vitro(7) or in vivo assays (8)for drug toxicity. Successful pharmacovigilance uses many methodologies, and synthesizes insights from computational or systems biology prediction of toxic effects, with cell and animal models and lastly clinical and post surveillance data in humans. DARE-SAFE provides risk estimation from existing pharmacovigilance data.

 

Reviewer: (page 1, abstract) Authors need to confirm that all abbreviations are defined before being used for the first time. For instance, CDC, US FDA, COVID-19.

Response: We have corrected the use of acronyms so that they are first spelled out at first use, and the acronym is used subsequently.

 

Reviewer: (page 2, lines 54-55) the authors stated that “This promotion by medical professional belies significant financial incentives for physicians to promote vaccines, which includes bonuses up to tens of thousands of dollars.” However, in my opinion, this statement might be misleading as it claims that medical professionals promote vaccines merely for their benefits, in this case financial gain, which misrepresents the incentive structure. There are indeed incentives for doctors to get vaccines, but they are usually based on quality-of-care metrics and patient coverage thresholds rather than direct financial bonuses for each vaccine. In addition, the claim of “bonuses up to tens of thousands of dollars” is not universal and should be revised by giving more context. Consider revising this sentence to sound more neutral. For example, "Rather than direct payments for each vaccine, medical professionals may get financial incentives based on how many people get vaccinated. This is usually part of quality-of-care programs." These rewards are meant to get people to follow public health advice.”

 

Response: We thank the reviewer for their comment, helping with the neutrality of the manuscript. We have changed the selection to:

 

“Promotion by physicians is encouraged by public health agencies and quality of care programs of medical organizations or insurance providers. This may include financial incentives based on how many people get vaccinated (3,4).”

 

Reviewer: (page 1, lines 32-33) I suggest that the authors first introduce the acronyms of “US FDA Adverse Event Reporting System” and “Vaccine Adverse Events Reporting System” on their first mention in the introduction so that the writers can use the acronyms later on in their manuscript. Repetition of such long terms may not be as effective, and a late introduction of those acronyms could potentially cause confusion among readers. Thanks!

Response: We thank the reviewer for this suggestion and have changed all instances to VAERS and FAERS respectively.

Reviewer: (page 2, lines 50-52) this sentence “…vaccines are understood to have side effects, ranging from site pain and cold and flu symptoms to severe impairments and death…” will be more significant if it’s also supported by another relevant study. For example, https://pmc.ncbi.nlm.nih.gov/articles/PMC10914054/

Response: We thank the reviewer for this useful article, we have added this citation on the range of side effects experienced.

 

Reviewer: (page 2, lines 60 and 64) the term for “COVID-19” should be uniformly written throughout this manuscript. Please change the current “Covid-19” to “COVID-19” for consistency.

Response: We have replaced all instances of “Covid-19” or other variants to COVID-19.

Reviewer: Authors are suggested to proofread the manuscript after addressing all comments to avoid any typological, grammatical, and lingual mistakes and errors. For example, “If the number of doses is known, than it is possible to provide” on page 2 lines 67-68.

Response: We thank the reviewer for pointing out this mistake and have conducted a grammar check beyond correcting this error.

Reviewer: (page 3, line 101) the research motivation appears somewhat superficial. Please clarify the significance of understanding adverse events for drugs and vaccines, and explicitly state how your study contributes to the expanding body of knowledge in this field.

 

Reviewer: (page 3, methods) the methods section lacks information on how the data was analyzed. Please specify how reporting rates were normalized or if any statistical tests were performed (for example, confidence intervals, correlation analysis).

Response: We have elaborated on our weighted fits in MS Excel using the LINEST function. We have included error values on the parameters which allow the reader to assess the possible range of values.

Reviewer: (page 3, lines 138-139) authors reported that "Using a fixed intercept model, and weighting by the number of VAERS reports, the ratio of total reports to deaths is 70 (R²=0.966)." However, they did not explain why this model was chosen in the methods section. And the R² value of 0.966 is high, which indicates an almost perfect correlation. This strong association might be due to data limitations, and I suggest that the authors cross-check the results one more time to make sure that they are correctly reported.

Response: We have changed our weighting to by dose rather than by AE report number, this has changed our results slightly, using the new weighting, the R² is 0.94. We have elaborated on our choice of a fixed intercept linear model, basically that we assume a direct proportionality between adverse events and deaths, as there will be a proportion of adverse events which result in death. The non-zero intercept would either correspond to having deaths without AE reports, or having AE reports without deaths, which may happen as a statistical artifact for low sample sizes with no deaths, but is unlikely to be reflective of a relationship when there is sufficient data.

 

Reviewer: (page 6, lines 177-179) authors mentioned that "The analysis also revealed a consistent ratio of approximately 70:1 for total VAERS reports to deaths for vaccines. This ratio was determined using a fixed intercept model weighted by the number of VAERS reports, with an R² value of 0.966, indicating a strong correlation." Nevertheless, the correlation (R²=0.966) does not imply causation. A fixed intercept model may oversimplify the relationship between VAERS reports and deaths, as other factors (age, comorbidities, underreporting of non-fatal events) could influence the ratio. I found no statistical validation (e.g., confidence intervals, sensitivity analyses) mentioned to support the reliability of this model.

Response: We have re-run our analysis weighting by the number of doses (for vaccines) or the number of prescriptions (for drugs). We get somewhat different results. For each fit, we have included the parameters, with errors, in supplemental tables.

 

Reviewer: (Page 6, Discussion) This section lacks a detailed explanation and comparison to similar studies. Additionally, the authors have not provided an analysis of the study’s strengths and limitations. Furthermore, the conclusion section is missing.

Response: We thank the reviewer for pointing out the need for an overview of the study, including its limitations. While we have emphasized these int eh previous sections, we believe the main strength is providing denominators for the calculation of adverse event rates of pharmaceuticals and vaccines. The major strength of the study is it provides quantification of the observed relationship between different severities of adverse events in databases. Additionally, we provide the adverse event rates per does for the top 250 pharmaceuticals (by prescriptions) and the major vaccines used in the USA.

This can be built upon by explaining the differences in danger signals between various pharmaceuticals and vaccines. Adverse event rates for vaccines differ by several orders of magnitude, and the same applies within pharmaceuticals. Explaining the difference in risk profile can yield benefits in the understanding of adverse effects from medications and vaccinations.

 

 

Reviewer: (page 2, lines 48-50) this sentence “Pharmaceutical drugs are commonly understood to have side effects associated with These side effects are measured against their benefits to determine if it is in the patient’s best interest to take a prescription for a drug.” also shares the same idea with a similar study by https://www.narrax.org/main/article/view/80/48. Kindly include this reference to make the sentence stronger.

 

Response: We reiterate that the putative benefits of a drug or vaccine must be balanced against the potential benefit. While we thank the reviewer for the suggestion, we do not believe it to only partially relevant to the topic at hand. We agree with the article in that chemometrics can be used for determination of biodistribution and excretion of drugs and their metabolites. Furthermore, we believe the integration of grater omics technologies in clinical trials provides higher fidelity information about drug response.

Reviewer 2 Report

Comments and Suggestions for Authors

A good study with the right intent to high light potential adverse events associated with medicines and vaccines but no mention of the power of the study nor any mention of statistical significance.

Would advise to add potential co-morbidities in patients particularly in the ones receiving COVID vaccines . 
 
Also there is a potential for bias as these adverse events are either reported by the patients themselves or by the health care professionals themselves which could be proved based on certain clinical /diagnostic but they might also be self perceived potentially causing bias.

Suggestion to confirm whether the adverse events reported were confirmed to be scientifically proven .

Author Response

Reviewer Comment 1

A good study with the right intent to highlight potential adverse events associated with medicines and vaccines but no mention of the power of the study nor any mention of statistical significance.

Author Response

We have added error values to our fits as well as provided the reader access to the statistical parameters in the supplemental information.

Reviewer Comment 2

Would advise to add potential co-morbidities in patients particularly in the ones receiving COVID vaccines.

Author Response

There is a potential confounder in that those that receive standard vaccines may have a different health status than those receiving Covid-19 vaccines. However, we do not have access to this in a structured way with VAERS or FAERS reports.

Reviewer Comment 3

Also there is a potential for bias as these adverse events are either reported by the patients themselves or by the health care professionals themselves which could be proved based on certain clinical/diagnostic but they might also be self perceived potentially causing bias.

Author Response

This is an accurate point, and we have discussed the caveat that reports are passive and made by individuals or their medical providers. As not all events are reported, there is a potential for inconsistencies in reporting between different vaccines or pharmaceuticals. It would be possible to observe this through access to an active surveillance tool, such as the vaccine safety datalink, but they only provide access to datasets published within the last ten years, and there are presently no published datasets that are relevant or even accessible.

Reviewer Comment 4

Suggestion to confirm whether the adverse events reported were confirmed to be scientifically proven.

Author Response

Unfortunately, the database does not provide any identifying data, due to patient privacy, and so it is not possible to verify individual reports.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I appreciate the effort that authors put in to provide the contextual background in the abstract. However, I still want to provide room for improvement on that part where authors should have put the meaning of the AE acronym on line 11. Afterwards, authors can use AE instead of repeatedly providing its full definition throughout the manuscript.

(lines 62-63) Authors replied that they have added the citation I suggested before in the updated manuscript. However, I could not find it anywhere. Here is the article that also shares similar ideas for the related phrase: https://pmc.ncbi.nlm.nih.gov/articles/PMC10914054/  

(lines 72-80) Where is the citation for this claim?

The knowledge gap and the motivation behind the research are still shallow and need more depth. Please rectify this by highlighting why this research is important and what novelty this study brings to the table.

The tenses used to explain the methods should be in past tenses.

The current manuscript is difficult to re-review as authors did not upload a clear one without track changes. Please do so next time. Thanks!

Author Response

Comment 1: I appreciate the effort that authors put in to provide the contextual background in the abstract. However, I still want to provide room for improvement on that part where authors should have put the meaning of the AE acronym on line 11. Afterwards, authors can use AE instead of repeatedly providing its full definition throughout the manuscript.

Response 1: We apologize for this mistake, and thank the reviewer for pointing it out. We have defined the acronym at its first usage. 

 

Comment 2: (lines 62-63) Authors replied that they have added the citation I suggested before in the updated manuscript. However, I could not find it anywhere. Here is the article that also shares similar ideas for the related phrase: https://pmc.ncbi.nlm.nih.gov/articles/PMC10914054/  

Response 2: We thank the reviewer for pointing this out. We had not updated the references, and the reference mentioned by reviewer 2 is now included. 

Comment 3: (lines 72-80) Where is the citation for this claim?

Response 3: We thank the reviewer for this clarification. We have changed this paragraph considerably and added references. It now reads

"

While vaccines can lower the incidence and severity of infectious diseases [5], they may not be appropriate in every case [6]. For a proper comparison of utility with downsides, it is important to quantify levels of risk to compare with benefits [7]. Fear of adverse effects is a prominent reason for vaccine hesitancy [8], and vaccine hesitant people tend to not be swayed by messaging they perceive as dismissive of safety concerns [9]. Polls of parents show a higher degree of skepticism towards Covid-19 vaccines than other vaccines, such as Measles, Mumps and Rubella (MMR), commonly used in the childhood vaccination schedule [10]. A comparison of safety profiles may help researchers to address vaccine hesitancy."

 

Comment 4. The knowledge gap and the motivation behind the research are still shallow and need more depth. Please rectify this by highlighting why this research is important and what novelty this study brings to the table.

Response 4. The reviewer correctly points out that we have not explained the knowledge gap accurately, nor the implications of the study. We have pointed out the knowledge gaps in the current lack of comparisons between reporting rates between various vaccines at the end of the section 'The problem of inferring rates in the Vaccine Adverse Event Reporting System'.

"Currently, a lack of standardized methods for comparing reporting rates between different vaccines hampers regulator’s ability to assess and compare safety profiles of vaccines, which DARE-SAFE aims to enable."

 

We have also pointed out a similar knowledge gap for pharmaceuticals using FAERS: "DARE-SAFE provides risk estimation from existing pharmacovigilance data, enabling comparisons between the reporting rates of various drugs, which the FAERS dashboard by itself does not provide."

We have also explained the need for the study in the introduction, as DARE-SAFE may be a tool for identifying different reporting behaviour between products. 

"Pharmacovigilance is crucial for monitoring the safety of drugs and vaccines post-marketing. However, passive reporting systems like VAERS and FAERS face limitations, including underreporting and inconsistent reporting practices. Understanding these reporting rates is vital for assessing the relative safety of different medical products."

 

 

Comment 5: The tenses used to explain the methods should be in past tenses.

Response 5: We apologize, and have corrected our verbs in present tense to past tense.

"We report both the total..." to  "We reported both the total..." 

"We search the number of VAERS reports for a given vaccine type using the resource OpenVAERS.com..." to "We searched the number of VAERS reports for a given vaccine type using the resource OpenVAERS.com..."

"We report both the total number of reports, as well as the deaths for a given vaccine type..." to "We reported both the total number of reports, as well as the deaths for a given vaccine type..."

"We delineate the VAERS reporting rates by COVID-19 vaccine manufacturer..." to "We delineated the VAERS reporting rates by COVID-19 vaccine manufacturer..."

"Data on the number of prescriptions for a given drug per year are obtained through the website https://clincalc.com..." to "Data on the number of prescriptions for a given drug per year were obtained through the website https://clincalc.com..."

Comment 6: The current manuscript is difficult to re-review as authors did not upload a clear one without track changes. Please do so next time. Thanks!

Response 6: We apologize for this oversight. We have accepted changes to produce a clean version. 

 

Reviewer 2 Report

Comments and Suggestions for Authors

Appreciate your answers to the questions put forth .The only concern I have is that the questionnaire is passive and there is a possibility of bias from the patients side as well as from the providers side .

appreciate the sharing of the statistics. Overall a good presentation of data

Author Response

Comments 1: 

Appreciate your answers to the questions put forth .The only concern I have is that the questionnaire is passive and there is a possibility of bias from the patients side as well as from the providers side .

appreciate the sharing of the statistics. Overall a good presentation of data

 

Response 1: We appreciate the reviewer's comments and we have added to the discussion on the study's limitations.

"

Any passive pharmacovigilance system carries with it the caveat that many of the reports are not verified, and reports cannot be verified as reports are anonymous. Ideally, an analysis would include active surveillance data; active surveillance data, such as with the Vaccine Safety Datalink (VSD) has several advantages, as it follows a group of people for a period of time following vaccination, the amount of doses administered and patients is well known, and rates of AEs can be calculated. However, access to VSD data is limited for scientists outside the CDC and VSD network. However, the VSD only allows the public to access datasets from already published studies, published within the last ten years, and does not provide raw data [25]. Additionally, they provide links to only two published datasets available to the public [26,27], which are both published more than ten years ago (in 2007 and in 2010), meaning that their datasets are inaccessible to the public."

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

One thing that needs to improve is the quality of Figure 3. The labels are very small. 
The font of all figures should be changed to be consistent with the font of the texts. 

The layout and the texts are very messy. The title of the table should be placed on top of the tables.
Authors really need to follow the guidelines from the journal. Use the heading appropriately as prepared by the template of the journal. 

Author Response

Comment 1: One thing that needs to improve is the quality of Figure 3. The labels are very small. 
The font of all figures should be changed to be consistent with the font of the texts.

Response 1: We thank the reviewer for this suggestion to improve readability. We have converted all fonts to Times New Roman, consistent with the text and have increased the size of the labels in Figure 3. 

Comment 2: The layout and the texts are very messy. The title of the table should be placed on top of the tables.

Response 2: We thank the reviewer for this comment. We have changed the cell size in Table 3 so that all the drug names fit on a single line. The table appeared more messy before. Additionally, we have put all table captions on top of the tables.

Comment 3: Authors really need to follow the guidelines from the journal. Use the heading appropriately as prepared by the template of the journal.  

Response 3: We thank the reviewer for this suggestion. We have adopted the heading conventions of MDPI and have labelled our headings with decimal numbers for subheadings.