Next Article in Journal
Impact of Incretin Mimetics on Thyroid Cancer Among Patients with Type 2 Diabetes: A Retrospective Cohort Time-to-Event Analysis
Previous Article in Journal
DARE-SAFE: Denominator-Adjusted Rate Estimates of Substance Adverse Events Frequency Evaluation in Pharmaceuticals and Vaccines
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Pharmacoepidemiology
Volume 4
Issue 2
10.3390/pharma4020008
pharmacoepidemiology-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Timing Matters: Exploring the Role of the Time to Onset in Recall Bias for Adverse Events Following Immunization (AEFIs) of COVID-19 Vaccines from Spontaneous Reports

by
Joep Scholl
1,
Florence van Hunsel
1,2,* and
Eugene van Puijenbroek
1,2
1
Netherlands Pharmacovigilance Centre Lareb, Goudsbloemvallei 7, 5237 MH ’s-Hertogenbosch, The Netherlands
2
Department of PharmacoTherapy—Epidemiology & Economics, Groningen Research Institute of Pharmacy (GRIP), University of Groningen, Broerstraat 5, 9712 CP Groningen, The Netherlands
*
Author to whom correspondence should be addressed.
Pharmacoepidemiology 2025, 4(2), 8; https://doi.org/10.3390/pharma4020008
Submission received: 27 January 2025 / Revised: 24 March 2025 / Accepted: 2 April 2025 / Published: 7 April 2025
Browse Figures
Versions Notes

Abstract

:
Objective: The aim of this study was to investigate a possible relationship between the time to onset (TTO) of adverse events following immunization (AEFIs) and recall bias and to compare it between AEFIs of COVID-19 vaccines reported though the spontaneous reporting system (SRS) and those from a cohort event monitoring (CEM) study. Methods: A retrospective study comparing TTO patterns of AEFIs of four COVID-19 vaccines from the SRS and those from a CEM study was performed. Reports concerning AEFIs related to COVID-19 vaccination were used for the study. TTO patterns were stratified for vaccination dose number, perceived burden of the AEFI, and for being pre-defined. Additionally, since menstrual disorders received much media attention, their effect on the TTO pattern was investigated for SRS reports only. Results: A total of 160,613 reports from the SRS and 19,979 from the CEM, containing 755,647 and 103,703 AEFIs, respectively, were included. For AEFIs with a short TTO, no differences in TTO patterns were observed. However, the median TTO for AEFIs from the SRS was lower with increasing TTO duration. There were differences in both median TTO and time to reporting for AEFIs reported before and during episodes of media attention, but no correlation between the two could be found. Conclusions: Based on the performed TTO analyses, recall bias does not seem to be more evident in SRS compared to CEM studies for AEFIs with a short TTO. For AEFIs with a longer TTO, this may be more pronounced.

1. Introduction

Spontaneous reporting of adverse drug reactions (ADRs) and adverse events following immunization (AEFIs) has a vital role in pharmacovigilance. Spontaneous reporting systems (SRS) have proven to be fast and efficient ways to generate a hypothesis on the safety signals of medicines, including vaccines, based on clinical observations [1,2]. A signal in this context is ‘information on a new or known adverse event that is potentially caused by a medicine and that warrants further investigation’ [3].
The signal detection process in pharmacovigilance is usually based on either a qualitative analysis of individual case safety reports (ICSRs) or a quantitative analysis of the dataset, e.g., disproportionality analysis. These methods may be applied to different data sources. During the COVID-19 pandemic, SRS had an important place in the (global) safety monitoring of the COVID-19 vaccines [4], but AEFIs were also monitored prospectively with other methods such as in a large longitudinal cohort event monitoring study. Often multiple, complementary monitoring systems were used in pharmacovigilance for the detection, assessment, and verification of vaccine safety signals [5,6,7,8,9,10].
Considering the detection of signals, various types of bias may interfere with both these qualitative and quantitative signal detection processes [11,12,13,14]. One of these is recall bias, a type of information bias that is based on differences in the ability to correctly remember information between groups. It is conceivable that this type of bias affects spontaneously reported AEFIs more than those acquired from other data sources. In particular, it is conceivable that the absence of a trigger for reporting other than implied causality would make spontaneous reporting systems more susceptible to recall bias. This would be less of an issue for AEFIs occurring in prospective studies (clinical trials/prospective cohort studies) because participants are actively asked about possible AEFIs on multiple occasions during the study, which may be a trigger for reporting. However, to the best of our knowledge, differences in reporting patterns possibly related to recall bias between spontaneous data and trial/cohort data have never been investigated, and no publications on this topic could be found in the scientific literature.
Investigating recall bias in spontaneous reporting poses a challenge as this type of bias may lead to the underreporting of AEFIs. In other types of information bias, it may be possible to gather additional information from reporters or involve their healthcare providers to obtain accurate information. For example, the risk of misclassification of an AEFI can be reduced using specific (follow-up) questions that support the reporting of a correct diagnosis. In the case of spontaneous reporting, recall bias likely results in the underreporting of AEFIs with a longer time to onset (TTO) or in a relative higher reporting of those with a short TTO. This is because, in general, a reporter is more likely to assume a causal relationship between the drug and the complaint when the TTO is shorter. This renders the received information on AEFIs biased or absent and therefore hampers the request for additional information. As mentioned before, this may be less pronounced for AEFIs reported in prospective cohort studies because individuals are asked at multiple time points if any AEFIs have occurred. Based on the above, the TTO may be useful when addressing recall bias and, although this parameter has been studied extensively in the field of statistical signal detection [15,16,17,18,19,20,21], its possible relationship with recall bias has not been investigated previously.
The aim of this study was to investigate the TTO patterns of COVID-19-related AEFIs and explore potential differences in these patterns between spontaneous reports and reports derived from cohort studies. The two methods for vaccine safety monitoring present at the Dutch Pharmacovigilance Centre for COVID-19 vaccine monitoring, SRS and a large CEM study, offered the possibility to study difference in timing of AEFIs in both approaches. The TTO patterns were stratified for vaccination dose number, perceived burden of the AEFI, and for solicitation. Furthermore, we explored whether these differences could be explained by recall bias. The AEFIs related to the COVID-19 vaccination campaign were used, since a substantial number of spontaneous reports (>230,000 reports received since 2021) and inclusions in a prospective cohort study were available [6,22]. During the COVID-19 vaccination campaign, the number of reports of menstrual abnormalities increased rapidly [23]. An unprecedented number of reports were received on this association, with peak moments in reporting after media attention [24]. Thus, additionally, our secondary aim was to investigate the effect of media attention on TTO patterns for AEFIs related to menstrual disorders for SRS only.

2. Results

2.1. Descriptive Statistics

A total of 160,613 reports from the SRS and 19,979 from CEM, containing 755,647 and 103,703 AEFIs, respectively, were included into the analysis. There were 2862 (0.38% of all AEFIs from SRS) spontaneously reported AEFIs excluded from the analyses because the onset of the AEFI incorrectly preceded the moment of vaccination. Additional descriptive information is presented in Table 1.

2.2. TTO Analyses (All AEFIs)

Overall, no differences in the TTO pattern were observed between reports from the SRS and those from the CEM (see Figure 1). Furthermore, after stratification for burden and vaccination dose number, TTO patterns were similar for reports from the SRS and CEM. For non-solicited AEFIs, the was a difference in the median TTO (SRS: 3 days; CEM: 1 day; p < 0.001). However, this difference could not be attributed to a particular AEFI or group of AEFIs. Although no relevant differences in TTO patterns were found in general, we observed an increasing difference in reported TTOs between the CEM and SRS with an increase in the median TTO (see Figure 2). In particular, the reported median TTOs for a particular AEFI were, in general, lower for reports from the SRS compared to CEM. It should be noted, however, that these results mainly apply to menstrual disorders and the number of reports from the CEM were small for several AEFIs.

2.3. Effect of Media Attention (Menstrual Disorders Only)

Descriptive statistics of menstrual disorder reports included in media attention analysis are shown in Table 2. The comparison of spontaneously reported menstrual disorders showed distinct differences in the median TTO depending on the time of reporting in relation to the media attention. Figure 3A shows that reported median TTOs were significantly higher for AEFIs reported during media attention than for those reported before (9 vs. 3 days, respectively; p < 0.001). A similar significant difference was seen for the reporting delay with median values of 32 days (during) and 5 days (before), respectively (see Figure 3B). This led to the hypothesis that the TTO might somehow be related to the reporting delay, but no correlation between these two variables could be found (see Figure 3C).

3. Discussion

For vaccine pharmacovigilance, it is crucial that reliable methods are in place to ensure that safety issues can be detected in a timely manner. Aspects such as time to onset (TTO) and time to recovery (TTR) of AEFIs can be important factors in signal detection but also in the characterization of the AEFI profile of a vaccine [18,19,25]. In this light, it is also important that we have a good understanding of the potential biases in the methods that we use for vaccine safety surveillance. In this exploratory study, we investigated differences in TTO patterns of AEFIs for COVID-19 vaccines from SRS and CEM and the possible relationship the TTO has with recall bias. The results show that the TTO is relatively short (less than 2 days) for the majority of AEFIs, which comes as no surprise since this is the case for the majority of vaccine-related AEFIs (mainly AEFIs related to reactogenicity, such as injection site reactions and pyrexia) [26]. For these fast-occurring AEFIs, no differences in TTO pattern were observed between SRS and CEM. And, although no unambiguous conclusions can be drawn from these results, it is an indication that recall bias is not more pronounced in spontaneous reporting than it is in cohort studies, at least for AEFIs with a short TTO. The situation may be different for AEFIs with a longer TTO. With an increasing median TTO, we observed an increasing difference in TTO between SRS and CEM (where TTOs from SRS were, in general, shorter than those from CEM), suggesting that recall bias in SRS may be more substantial for these AEFIs. An explanation for this could be that, for SRS, reporters are less likely to assume a causal relationship between vaccination and an event as the TTO increases. As a result, AEFIs with a longer TTO are reported less frequently in SRSs than those with a shorter TTO, and the median TTO for that particular AEFI is lower compared to the situation where the ones with a longer TTO would have been reported. For CEMs, however, the questionnaires triggered the participants to evaluate their events and a possible causality, and AEFIs with a longer TTO are more likely to be reported, leading to higher median TTOs in our analysis. However, one could also debate that the triggering by the questionnaire in a CEM results in events with a longer TTO, not causally related to the vaccine, being misclassified as AEFIs where they should be considered adverse events, not reactions. Because of the set-up of the CEM study (pre-defined questions on common adverse events to all participants and multiple questionnaire moments), the rate of underreporting is likely to be low. However, not all reported events in the CEM study will be true adverse reactions. That is incorporated in the definition of an AEFI: an adverse event following immunization is any untoward medical occurrence that follows immunization and that does not necessarily have a causal relationship. An experienced AEFI can also be due to nocebo effect or background incidence [27,28]. A complicating factor in this analysis is that most AEFIs were related to menstrual disorders, which may have a cyclic pattern, making TTO analysis more challenging. Furthermore, as women may sometimes experience an irregular cycle, this may not immediately be a point of concern. Finally, the number of CEM reports for several of these AEFIs was relatively low. This hampers the generalizability of the results.
The effect of media attention on the reported TTOs is not quite clear from our study. It was difficult to exactly determine the time window for media attention, as social media played an important role and media outside the Netherlands also reported on this association [23,29]. We hypothesized a relationship with the reporting delay, which was higher for AEFIs reported during the window of media attention and lower prior to it. In itself, this difference in reporting delay could make sense when someone experiencing the complaints reports them when they occur in the period prior to media coverage (based on presumed causality), whereas someone who learns about the AEFI in the media sometime after it occurred (or discusses it with someone who learned about it that way) may report it in retrospect. Due to recall bias, this reporting delay could translate into a different TTO pattern, and, although a difference in median TTO among the different time windows was observed, a correlation between the two was clearly not present (see Figure 3C). As indicated above, this analysis was limited to menstrual complaints. These types of complaints are characterized by a cyclic nature and are, therefore, not representative of all AEFIs that may occur.
The major strength of this study is the large numbers of AEFI reports, both from SRS and CEM, allowing for a proper analysis of the TTO patterns. Furthermore, the reports were obtained in the same time window, limiting the effect of time-varying confounding. Finally, the extensive media attention on COVID-19-vaccine-related menstrual disorders allowed us to explore their effect on TTO patterns for spontaneously reported AEFIs. An important limitation is the use of COVID-19 vaccines only, and the question remains whether our findings can be extrapolated to other vaccines and/or drugs. Additionally, TTOs could not be compared for all types of AEFIs because not all AEFIs reported spontaneously were reported in sufficient amounts in the CEM studies. Also, the use of days as the unit for TTO for vaccines could be considered a rough estimate, given that the majority of AEFIs occur within two days, leading to limited variability in the data. Related to this, spontaneous reporting tends to cause TTO clustering around particular units of time (days/weeks), diminishing statistical precision [15]. For example, an injection site reaction that occurs 12 h after administration is likely to be reported having a TTO of 1 day. Our study aimed to investigate the relationship between reported TTOs for different reporting methods. Based on this aim, determining the causality (which includes biological plausibility) of individual AEFIs was not within the scope of this study. In assessments of the individual AEFI, biological plausibility is a key feature. At the Dutch Pharmacovigilance Centre Lareb, where this study was performed, causality assessment is a continuous process that is performed by trained clinicians. When a signal is suspected, a more detailed analysis is performed. Regarding the menstrual disorders, it is important to highlight that media attention is not the sole factor in this association. The causal relationship with COVID-19 vaccination was the subject of investigation of prior study [30,31].
The National Institute for Public Health and the Environment (RIVM) collects vaccination data on a national level in the COVID Vaccination Information and Monitoring System (CIMS) database. Based on data from CIMS, we estimated that the number of vaccinees in the ‘before’ group was approximately 1.3 million and in the ‘during’ group, approximately 11.3 million. It should be noted that persons that did not consent to share their data with the RIVM, i.e., around 6% of the Dutch vaccinated persons, are not included in CIMS [32,33]. Our centre previously found that the population in our CEM study was not entirely representative of the vaccinated population. For the Pfizer vaccine, the study population was predominated by men and an older age group compared with the other vaccines. This was largely due to the vaccination strategy in the Netherlands at the time the study started [6]. Finally, it should be emphasized that recall bias cannot be measured directly and the use of differences in TTO patterns as a proxy for recall bias is hypothetical. We therefore recommend additional research to confirm or disprove our findings.

4. Methods

In this study, we performed a retrospective analysis of reports of suspected AEFIs related to the COVID-19 vaccination campaign in the Netherlands. TTO patterns were compared between AEFIs reported spontaneously and those reported in a cohort monitoring event study and used as a possible proxy for recall bias. Differences in TTO patterns may provide information on AEFIs underrepresented in SRS, indicating possible recall bias. All statistical analyses and TTO plotting were performed using R statistics version 4.3.2.

4.1. Data Selection

Data from the SRS maintained by The Netherlands Pharmacovigilance Centre Lareb received from 6 January 2021 at the start of the vaccination campaign until 18 January 2022 and from a cohort event monitoring (CEM) study on the safety of COVID-19 vaccines in the Netherlands from February 2021 until March 2022 were used. Study design and inclusions of the CEM study has been described in detail elsewhere [5,6,7]. All AEFIs from reports having one of the four COVID-19 vaccines available in the Netherlands during the COVID-19 pandemic (Pfizer-BioNTech’s Comirnaty, Moderna’s SpikeVax, AstraZeneca’s Vaxzevria, Janssen’s Jcovden) as the suspect vaccine were included. In this study, we used a reporting form with predefined terms for the most commonly reported AEFIs (e.g., pyrexia, injection site reactions) that we considered to be solicited AEFIs for the scope of this study. If the time of onset of the reported reaction erroneously preceded the moment of vaccination, the ICSR was excluded. Because the CEM studies contained AEFIs related to the first and second doses of COVID-19 vaccine only, AEFIs for subsequent doses from the SRS were excluded. All AEFIs were coded using the preferred terms (PTs) from the Medical Dictionary for Regulatory Activities (MedDRA, version 26.1). A subset of PTs that was sufficiently present in both SRS and CEM was included for analysis (see also Supplementary Table S1). Common, listed AEFIs were solicited in the spontaneous reporting form and the CEM questionnaires: injection site reactions, fatigue, malaise, myalgia, headache, arthralgia, chills, nausea, and pyrexia. In both the SRS reporting form and the CEM questionnaires, vaccinated persons were asked to score the burden of the reported AEFI. The perceived burden was retrieved in a question with five answer options ranging from ‘not at all’, ‘slightly’, ‘somewhat’, ‘moderately’, to ‘extremely’ burdensome [7,34].

4.2. Variable Selection

The following variables were used for stratification of the results:
  • Solicited vs. unsolicited AEFIs. Solicited AEFIs were those specifically mentioned in the reporting forms and questionnaires and could be selected. Unsolicited reports were those that could be reported in open text-fields and subsequently coded (see Supplementary Table S1 for details).
  • Vaccination dose number (1 vs. 2). Since no information on subsequent vaccination dose numbers was available for the CEM studies, only these two were included from the spontaneous reports.
  • Perceived burden of the AEFI (high vs. low).
The perceived burden of the AEFI was based on the patient-reported burden.

4.3. TTO Analysis

The TTO was defined as the time window (in days) between the vaccination date and the date of the first occurrence of the AEFI. TTO patterns were inspected visually using empirical cumulative distribution plots. For accurate visual inspection of the TTO distributions, plots were limited to a TTO of 0–31 days. Differences in median TTO were tested two-sided using the Mann–Whitney U test at a significance level of α = 0.05. Additionally, for AEFIs with a TTO of more than 2 days, the difference in TTO between SRS and CEM was plotted to investigate a possible correlation between an increasing median TTO and increasing differences in TTO between SRS and CEM. The cut-off value of a minimum of 2 days was based on the fact that the vast majority of AEFIs (mainly injection site reactions) occur within the first 2 days, which may bias the results.

4.4. Effect of Media Attention, Menstrual Disorders as an Example

There was wide media coverage on menstrual disorders as a possible AEFI of COVID-19 vaccination in the Netherlands, both via traditional media such as newspapers and TV and via social media [24]. Details on categories of menstrual disorders can be found in Supplementary Table S2. The TTO patterns for this type of AEFI were compared for reports received prior to and during media coverage. The time period prior to 1 July 2021 was chosen as the ‘before’ period and the period from 1 July 2021 until 18 January 2022 was chosen as the ‘during’ period. Because media attention is not limited to a specified period, it was decided to classify the moment of first media attention until the end of data collection as the ‘during’ period.
To investigate the role of media attention in this setting, three analyses were performed. First, the cumulative probability distribution of the TTO for the time frames ‘before’ and ‘during’ media attention were analysed. Second, differences in reporting delay (defined as the time window between the start of the AEFI and the moment of reporting it to Lareb) were inspected because an increase in this delay could be correlated with recall bias. Finally, we examined the possible association between the reported TTO and the reporting delay to see if an increasing reporting delay was associated with increasing TTO values.
Differences in median TTO were tested two-sided using the Mann–Whitney U test at a significance level of α = 0.05. A temporal overview of received reports of menstrual disorders associated with COVID-19 vaccines can be found in Supplementary Figure S1.

5. Conclusions

For AEFIs with a short TTO, this study does not show significant changes in TTO between spontaneously reported AEFIs and those reported through CEM studies. We, therefore, hypothesize that recall bias in this subgroup of AEFIs is limited. For AEFIs with a longer TTO, this seems to be less evident and studies with larger datasets are needed. The effect of media attention on the TTO and possible recall bias could not be confirmed.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/pharma4020008/s1: Table S1: Overview of solicited and unsolicited ADRs in SRS and CEM based on MedDRA® Preferred Terms (PTs). Table S2: Categories of menstrual abnormalities. Figure S1: Temporal overview of reports of menstrual disorders associated with COVID-19 vaccines.

Author Contributions

Conceptualization, J.S., F.v.H., and E.v.P.; methodology, J.S., F.v.H., and E.v.P.; formal analysis, J.S.; writing—original draft preparation, J.S.; writing—review and editing, F.v.H. and E.v.P.; visualization, J.S.; supervision, F.v.H. and E.v.P. All authors have read and agreed to the published version of the manuscript.

Funding

The CEM study that was used as a data source for this research was funded by a grant from the Dutch Ministry of Health, Welfare and Sport. Grant number 331.880. The Dutch Ministry of Health, Welfare and Sport had no role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication. There was no additional financial support for this study.

Institutional Review Board Statement

Data from the Dutch COVID Cohort Event Monitoring Study (CEM) were used. If a study in the Netherlands is subject to the Medical Research Involving Human Subjects Act (WMO), it must undergo a review by an accredited Medical Research Ethics Committee or the central committee on research involving human subjects (CCMO). After submission to an accredited review committee (METC Brabant), this CEM study was deemed not to fall under the WMO act. No ethics approval was needed for this study.

Informed Consent Statement

For the CEM study, participants in the study provided a written statement of consent at the time of registration for participation and their data to be used for the purpose of this research and publication of study results. For reports from the Dutch Spontaneous Reporting System, patients gave consent for their data to be used in the reporting form.

Data Availability Statement

The data from the Dutch SRS and CEM study cannot be made fully publicly available due to the General Data Protection Regulation (GDPR) and the general privacy regulation of Pharmacovigilance Centre Lareb. Request to access the data may be granted on reasonable request via the first author.

Acknowledgments

We thank Thomas Lieber for his assistance in data extraction and preparation.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
ADRAdverse drug reaction
AEFIAdverse event following immunization
CEMCohort event monitoring
ICSRIndividual case safety reports
SRSSpontaneous reporting system
TTOTime to onset

References

  1. Lester, J.; Neyarapally, G.A.; Lipowski, E.; Graham, C.F.; Hall, M.; Dal, P.G. Evaluation of FDA safety-related drug label changes in 2010. Pharmacoepidemiol. Drug Saf. 2013, 22, 302–305. [Google Scholar] [CrossRef]
  2. Montane, E.; Santesmases, J. Characteristics of drug safety alerts issued by the Spanish Medicines Agency. Front. Pharmacol. 2023, 14, 1090707. [Google Scholar] [CrossRef]
  3. European Medicines Agency. Safety Signal. Available online: https://www.ema.europa.eu/en/glossary/safety-signal#:~:text=Information%20on%20a%20new%20or,studies%20and%20the%20scientific%20literature (accessed on 10 November 2023).
  4. Rudolph, A.; Mitchell, J.; Barrett, J.; Skold, H.; Taavola, H.; Erlanson, N.; Melgarejo-Gonzalez, C.; Yue, Q.Y. Global safety monitoring of COVID-19 vaccines: How pharmacovigilance rose to the challenge. Ther. Adv. Drug Saf. 2022, 13, 20420986221118972. [Google Scholar] [CrossRef]
  5. Rolfes, L.; Harmark, L.; Kant, A.; van Balveren, L.; Hilgersom, W.; van Hunsel, F. COVID-19 vaccine reactogenicity—A cohort event monitoring study in the Netherlands using patient reported outcomes. Vaccine 2022, 40, 970–976. [Google Scholar] [CrossRef] [PubMed]
  6. Kant, A.; Jansen, J.; van Balveren, L.; van Hunsel, F. Description of Frequencies of Reported Adverse Events Following Immunization Among Four Different COVID-19 Vaccine Brands. Drug Saf. 2022, 45, 319–331. [Google Scholar] [CrossRef]
  7. Duijster, J.W.; Lieber, T.; Pacelli, S.; Van Balveren, L.; Ruijs, L.S.; Raethke, M.; Kant, A.; Van Hunsel, F. Sex-disaggregated outcomes of adverse events after COVID-19 vaccination: A Dutch cohort study and review of the literature. Front. Immunol. 2023, 14, 1078736. [Google Scholar] [CrossRef]
  8. Ciccimarra, F.; Luxi, N.; Bellitto, C.; L’Abbate, L.; Raethke, M.; van Hunsel, F.; Lieber, T.; Mulder, E.; Riefolo, F.; Dureau-Pournin, C.; et al. Safety Monitoring of COVID-19 Vaccines in Persons with Prior SARS-CoV-2 Infection: A European Multi-Country Study. Vaccines 2024, 12, 241. [Google Scholar] [CrossRef]
  9. Luxi, N.; Ciccimarra, F.; Bellitto, C.; Raethke, M.; van Hunsel, F.; Lieber, T.; Mulder, E.; L’Abbate, L.; Marques, F.B.; Furci, F.; et al. Safety of COVID-19 Vaccines among People with History of Allergy: A European Active Surveillance Study. Vaccines 2024, 12, 1059. [Google Scholar] [CrossRef]
  10. Gee, J.; Shimabukuro, T.T.; Su, J.R.; Shay, D.; Ryan, M.; Basavaraju, S.V.; Broder, K.R.; Clark, M.; Buddy Creech, C.; Cunningham, F.; et al. Overview of U.S. COVID-19 vaccine safety surveillance systems. Vaccine 2024, 42 (Suppl. S3), 125748. [Google Scholar] [CrossRef]
  11. Hazell, L.; Shakir, S.A. Under-reporting of adverse drug reactions: A systematic review. Drug Saf. 2006, 29, 385–396. [Google Scholar] [CrossRef]
  12. Pariente, A.; Gregoire, F.; Fourrier-Reglat, A.; Haramburu, F.; Moore, N. Impact of safety alerts on measures of disproportionality in spontaneous reporting databases: The notoriety bias. Drug Saf. 2007, 30, 891–898. [Google Scholar] [CrossRef]
  13. de Boissieu, P.; Kanagaratnam, L.; Abou Taam, M.; Roux, M.P.; Drame, M.; Trenque, T. Notoriety bias in a database of spontaneous reports: The example of osteonecrosis of the jaw under bisphosphonate therapy in the French national pharmacovigilance database. Pharmacoepidemiol. Drug Saf. 2014, 23, 989–992. [Google Scholar] [CrossRef] [PubMed]
  14. Hartnell, N.R.; Wilson, J.P. Replication of the Weber effect using postmarketing adverse event reports voluntarily submitted to the United States Food and Drug Administration. Pharmacotherapy 2004, 24, 743–749. [Google Scholar] [CrossRef] [PubMed]
  15. Scholl, J.H.; Van Puijenbroek, E.P. The value of time-to-onset in statistical signal detection of adverse drug reactions: A comparison with disproportionality analysis in spontaneous reports from the Netherlands. Pharmacoepidemiol. Drug Saf. 2016, 25, 1361–1367. [Google Scholar] [CrossRef]
  16. Scholl, J.H.G.; van Hunsel, F.; Hak, E.; van Puijenbroek, E.P. Time to onset in statistical signal detection revisited: A follow-up study in long-term onset adverse drug reactions. Pharmacoepidemiol. Drug Saf. 2019, 28, 1283–1289. [Google Scholar] [CrossRef] [PubMed]
  17. van Holle, L.; Bauchau, V. Signal detection on spontaneous reports of adverse events following immunisation: A comparison of the performance of a disproportionality-based algorithm and a time-to-onset-based algorithm. Pharmacoepidemiol. Drug Saf. 2014, 23, 178–185. [Google Scholar] [CrossRef]
  18. van Holle, L.; Tavares Da Silva, F.; Bauchau, V. Signal detection based on time-to-onset: Extending a new method from spontaneous reports to observational studies. Pharmacoepidemiol. Drug Saf. 2014, 23, 849–858. [Google Scholar] [CrossRef]
  19. van Holle, L.; Zeinoun, Z.; Bauchau, V.; Verstraeten, T. Using time-to-onset for detecting safety signals in spontaneous reports of adverse events following immunization: A proof of concept study. Pharmacoepidemiol. Drug Saf. 2012, 21, 603–610. [Google Scholar] [CrossRef]
  20. Norén, G.N.; Hopstadius, J.; Bate, A.; Star, K.; Edwards, I.R. Temporal pattern discovery in longitudinal electronic patient records. Data Min. Knowl. Discov. 2010, 20, 361–387. [Google Scholar] [CrossRef]
  21. Cornelius, V.R.; Sauzet, O.; Evans, S.J. A signal detection method to detect adverse drug reactions using a parametric time-to-event model in simulated cohort data. Drug Saf. 2012, 35, 599–610. [Google Scholar] [CrossRef]
  22. Oosterhuis, I.; Scholl, J.; van Puijenbroek, E.; Kant, A.; van Hunsel, F. Optimizing Safety Surveillance for COVID-19 Vaccines at the National Pharmacovigilance Centre Lareb: One Year of COVID-19 Vaccine Experience. Drug Saf. 2023, 46, 65–75. [Google Scholar] [CrossRef]
  23. Gordillo-Maranon, M.; Szmigiel, A.; Yalmanova, V.; Caplanusi, I.; Genov, G.; Olsen, D.B.; Straus, S. COVID-19 Vaccines and Heavy Menstrual Bleeding: The Impact of Media Attention on Reporting to EudraVigilance. Drug Saf. 2024, 47, 783–798. [Google Scholar] [CrossRef]
  24. Duijster, J.W.; Schoep, M.E.; Nieboer, T.E.; Jajou, R.; Kant, A.; van Hunsel, F. Menstrual abnormalities after COVID-19 vaccination in the Netherlands: A description of spontaneous and longitudinal patient-reported data. Br. J. Clin. Pharmacol. 2023, 89, 3126–3138. [Google Scholar] [CrossRef] [PubMed]
  25. Raethke, M.; van Hunsel, F.; Luxi, N.; Lieber, T.; Bellitto, C.; Mulder, E.; Ciccimarra, F.; Riefolo, F.; Thurin, N.H.; Roy, D.; et al. Frequency and timing of adverse reactions to COVID-19 vaccines; A multi-country cohort event monitoring study. Vaccine 2024, 42, 2357–2369. [Google Scholar] [CrossRef] [PubMed]
  26. Herve, C.; Laupeze, B.; Del Giudice, G.; Didierlaurent, A.M.; Tavares Da Silva, F. The how’s and what’s of vaccine reactogenicity. NPJ Vaccines 2019, 4, 39. [Google Scholar] [CrossRef]
  27. Kant, A.; van Hunsel, F. Authors’ Reply to Mungmunpuntipantip et al.’s Comment on “Description of Frequencies of Reported Adverse Events Following Immunization Among Four Different COVID-19 Vaccine Brands”. Drug Saf. 2022, 45, 925–926. [Google Scholar] [CrossRef] [PubMed]
  28. Mungmunpuntipantip, R.; Wiwanitkit, V. Comment on “Description of Frequencies of Reported Adverse Events Following Immunization Among Four Different COVID-19 Vaccine Brands”. Drug Saf. 2022, 45, 923. [Google Scholar] [CrossRef]
  29. Male, V. Menstrual changes after COVID-19 vaccination. BMJ 2021, 374, n2211. [Google Scholar] [CrossRef]
  30. Jajou, R.; Lieber, T.; van Puijenbroek, E.P.; Mulder, E.; Overbeek, J.; Hek, K.; van Hunsel, F.; Kant, A. GP consultations for menstrual disorders after COVID-19 vaccination—A self-controlled cohort study based on routine healthcare data from the Netherlands. Vaccine 2024, 42, 126130. [Google Scholar] [CrossRef]
  31. Smaardijk, V.R.; Jajou, R.; Kant, A.; van Hunsel, F.P.A.M. Menstrual disorders following COVID-19 vaccination: A review using a systematic search. Front. Drug Saf. Regul. 2024, 4, 1338466. [Google Scholar] [CrossRef]
  32. National Institute for Public Health and the Environment (RIVM). COVID-19. Available online: https://www.rivm.nl/en/coronavirus-covid-19 (accessed on 24 March 2025).
  33. Huisman, C. Percentage Gevaccineerden Dat Data Wil Delen Met RIVM Daalt Onder de Kritische Grens van 95 Procent. Available online: https://www.volkskrant.nl/nieuws-achtergrond/percentage-gevaccineerden-dat-data-wil-delen-met-rivm-daaltonder-de-kritische-grens-van-95-procentbbe45557/ (accessed on 24 March 2024).
  34. Rolfes, L.; Haaksman, M.; van Hunsel, F.; van Puijenbroek, E. Insight into the Severity of Adverse Drug Reactions as Experienced by Patients. Drug Saf. 2020, 43, 291–293. [Google Scholar] [CrossRef]
Pharmacoepidemiology 04 00008 g001
Figure 1. Empirical cumulative distribution plot of the TTO for (A) all AEFIs, (B) solicited AEFIs, and (C) unsolicited AEFIs.
Figure 1. Empirical cumulative distribution plot of the TTO for (A) all AEFIs, (B) solicited AEFIs, and (C) unsolicited AEFIs.
Pharmacoepidemiology 04 00008 g001
Pharmacoepidemiology 04 00008 g002
Figure 2. Difference in median TTO between CEM and SRS. The x-axis displays the median TTO for AEFIs from CEMs, whereas the y-axis displays the difference in the median TTO between CEM and SRS (calculated as median TTO SRS–median TTO CEM). Numbers in brackets represent the number of reports for the particular AEFI. The grey area represents the 95% confidence interval of the regression line. CEM: cohort event monitoring; SRS: spontaneous reporting system.
Figure 2. Difference in median TTO between CEM and SRS. The x-axis displays the median TTO for AEFIs from CEMs, whereas the y-axis displays the difference in the median TTO between CEM and SRS (calculated as median TTO SRS–median TTO CEM). Numbers in brackets represent the number of reports for the particular AEFI. The grey area represents the 95% confidence interval of the regression line. CEM: cohort event monitoring; SRS: spontaneous reporting system.
Pharmacoepidemiology 04 00008 g002
Pharmacoepidemiology 04 00008 g003
Figure 3. Relationship between the TTO and reporting delay for spontaneously reported menstrual disorders in and its relationship with media attention. (A) Empirical cumulative distribution plot of the TTO for different time frames. (B) Boxplots comparing the reporting delay for different time frames. (C) Scatterplot for the TTO vs. the reporting delay for different time frames.
Figure 3. Relationship between the TTO and reporting delay for spontaneously reported menstrual disorders in and its relationship with media attention. (A) Empirical cumulative distribution plot of the TTO for different time frames. (B) Boxplots comparing the reporting delay for different time frames. (C) Scatterplot for the TTO vs. the reporting delay for different time frames.
Pharmacoepidemiology 04 00008 g003
Table 1. Descriptive statistics of reports included in the study.
Table 1. Descriptive statistics of reports included in the study.
SRSCEM
Number of reports160,61326,730 *
       Vaccination dose number 1103,127 (64.2%)18,177 (68.0%)
Number of AEFIs755,647103,703
       Solicited696,965 (92.2%)101,466 (97.8%)
       High burden202,399 (26.8%)13,910 (13.4%)
Median TTO in days (IQR)1 (0.1–1)1 (0–1)
* The reports were received from 19,979 participants. Due to the longitudinal character of the CEM study, more reports per participants are possible. SRS, spontaneous reporting system; CEM, cohort event monitoring.
Table 2. Descriptive statistics of menstrual disorder reports included in media attention analysis.
Table 2. Descriptive statistics of menstrual disorder reports included in media attention analysis.
SRS
Number of reports22,296
Number of AEFIs30,016
       Before media attention702 (2.4%)
       During media attention29,314 (97.6%)
SRS, spontaneous reporting system.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Scholl, J.; van Hunsel, F.; van Puijenbroek, E. Timing Matters: Exploring the Role of the Time to Onset in Recall Bias for Adverse Events Following Immunization (AEFIs) of COVID-19 Vaccines from Spontaneous Reports. Pharmacoepidemiology 2025, 4, 8. https://doi.org/10.3390/pharma4020008

AMA Style

Scholl J, van Hunsel F, van Puijenbroek E. Timing Matters: Exploring the Role of the Time to Onset in Recall Bias for Adverse Events Following Immunization (AEFIs) of COVID-19 Vaccines from Spontaneous Reports. Pharmacoepidemiology. 2025; 4(2):8. https://doi.org/10.3390/pharma4020008

Chicago/Turabian Style

Scholl, Joep, Florence van Hunsel, and Eugene van Puijenbroek. 2025. "Timing Matters: Exploring the Role of the Time to Onset in Recall Bias for Adverse Events Following Immunization (AEFIs) of COVID-19 Vaccines from Spontaneous Reports" Pharmacoepidemiology 4, no. 2: 8. https://doi.org/10.3390/pharma4020008

APA Style

Scholl, J., van Hunsel, F., & van Puijenbroek, E. (2025). Timing Matters: Exploring the Role of the Time to Onset in Recall Bias for Adverse Events Following Immunization (AEFIs) of COVID-19 Vaccines from Spontaneous Reports. Pharmacoepidemiology, 4(2), 8. https://doi.org/10.3390/pharma4020008

Article Metrics

Back to TopTop