IRVINE: An Interactive Visualization for Spontaneous Reporting Systems Databases Missing Values

Abstract

Large-scale post-marketing drug safety data from spontaneous reporting systems offer new opportunities to explore adverse drug events (ADEs). However, these datasets often contain high rates of missing and incomplete data, undermining the reliability and interpretability of pharmacovigilance analyses. Effective management of these data quality issues requires interactive tools to explore patterns of missingness across multiple dimensions. We present IRVINE (Interactive Visualization for Spontaneous Reporting Systems Databases Missing Values), an interactive visualization system designed to explore and compare missing data in spontaneous reporting systems. IRVINE integrates multiple coordinated components—including a global overview, detailed attribute-level breakdowns, a temporal analysis interface, and a cross-database comparison environment—allowing users to fluidly transition between global summaries and fine-grained diagnostic views. The system supports dynamic filtering, drill-down exploration, and interactive temporal analysis to examine changes in data completeness over time and across categories. Through three usage scenarios and a user study, we demonstrate how IRVINE supports effective exploration of reporting completeness. Results indicate that users perceived the system as easy to use and effective for identifying missingness patterns, with particular strengths in comparative and detail-level analysis. This work lays a foundation for improved transparency, interpretability, and data quality assessment in large-scale pharmacovigilance systems.

1. Introduction

Adverse drug events (ADEs) are common, preventable causes of mortality, hospitalizations, and financial burden [1]. The global rate of ADE-related mortality increased 3.3-fold between 2001 and 2019, indicating a rising burden of ADEs [2]. Although drugs undergo rigorous safety evaluations in clinical trials before approval, unanticipated adverse events often emerge once medications are widely used, underscoring the importance of post-marketing surveillance [3].

Spontaneous reporting systems (SRS) are vital for post-marketing surveillance and form a cornerstone of pharmacovigilance worldwide [4]. Among the most prominent SRS databases are the U.S. Food and Drug Administration’s Adverse Event Reporting System (FAERS) and Health Canada’s Canada Vigilance database. FAERS is one of the largest global repositories, widely used for detecting early safety signals and informing regulatory decisions [5]. Canada Vigilance serves as the national counterpart, capturing adverse event reports from Canadian healthcare professionals, consumers, and manufacturers. Together, these databases provide complementary perspectives on drug safety, enabling comparative analyses across healthcare systems and populations.

The FAERS database is a publicly accessible resource supporting pharmacovigilance by capturing adverse event reports from healthcare professionals and consumers [6]. Reports in FAERS span diverse patient demographics and drug types, providing valuable insights into drug safety [7]. Its regular updates and detailed event-level data make FAERS a powerful resource for detecting safety signals, studying drug effects, and guiding regulatory decisions [8,9]. Similarly, the Canada Vigilance database plays a parallel role in the Canadian context, collecting adverse event reports from healthcare professionals, consumers, and manufacturers [10,11] and offering a complementary perspective to FAERS. Together, these large-scale SRS databases enable cross-national comparisons and strengthen global pharmacovigilance efforts [12,13].

Despite their value, both databases suffer from high rates of missing data, which can limit the accuracy and utility of analyses [6,7,9,14]. Key fields—such as event date, drug indication, route of administration, and patient weight—are often left blank, particularly in reports submitted by consumers [6,14]. Inconsistent reporting hampers efforts to assess causality and compare reports across different reporter types. Furthermore, variability in report completeness introduces significant challenges for tasks such as data mining, signal detection, and cross-study comparisons [8,14].

Interactive visualization offers a promising approach to address these challenges. This field transforms complex datasets into visual representations that users can directly explore and manipulate [15,16,17]. By providing intuitive interfaces, interactive visualization leverages human perceptual and cognitive strengths to reveal patterns, trends, and anomalies that may be hard to detect in raw data [16,17]. Exploratory visualization emphasizes user-driven engagement with data, allowing insights to emerge through direct manipulation and visual exploration. It supports open-ended investigation, enabling researchers to frame and refine questions dynamically, compare subsets, and generate new hypotheses [15,18,19].

Previous studies have visualized missing data to summarize patterns of incompleteness across datasets [20,21,22,23]. However, these are typically static visualizations that offer only a fixed snapshot of the data, without support for interactive exploration or user-driven filtering. Consequently, they fall short of enabling deeper analysis—such as drilling into specific subsets (e.g., by country, age, or seriousness of outcome), comparing missingness across groups, or dynamically adjusting views based on analytical needs. These limitations hinder analysts’ ability to contextualize and reason about missing data in complex, multi-dimensional datasets, where missingness can vary widely across demographics, time periods, and report sources.

To address this gap, we developed IRVINE (Interactive Visualization for Spontaneous Reporting Systems Databases Missing Values), an interactive system for exploring missing data in both the FAERS and Canada Vigilance databases. IRVINE enables dynamic filtering, comparison, and drill-down across these two large reporting systems, allowing a flexible and nuanced understanding of missing data patterns within and across national pharmacovigilance datasets.

In contrast to the majority of existing pharmacovigilance tools that predominantly emphasize signal detection [24,25], our approach shifts the analytical focus toward data quality by enabling in-depth exploration of report completeness. This perspective is crucial yet underrepresented in current pharmacovigilance tools, as incomplete spontaneous reports are known to hinder reliable signal interpretation [25,26,27]. By operationalizing the evaluation of missing data patterns across large pharmacovigilance databases, IRVINE treats data completeness as a first-class concern in post-market surveillance rather than a minor technical nuisance. This approach is novel in that it repurposes established visualization techniques into an interactive workflow centered on assessing report quality as a precursor to robust signal detection, a focus largely overlooked by standard pharmacovigilance dashboards and signal-detection tools [27].

While existing visualization techniques for missing data, such as UpSet plots, Missingno, or glyph-based visualizations [28,29,30,31], are highly effective at identifying row-level missingness, they are not designed to navigate the specific hierarchical and temporal complexities of pharmacovigilance data. Spontaneous reporting systems data is commonly multi-entity and nested at the report level (for example, a record can include multiple drugs, multiple reactions, and additional contextual fields) so auditing completeness requires field-level interpretation and structured aggregation rather than row-wise missingness inspection [14,32]. IRVINE closes this gap by turning missingness from a static, row-level diagnosis into a coordinated completeness-auditing workflow that supports overview-first exploration with interactive drill-down and aggregation by reporting context and time, and cross-database comparison to attribute and explain completeness differences rather than merely listing missing values.

The remainder of this paper is organized as follows. Section 2 describes the data sources, preprocessing procedures, and design goals that guided the development of IRVINE. Section 3 presents the system’s design, visualization components, and interaction mechanisms in detail. Section 4 illustrates the analytical capabilities of IRVINE through representative usage scenarios. Section 5 reports the results of the user evaluation, including quantitative and qualitative findings. Finally, Section 6 discusses the study’s limitations and summarizes the main contributions and future directions of this work.

2. Materials and Methods

This section describes the methodology employed to design the proposed interactive visualization system, namely IRVINE.

2.1. Defining Missingness in IRVINE

In this work, we define missingness as the absence of a reported value for a given structured field within an individual adverse event report. Missingness is therefore operationalized at the field level and quantified as counts or proportions of reports in which a specific attribute (e.g., age, dosage, route of administration) is not provided. This definition differs from alternative notions of missingness that focus on the absence of specific values or value ranges within an otherwise populated field (e.g., selectively missing extreme ages or dosages).

By adopting a field-level definition, our analysis emphasizes reporting completeness rather than value-dependent omission mechanisms. This choice aligns with the primary goals of pharmacovigilance data auditing, where identifying patterns of reporting incompleteness across patient characteristics and reporting contexts is essential for flagging potential data quality issues and motivating further investigation into their underlying causes.

While IRVINE visualizes missingness across patient characteristics, these attributes are interpreted as contextual indicators rather than direct sources of missingness. Because adverse event reports are completed by intermediaries (such as healthcare professionals, consumers, or lawyers) rather than patients themselves.

2.2. Data Source

IRVINE was developed using data from two major spontaneous reporting system databases: the U.S. FDA’s Adverse Event Reporting System (FAERS, accessed via openFDA [32]) and Health Canada’s Canada Vigilance database. Both databases capture adverse drug event (ADE) reports from healthcare professionals, consumers, and manufacturers, and are widely used in pharmacovigilance research to detect safety signals and monitor post-marketing outcomes.

Due to resource constraints, we limited our study to reports involving direct oral anticoagulants (DOACs) submitted between 2009 and 2025. This timeframe ensures consistency across both systems and captures recent reporting trends. From these reports, we extracted structured variables related to patient demographics (e.g., age, sex), event characteristics (e.g., seriousness, report type, date), and drug-related information (e.g., medicinal product, dosage, route of administration, treatment duration, indication). These variables represent core attributes frequently analyzed in pharmacovigilance studies and are also prone to missing or incomplete reporting [33,34,35].

2.3. Data Preparation

We applied several preprocessing steps to harmonize and prepare the FAERS and Canada Vigilance datasets for integration into IRVINE.

First, data cleaning was performed on demographic variables. Patient age values greater than 110 years or less than zero were recorded as missing, as such values were deemed implausible. Report dates were reformatted into a standardized structure across both datasets to ensure temporal consistency.

Second, we normalized categorical variables for consistent encoding. For example, seriousness categories and gender fields were harmonized between FAERS and Canada Vigilance to enable direct comparability.

Third, we transformed the datasets into a flat representation in which each patient corresponds to a single record. Drug-related features (e.g., medicinal product, dosage, route of administration, treatment duration, indication) often have multiple entries per patient; we retained all entries by concatenating values with a “|” delimiter. This preserves all reported information while maintaining a unified structure for visualization and analysis. We used the pipeline developed by Chalabianloo et al. for this step [36].

2.4. System Architecture

IRVINE uses a modular architecture consisting of a backend data processing pipeline and a client-side visualization interface. The backend preprocesses and aggregates data from FAERS and Canada Vigilance using Python (vr 3.9.7) scripts and serves the processed data via a FastAPI service. The frontend is implemented in JavaScript (vr 1.5) with D3.js (vr 7.9.0), enabling interactive visual exploration.

This design ensures that IRVINE is scalable to additional datasets, maintainable as reporting structures evolve, and responsive to user interactions. It provides a flexible platform for exploring missing data patterns in pharmacovigilance.

2.5. Design Goals

IRVINE’s design was driven by key challenges in exploring missing data in pharmacovigilance. We translated these challenges into task-oriented design goals that specify what users should be able to accomplish with the system:

Challenge 1. Heterogeneous missingness across multiple dimensions:

Missing data is not evenly distributed; it varies widely across demographics, drug-related attributes, and reporting contexts. Analysts need to move fluidly between high-level summaries and detailed breakdowns to understand these patterns [37,38,39].

•

Goal 1 (Multi-level exploration): Enable users to explore missingness at multiple levels, from overall summaries down to detailed attribute- and subset-level breakdowns, supporting both broad and fine-grained analysis.

Challenge 2. Cross-national and cross-database differences in reporting practices:

FAERS and Canada Vigilance differ in structure, scope, and reporting culture, making it difficult to determine whether missingness patterns are consistent or database-specific [14,40].

•

Goal 2 (Cross-database comparability): Allow users to compare missingness patterns across databases, countries, and reporter types, highlighting both similarities and discrepancies.

Challenge 3. Complexity and opacity of missing data:

Large pharmacovigilance datasets contain complex, multi-valued fields that are hard to interpret without appropriate encoding. Static summaries can obscure subtle differences in missingness across categories [39,41,42].

•

Goal 3 (Interpretability and usability): Provide interpretable visual encodings, symbolic markers, and responsive interactions to make missing data patterns accessible, transparent, and meaningful to diverse users.

Accordingly, IRVINE is designed to support exploratory diagnosis of reporting completeness patterns at the level of reporting practices, highlighting how missingness varies across patient characteristics, reporter types, and regulatory contexts, rather than modeling value-dependent missing data mechanisms.

3. The Design of IRVINE

IRVINE provides a clear and flexible way to explore missing data in pharmacovigilance systems. Its design emphasizes usability, interpretability, and multi-level exploration across demographics, drug attributes, and reporting dimensions in FAERS and Canada Vigilance. A demonstration of IRVINE and its modules is provided in Supplementary Video S1.

3.1. Visualization Components

Guided by our design goals, IRVINE is structured around two primary interface configurations: the Single-Database Analysis View, used for exploring individual repositories, and the Comparison Module, used for cross-database exploration. Figure 1 illustrates the system’s navigation flow, showing how users progress from the landing page to these analysis environments. The OpenFDA module includes a choropleth map for selecting and filtering countries before entering its analysis page. The Canada Vigilance module links directly to its analysis page (all reports originate from Canada). The Comparison module provides an integrated environment for cross-database exploration of missingness patterns.

3.1.1. Single-Database Analysis Module

This module serves as the core analytical interface for both the OpenFDA (FAERS) and Canada Vigilance modules. While the underlying data source differs, the visual encodings and interaction mechanisms remain consistent to ensure a uniform user experience across different pharmacovigilance repositories.

Geographic Entry Layer

For datasets spanning multiple nations, such as FAERS, the system includes an initial entry layer using a choropleth world map to provide a global overview of reporting (Figure 2). For single-country datasets like Canada Vigilance, the system bypasses this layer and proceeds directly to analysis.

Users can switch between two modes on the map: (1) a missing percentage ranking mode, where darker colors indicate higher proportions of missing data; and (2) a data contribution mode, where darker colors reflect higher report volumes. To complement the map, an ordered list on the left side of the interface presents the same information in rank order. The map and list are synchronized: selecting a country on the map highlights the corresponding list item, and vice versa. This coordinated interaction lets users explore geographic variation in reporting quality and coverage while maintaining both spatial and ranked perspectives. Figure 3 shows the entry layer in both modes.

The choropleth world map leverages the user’s spatial cognition to function as an intuitive index, reducing the cognitive load required to locate regions of interest compared to textual lists [43,44]. While less precise than bar charts for value estimation, choropleth maps are highly effective for synoptic tasks, such as detecting regional clusters and assessing relative disparities in data quality across a global scale [45]. This allows analysts to rapidly screen the dataset, identifying countries with high missingness or low reporting volumes before initiating a deeper investigation into specific attribute patterns.

Analysis Layer

After country selection in the entry layer, users proceed to the analysis layer. This environment integrates multiple coordinated visualizations for exploring missingness across demographic, drug-related, and temporal dimensions. Figure 4 provides an overview of the analysis layer, organized into five main components: a country filter panel, a main radial overview, a detail view, a control module, and a temporal trends view.

Main View (Radial Overview of Missingness)

The central radial visualization provides an integrated overview of missingness across patient- and drug-related attributes. At the center, the total percentage of missing data is displayed as the anchor of the view. Surrounding the center, radial bars encode missingness in patient demographics (age, sex, weight, seriousness). Beyond these bars, arcs represent drug-related features such as indication, route of administration, dosage, treatment duration, generic name, and brand name.

Within each arc, a circular marker is drawn for every 10% of missing data. This design offers a quick visual impression of the scale of missingness while still retaining exact proportional encodings. Two additional rings—Source Qualification and Report Type—can be toggled on or off to show contextual differences in reporting. Figure 5 illustrates the main radial visualization and its structure in detail.

The radial overview was chosen to provide a compact, high-density summary of missingness across all attributes, aligning with the “Overview first, zoom and filter, then details-on-demand” principle of exploratory visualization [46,47,48]. By leveraging the inherent compactness of a circle, a radial layout distributes data in every direction to fill the entire canvas, efficiently accommodating high-dimensional datasets [46].

This design creates a holistic visual fingerprint of the dataset’s quality [47,48] while maintaining continuous focus and context [49]. The radial layout allows users to obtain an immediate overview of the entire data structure even when zooming into the details of a specific attribute, without losing sight of the overall context. While linear bar charts offer the most precise length comparisons [50,51], the radial layout promotes comprehension and enables rapid pattern recognition across many variables simultaneously [52,53], thus revealing overall patterns without requiring disruptive navigation. Furthermore, both radial and treemap layouts are widely used for visualizing hierarchical structures [54]. we selected a radial layout due to its advantages in facilitating global pattern recognition and spatial continuity, which align with our focus on overall structural clarity, whereas the treemap layout’s strengths in proportional representation and nested containment [55] were less pertinent to our objectives. Thus, our choice of a radial layout is grounded in its proven advantages of space efficiency, context continuity, and immediate overview for high-dimensional data.

This visualization is fully interactive. Clicking on any radial bar, arc, or the central total initiates the detail view, which provides a granular breakdown of the selected attribute or category. This design allows users to seamlessly transition between overview and detail while maintaining a clear separation between demographic and drug-related dimensions.

Detail View

The detail view provides a multidimensional breakdown of missingness for the selected attribute (Figure 6). It displays distributions across key dimensions, including age, gender, seriousness of outcome, report type, and source qualification. For age, the distribution can be viewed either as a continuous variable or grouped into discrete age categories, allowing users to choose the level of granularity. By combining demographic and reporting-context perspectives, these coordinated visualizations offer a comprehensive understanding of missingness patterns for the chosen attribute. Users can also add any of these visualizations to a comparison cart for side-by-side analysis of multiple attributes or categories.

This view is designed to support details-on-demand exploration [47,56] following the identification of patterns in the overview. Once an attribute is selected, presenting multiple coordinated distributions allows users to examine missingness in relation to key patient characteristics and reporting contexts simultaneously. This multivariate breakdown supports analytical reasoning by enabling users to assess whether observed incompleteness is concentrated within specific subpopulations or reporting categories, rather than relying on a single aggregated measure [57,58]. Together, this design aligns with established principles of drill-down interaction and coordinated multiple views, enabling users to contextualize missingness patterns without losing analytical continuity.

Country Filter Panel

The country filter panel appears only for datasets receiving reports from multiple nations and lists all the selected countries (from the entry layer) alongside each country’s percentage of missing data. Users can select one or multiple countries, and their selection immediately updates all other panels in the analysis view. This panel serves as both a filter and a navigation tool, allowing analysts to align geographic subsets with missingness patterns. Combined with the comparison functionality in the detail view, the country filter panel supports comparing detail visualizations across different geographic contexts.

Control Module

The control module provides interactive controls to filter the detail view and focus on specific attributes or categories. Targeted filtering helps streamline exploration by reducing visual clutter and highlighting relevant data subsets. The control module also includes a button to open a compare sidebar, which reveals the comparison cart. From the sidebar, users can manage previously added visualizations and perform side-by-side comparisons across different attributes, categories, or country selections, further extending the system’s analytical flexibility.

Temporal View

The temporal view shows missingness trends over time (Figure 7). Users can select a date range (start and end year) and apply a threshold filter to highlight periods where the change in missingness exceeds a specified percentage. This draws attention to the most substantial shifts in reporting completeness. The visualization displays year-to-year changes in missing data percentages, allowing analysts to identify periods of improvement or deterioration in data quality. Clicking on a specific year expands the view to show month-by-month changes for that year, supporting a detailed examination of when changes occurred. Hovering over a monthly data point provides a tooltip with additional information, such as the number of reports and the exact missingness percentage at that time. These interactions provide both a historical overview and a way to drill down into finer temporal patterns, helping analysts contextualize how reporting practices have improved or declined over time.

We employed a bar-based temporal timeline, rather than a continuous line chart, to emphasize the discrete nature of annual reporting intervals. Bar encodings make individual year-to-year changes explicit and avoid implying continuity between observations, which is appropriate when data are aggregated at fixed temporal units [59,60,61]. A diverging color scheme is applied to reinforce directionality [62,63], with decreases in missingness encoded as improvement (green) and increases as deterioration (red). This encoding leverages preattentive processing, allowing analysts to identify significant shifts in reporting quality at a glance without prior serial scanning of the numerical values [64]. While red and green were selected for their high semantic resonance, we utilized distinct positional encoding (increase in missingness going up and decrease going down) to ensure the trends remain distinguishable for users with color vision deficiency. Together, the discrete bar representation and directional color cues facilitate rapid identification of periods of interest and focused investigation of changes in data quality over time.

To aid interpretation of temporal patterns, IRVINE includes an interactive “magic lens” in the temporal view. When activated, this lens overlays detailed information about the composition of changes in missingness by reporter type. For a selected year or month, the lens displays a treemap showing each reporter category’s proportional contribution to the change, along with the corresponding percentage point differences.

In yearly mode, the lens highlights which reporter type had the highest and lowest missing data percentages over the entire period. In monthly mode, these statistics update to reflect the selected year. This capability allows users to move beyond overall completeness trends and pinpoint which reporter groups are driving fluctuations in data quality over time.

The magic lens functions as a mediator of focus+context [47,65,66,67], allowing for a seamless transition between different levels of informational granularity [68,69,70]. By embedding a treemap encoding as an on-demand overlay, the tool supports analytical reasoning through a drill-down interaction that does not disrupt the user’s mental flow [71]. This design ensures that the attribution analysis is not a fragmented task, but rather a continuous extension of the user’s inquiry into the drivers of temporal change.

3.1.2. Comparison Module

The Comparison module extends IRVINE’s capabilities by enabling direct cross-database exploration of missingness patterns between OpenFDA and Canada Vigilance. This environment supports both structural and temporal comparisons, allowing analysts to investigate differences in reporting completeness between these two major SRS databases.

At the structural level, the interface displays parallel visualizations of OpenFDA and Canada Vigilance side by side (Figure 8). Radial overviews summarize missingness across demographic and drug-related attributes in each system, and coordinated detail views show distributions by age, seriousness of outcome, gender, report type, and source qualification. This side-by-side arrangement lets users align features of interest and observe differences in data completeness between the two systems.

The Comparison module also includes temporal analysis, enabling users to track changes in missingness in each database over time (Figure 9). Viewing patterns in a unified temporal frame helps identify periods when reporting practices in the two systems diverged or converged.

The magic lens feature is extended to the comparison module’s temporal view as well. When enabled, it generates parallel treemaps for OpenFDA and Canada Vigilance, allowing side-by-side comparison of how each reporter type contributes to changes in missingness over the same period. This comparative lens helps analysts determine whether changes in data quality stem from specific reporter groups in one system or reflect broader trends across both systems. By coupling this detailed breakdown with interactive filtering, the feature improves cross-database interpretability and supports evidence-based reasoning about the sources of reporting variability.

The comparison module adopts a side-by-side (juxtaposed) design to support cross-database analysis while preserving the internal structure of each reporting system. Juxtaposition is particularly well suited to analytical contexts where understanding within-dataset distributions is as important as identifying differences, as it allows analysts to inspect each dataset in its entirety rather than reducing comparison to aggregate differences [72,73]. This is particularly important in pharmacovigilance, where patterns of missingness must be interpreted in relation to reporter type, attribute category, and reporting context. By presenting each dataset in a separate panel using identical visual encodings, the system preserves the contextual integrity of each reporting source while avoiding the visual confusion that could arise from superposition [74]. Placing the views adjacent within the same visual field also enables analysts to quickly shift their attention between panels to identify similarities and differences. We acknowledge that juxtaposed layouts require viewers to mentally align corresponding elements across panels, a known trade-off of this design approach [72]. However, the cognitive effort of such mental alignment is offset by the benefits of clarity and separation of elements in side-by-side layouts. However, this cognitive effort is offset by the increased clarity and separation of elements afforded by side-by-side representations. To further reduce the perceptual effort required for alignment, interactive linking highlights matched attributes across views on demand, facilitating coordinated comparison [57,58,75]. Together, this design balances contextual integrity with comparative clarity, supporting nuanced interpretation of similarities and differences between reporting systems.

Together, these features create a comprehensive comparative environment, allowing analysts to explore both structural and temporal dimensions of missingness across OpenFDA and Canada Vigilance. This module shifts the analytical focus from single-system exploration to direct cross-system comparison, offering new opportunities to assess how differences in regulatory frameworks, reporting mechanisms, and healthcare contexts influence data completeness.

3.2. Interaction Mechanisms

IRVINE incorporates various interaction mechanisms that let users flexibly explore missingness patterns in FAERS and Canada Vigilance. These mechanisms ensure seamless transitions between overview and detail views, support comparative analysis, and provide contextual information for interpreting missing data.

3.2.1. Selection and Drill-Down

The radial overview serves as the primary entry point for interaction. By clicking on any radial bar, arc, or the central element, users can drill down into the detail view where missingness is broken down by age, gender, seriousness, source qualification, and report type. This iterative drill-down allows analysts to move between global summaries and specific subsets of interest.

3.2.2. Tooltips and Contextual Information

Hover interactions provide dynamic tooltips throughout the interface. In the temporal view, hovering over a year or month bar reveals details such as the number of reports and the missingness percentage for that period. In the detail view, tooltips show the exact proportions for each demographic or category breakdown, allowing users to obtain precise values without disrupting the visual flow.

3.2.3. Filtering and Reconfiguration

The control module enables users to filter which attributes are displayed in the detail view, focusing on the categories most relevant to their analysis. In the temporal view, a threshold slider highlights time periods with substantial changes in completeness. Users can also switch between continuous and binned age displays, reconfiguring the demographic representation to suit their needs.

Together, these interaction mechanisms support flexible, iterative exploration of missing data while maintaining coherence between views. They enable both within-database analyses and cross-database comparisons without loss of context.

4. Usage Scenarios

To demonstrate IRVINE’s utility, we present three usage scenarios highlighting how it supports different pharmacovigilance stakeholders. These scenarios illustrate how researchers, regulators, and healthcare analysts can use the system to identify, contextualize, and compare missingness patterns in spontaneous reporting systems.

4.1. Research Feasibility Assessment

A data scientist is considering developing a machine learning model to predict adverse drug events using reports from FAERS and Canada Vigilance. Before investing in model development, they use IRVINE to assess the feasibility of this endeavor. Specifically, they consider the following questions:

Q1. Which features in FAERS and Canada Vigilance are most affected by missingness, and are these features critical for the model?

Q2. How does missingness vary across different subpopulations (e.g., by age group)?

Q3. Are there systematic differences in reporting completeness between FAERS and Canada Vigilance that would affect combining the datasets for modeling?

Q4. For which subsets of the data might the data quality be sufficient to train reliable models?

Using IRVINE, the scientist begins by exploring the OpenFDA module. The radial overview reveals substantial missingness in fields like treatment duration, patient age, patient weight, and treatment start/end dates. In contrast, fields such as dosage, generic name, indication, and sex are relatively complete. By drilling into the detail view, the scientist notes that missing values occur more frequently in reports for older adults (Figure 10).

Switching to the Canada Vigilance module, the scientist finds a similar missingness pattern. Attributes such as patient weight, height, drug indication, dosage, route of administration, and treatment duration all show high levels of incompleteness, again with more missing data among older adults (Figure 11).

Using IRVINE’s comparison module, the scientist observes that dosage and drug indication fields are more complete in OpenFDA, while both databases have comparable completeness for demographic attributes like age and sex (Figure 12). Although overall reporting completeness is broadly similar, discrepancies in certain drug-related variables could introduce bias if the datasets were merged directly. Thus, a selective integration approach would likely yield a more reliable dataset for modeling.

Overall, this exploration shows that although missingness remains substantial for several variables, focusing on younger adult subpopulations and carefully integrating Canada Vigilance data can improve data quality and make building robust machine learning models more feasible.

4.2. Identifying Incomplete Reporting Patterns

A policy maker at the FDA wants to determine which reporter groups are systematically submitting incomplete reports, with the goal of informing targeted interventions or new reporting guidelines. Using IRVINE, the policy maker frames their exploration around questions such as:

Q1. Which reporter groups contribute the highest proportions of incomplete reports?

Q2. Do these groups consistently omit the same attributes, or are certain data fields frequently missing across all groups?

Q3. Have regulatory efforts aimed at improving reporting completeness led to measurable improvements over time?

In the OpenFDA module, the policy maker activates the Source Qualification ring in the radial overview. Immediately, it becomes clear that reports submitted by consumers, pharmacists, and other health professionals have consistently higher missingness across multiple attributes compared to physician reports. By drilling into the detail view and adding visualizations for these reporter types, they observe that reports involving older adults (65–85 years) and serious outcomes tend to have higher missingness (Figure 13).

In 2019, the U.S. FDA introduced the Postmarketing Adverse Drug Experience (PADE) Reporting Compliance Program [35] to strengthen oversight of post-marketing safety reports and ensure that submissions are complete, accurate, and timely. This initiative emphasized including all required data elements to improve the overall quality of reports submitted to FAERS.

The policy maker uses IRVINE to examine whether this regulatory intervention had its intended effect. By applying a 2% threshold filter and focusing the temporal view on reports from 2019 onward, they find that overall reporting completeness generally declined in the years after the program’s implementation, with a slight improvement in 2021 (Figure 14).

This suggests that despite the 2019 compliance initiative, reporting completeness did not show sustained improvement in subsequent years. The persistent high rates of missingness among certain reporter groups indicate that regulatory guidance alone may be insufficient to ensure high-quality reporting.

4.3. Cross-National Reporting Behavior Comparison

A data quality analyst aims to understand how reporting practices and data completeness differ between the FAERS and Canada Vigilance systems. Rather than focusing on a single product, their goal is to compare overall reporting behavior and completeness in the two databases to identify systemic strengths and weaknesses. Using IRVINE, the analyst asks:

Q1. How do the overall levels of missingness differ between FAERS and Canada Vigilance across key feature categories?

Q2. Which attributes contribute most to overall incompleteness in each system, and do these reflect structural or procedural differences in data collection?

Q3. How do temporal trends in missingness differ between the two databases, and do these changes correspond to known policy or practice shifts?

Q4. What do these patterns suggest about differences in reporting behavior between the two pharmacovigilance systems?

In the comparison module, the analyst first examines the radial overviews for both databases. FAERS shows a higher overall proportion of missing data, while Canada Vigilance exhibits more missingness in certain drug-related attributes (such as dosage and treatment duration). The detail views reveal some similar patterns—for example, both datasets show frequent missingness in reports involving older adults and serious outcomes. Overall, the highest missingness in both databases comes from reports submitted by pharmacists (Figure 15).

Before comparing temporal trends, the analyst notes two key regulatory measures introduced in 2019 to enhance reporting completeness: the FDA’s PADE program and Health Canada’s mandatory hospital reporting guidance [36], which encouraged inclusion of all relevant case information.

Using IRVINE’s temporal comparison view, the analyst examines how reporting completeness evolved following these interventions. The visualization shows that FAERS missingness remained relatively stable, with minor deterioration after 2019, whereas Canada Vigilance gradually improved, consistent with the introduction of the hospital reporting mandate (Figure 16A).

To investigate the drivers behind these trends, the analyst activates IRVINE’s magic lens. In FAERS, the lens reveals that lawyer-submitted reports account for much of the post-2019 increase in missingness. In Canada Vigilance, nurses contribute the highest proportion of incomplete submissions.

Digging deeper, the analyst focuses on the year 2020 and reactivates the magic lens for a month-by-month view (Figure 16B). Both systems show a sharp rise in missingness in January 2020 followed by gradual improvement toward the end of the year. Using the lens on January pinpoints which reporter groups drove this spike.

In OpenFDA, the January 2020 treemap shows that lawyers (21.2%), pharmacists (22.7%), and other health professionals (20.5%) collectively contributed nearly two-thirds of the missing data, each with increases of roughly 10–11 percentage points compared to prior months. This pattern suggests the early-year surge in missing data affected most reporter categories but was especially pronounced among non-clinical reporters (like lawyers) who may have limited access to complete clinical information when filing reports.

Conversely, in Canada Vigilance, the January 2020 treemap presents a different profile: nurses (27.3%) and other health professionals (28.0%) dominate the missing data contribution, each rising by about 13 percentage points. This indicates that the temporary drop in reporting completeness was driven primarily by frontline hospital reporters rather than external parties. Given that these increases occurred soon after Health Canada’s hospital reporting mandate took effect, it suggests that initial adaptation to the new reporting workflow may have temporarily reduced data completeness, a pattern that stabilized later in the year.

By comparing these treemaps, the analyst infers that while both systems saw early-2020 increases in missingness, the sources of incompleteness differed: in OpenFDA it stemmed largely from non-clinical reporters, whereas in Canada Vigilance it came from hospital-based reporters. Such fine-grained insights enable targeted interventions—for example, additional training for hospital reporters in Canada or enhanced submission validation for lawyer or third-party reports in FAERS. This scenario illustrates how IRVINE’s magic lens supports diagnostic reasoning about the causes of missing data, turning temporal trends into actionable evidence to improve reporting quality.

5. User Evaluation

To evaluate the usability and effectiveness of IRVINE, we conducted a user study consisting of an exploratory session followed by an interview. The study examined how effectively users could perform core analytical tasks and identified strengths and limitations in the system’s interaction design.

5.1. Study Design

The study was conducted between December 2025 and February 2026 at a University in Canada and consisted of an exploration session and a follow-up interview session. The exploration session comprised four components: a demographic questionnaire, a familiarization period, task-oriented questionnaire, and a post-task questionnaire. Following this session, participants who consented to be interviewed were randomly selected and took part in the interview phase. Supplementary File S1 contains the interview script and the demographic, task-oriented, and post-task questionnaires.

Participants were recruited from university students. Eligibility criteria required that participants be at least 18 years of age, be currently enrolled as students at the university, and be able to use a mouse or trackpad and keyboard without assistance. The selection of a general student population was deliberate. The primary objective of this evaluation was not to assess pharmacovigilance domain expertise, but rather to examine the usability, interpretability, and exploratory capabilities of IRVINE in supporting the identification and analysis of missing data patterns within complex datasets.

In total, 20 participants took part in the exploration session, of whom 5 were selected for the interview session. Participant demographics and self-reported familiarity with visualization and adverse event analysis are summarized in

Table 1. Participant demographics (N = 20).

Variable	Category	n (%)
Education	Undergraduate	7 (35%)
	Graduate (Master’s/PhD)	12 (60%)
	NA	1 (5%)
Familiarity with Visual Interfaces	Very Familiar	4 (20%)
	Somewhat Familiar	9 (45%)
	Slightly Familiar	4 (20%)
	Not at All Familiar	3 (15%)
Familiarity with ADEs	Very Familiar	4 (20%)
	Somewhat Familiar	3 (15%)
	Slightly Familiar	4 (20%)
	Not at All Familiar	9 (45%)
Familiarity with SRS	Very Familiar	4 (20%)
	Somewhat Familiar	4 (20%)
	Slightly Familiar	2 (10%)
	Not at All Familiar	10 (50%)

.

5.2. Quantitative Analysis

The familiarization phase consisted of viewing a tutorial video (Supplementary Video S1) followed by an unguided exploration of the system. Participants subsequently completed a set of predefined tasks (

Table 2. List of 8 tasks designed to evaluate the effectiveness of IRVINE.

Task #	Description
T1	Identify which pharmacovigilance database (FAERS or Canada Vigilance) has a higher overall percentage of missing data and estimate the missingness in each system.
T2	Determine which reporter type categories in FAERS exhibit the highest and lowest levels of missing data.
T3	Analyze the distribution of missingness in Canada Vigilance based on outcome seriousness and identify features with the highest and lowest missing rates.
T4	Compare missing data levels between FAERS and Canada Vigilance for key attributes, including drug indication, dosage, treatment duration, and patient weight.
T5	identify the country contributing the largest number of FAERS reports and the country with the highest overall missingness.
T6	Examine the highest temporal spikes in missingness (2019–2024) in both databases and determine their magnitude and dominant contributing reporter types.
T7	Assess age group and gender differences in missingness patterns within the FAERS dataset.
T8	Identify report types with the highest missingness in FAERS and compare missingness patterns across report types with Canada Vigilance.

) using the system and provided feedback on its ease of use through a Likert-scale questionnaire.

Table 3. Perceived ease of task completions results (7-point Likert scale; lower values indicate more positive ratings).

Task	Mean	SD	Median
T1	1.15	0.37	1
T2	1.70	0.80	2
T3	1.75	0.85	2
T4	1.35	0.93	1
T5	1.10	0.31	1
T6	2.00	0.73	2
T7	1.50	1.00	1
T8	1.90	1.17	2

presents the results of the perceived ease of task completions by participants. To validate the internal consistency of the task-oriented questionnaire, Cronbach’s alpha was calculated for the eight evaluation questions. The scale demonstrated good reliability (Cronbach’s α = 0.75, 95% CI [0.54, 0.89]). A Friedman test was conducted to evaluate differences in perceived ease of use across the eight tasks. The results indicated a statistically significant difference in task difficulty, χ2(7) = 39.14, p < 0.001. The Kendall’s W coefficient of 0.28 suggests a moderate effect size, indicating that while most participants found the tool easy to use, certain tasks consistently presented more challenges than others.

To determine if prior expertise influenced the perceived usability of IRVINE, Spearman’s rank correlation was conducted between participant demographics (Familiarity with visual interfaces, ADEs and SRS datasets) and their average task difficulty scores. No significant correlations were found for familiarity with visual interfaces (ρ = 0.05, p = 0.83), ADEs (ρ = 0.09, p = 0.72), or SRS datasets (ρ = 0.14, p = 0.57). These results suggest that the system’s design is accessible to a general population, effectively supporting users regardless of their prior domain expertise or technical background in visualization.

Subsequent to completion of the task oriented questionnaire participants completed a post-task questionnaire.

Table 4. Post-task questionnaire Q2–Q11 results (7-point Likert scale; lower values indicate more positive ratings).

Question	Description	Mean	SD	Median
Q2	Confidence in answers provided	1.55	0.69	1
Q3	Perceived task ease completion	2.05	0.89	2
Q4	Main view effectiveness for assessing overall data completeness	1.55	0.6	1.5
Q5	Detail view effectiveness for exploring missingness across categories	1.35	0.67	1
Q6	Choropleth map effectiveness for geographic exploration	1.2	0.52	1
Q7	Temporal view effectiveness for identifying missingness trends	1.65	0.75	1.5
Q8	Effectiveness of cross-database comparison	1.5	0.76	1
Q9	Integration and interpretation across visualizations	1.9	1.02	2
Q10	Ease of visualization control	1.9	0.91	2
Q11	Effectiveness of integrated multi-view design	1.55	0.76	1

presents the results for Q2–Q11 of the post-task questionnaire. The post-task questionnaire demonstrated excellent internal consistency (Cronbach’s α = 0.89, 95% CI [0.80, 0.95]).

5.3. Qualitative Analysis

The data for the qualitative evaluation of our study were collected through two primary channels: unstructured questions presented in the post-task questionnaire and subsequent semi-structured interview sessions. While all 20 participants completed the questionnaire, 18 expressed interest in participating in the follow-up interview. From this pool of 18 participants, 6 were randomly selected and invited to the interview session to provide deeper insights into their experience with the system.

5.3.1. Post-Task Questionnaire Unstructured Questions

The focus of the unstructured questions (Q13–Q14) in post-task questionnaire was on the participants’ subjective impressions of IRVINE and their experience using it to identify and analyze missing data. Our analysis of the comments suggests that the participants’ experiences with the tool were generally positive, though their ease of use sometimes depended on their prior familiarity with visual interfaces and ADEs.

Overall, the majority of participants agreed with the general premise of the tool, finding identifying sources of missing data and its ability to facilitate cross dataset comparison to be highly effective and efficient. According to participant 14: “In the overall view page, everything is clear and can be interpreted easily… the important part of the visualization that worked well is how you can find patterns that have caused spikes in the missing data”. 10 out of 20 participants explicitly highlighted the clarity, responsiveness, or visual attractiveness of the system as a primary strength. Participant 10 also mentioned: “bright and attractive colors made me easily point out the answers”. Participant 17 said: “the visualization helps in finding the source of missing data pretty quickly as it summarizes quite a lot of statistical information in a very intuitive manner”.

Constructive feedback for potential design improvements was also mentioned by the participants. Five participants highlighted a need for improved labeling or the inclusion of tooltips to define domain-specific terms. For example, participant P13 suggested that providing “a very brief description for some features when I put the mouse on it” would help clarify the definitions of variables like “drug role” versus “drug indication”. Additionally, several participants suggested enhancing the navigation by adding dedicated buttons within the web application to traverse between pages.

5.3.2. Interview Session

Semi-structured interviews were conducted with 6 randomly selected participants to gain deeper insights into their experiences with the system. Overall, the participants found the system to be intuitive and effective for exploring missing data patterns, though their initial learning curve varied slightly based on prior domain knowledge and experience with visual interfaces. From these interviews, several consistent themes emerged.

First, participants repeatedly described cross-database comparison as a primary analytic advantage, noting that side-by-side structures support rapid identification of differences in missingness patterns. For instance, Participant 6 noted, “I really liked the comparative module. I think it was the most helpful feature for identifying patterns across databases because it compares both databases side by side”. Second, participants frequently described the detail view as the clearest representation for diagnosing what is missing and why, because it shows missingness distributions by subgroups and reporting context. Participant 16 stated, “The detail view provided the clearest information about what data was missing and the source of that missing information”. Similarly, Participant 14 found the Detail View particularly helpful, stating it “allowed me to compare different distributions of missing data and see how they were separated by gender, seriousness, and other factors”.

Third, multiple participants indicated that learnability is strongly shaped by labeling and initial orientation, particularly for users without domain knowledge. Some participants experienced a slight learning curve, especially those less familiar with “pharmacovigilance” terminology or complex visual interfaces. Missing labels in the main view and reliance on hover-only information reduced immediate interpretability; Participant 16 pointed out that without the tutorial video, the main view’s lacking clear labels meant they appeared as just “white arcs with black dots” and required direct interaction to fully understand.

Fourth, temporal analysis interactions, including the magic lens, were repeatedly described as valuable but initially confusing, suggesting a limitation in the system’s immediate affordances. As Participant 14 noted, navigating the temporal view required initial trial and error; this suggests that providing clearer visual cues would reduce the cognitive load required to understand how to effectively work with this view and interpret changes over time.

Participants also provided valuable constructive feedback for potential design improvements. Proposed navigational upgrades included dedicated navigation buttons to bypass browser navigation (P5, P14, P16), and search functionality for country lists (P6, P16). Regarding visual refinements, participants recommended adjusting circle marker placement and sizing in the radial visualization to improve affordance and visibility (P11), alongside broader updates to the color palette and labeling for better readability (P14, P15). Finally, P11 suggested implementing cross-variable analysis to explore concurrent missingness patterns across multiple dimensions.

Despite these identified areas for improvement, all interviewed participants agreed that IRVINE holds significant utility for pharmacovigilance workflows. Participants 5 and 6 specifically noted its value for data scientists in preparing datasets and identifying appropriate, high-quality subsets for training machine learning models. Participant 11 summarized the system’s impact by stating, “In research aimed at improving human health, it is important to understand what we do not know. IRVINE helps reveal gaps in knowledge and highlights areas that require greater attention”.

5.4. User Evaluation Discussion

Overall, the user study indicates that IRVINE was perceived as easy to use and effective for exploring patterns of missing data, with systematic variation across tasks. Ease-of-use ratings were consistently positive but differed significantly by task, suggesting that task-specific interactions, rather than overall usability issues, accounted for perceived difficulty.

The most challenging task involved identifying temporal spikes, which aligns with interview feedback indicating that temporal interactions, including the magic lens, were valuable but initially confusing and required trial-and-error. In contrast, strong ratings for the choropleth map, detail view, and cross-database comparison were reflected in participants’ comments, which highlighted side-by-side comparison as especially helpful and described the detail view as the clearest source of information for diagnosing missingness.

Perceived usability appeared broadly accessible, as task difficulty was not significantly associated with participants’ prior familiarity with visual interfaces, adverse event data, or spontaneous reporting systems. Although some participants reported an initial learning curve due to unfamiliar terminology, this did not substantially hinder performance.

Qualitative feedback suggests that clearer labeling could further reduce initial learning effort. Many participants praised the system’s clarity, responsiveness, and visual appeal, while others explicitly requested tooltips or definitions for domain-specific terms. Participants also recommended dedicated in-app navigation controls, searchable country lists, and improved radial markers and labeling, particularly in areas where integration and control were rated less positively and where reliance on hover-based cues limited immediate interpretability.

Taken together, these results support the effectiveness of IRVINE’s multi-view comparative design, while highlighting the need for improved support for temporal exploration and clearer embedded labeling to maintain rapid pattern discovery across users with different levels of prior familiarity.

6. Conclusions

IRVINE is an interactive visualization system for exploring, interpreting, and comparing missing data in large spontaneous reporting systems. By integrating coordinated visual components, the system allows users to seamlessly transition from high-level summaries to fine-grained diagnostic views. Through three usage scenarios, we showed how diverse stakeholders (including researchers, policy makers, and data quality analysts) can use IRVINE to examine reporting completeness, assess data readiness for modeling, and evaluate the impacts of regulatory interventions.

The application of IRVINE to FAERS and Canada Vigilance data highlights several broader findings. Missingness remains a pervasive challenge, especially among non-clinical reporters and in drug-related fields such as dosage and treatment duration. However, comparative and temporal exploration reveals that missingness patterns are not random; they reflect systemic, contextual, and policy-driven factors that can be visualized and assessed directly. By turning abstract data-quality issues into interpretable visual evidence, IRVINE helps bridge the gap between data transparency and actionable insights.

Importantly, the results of our user evaluation provide empirical support for IRVINE’s usability and analytical effectiveness. Quantitative findings indicate that participants consistently perceived the system as easy to use across a wide range of analytical tasks, regardless of prior familiarity with visualization tools or pharmacovigilance data. Qualitative feedback further confirmed the value of IRVINE’s multi-view design, particularly its detail view and cross-database comparison module, which users identified as central to diagnosing missingness patterns. While temporal exploration and advanced interactions such as the magic lens were viewed as highly valuable, they also presented a steeper learning curve, highlighting opportunities for improved visual affordances and embedded guidance.

While IRVINE provides a flexible, interpretable framework for exploring missing data in spontaneous reporting systems, it has some limitations. First, we focused on a subset of attributes (primarily drug-related and demographic fields) to balance interpretability and comprehensiveness. Many technical or administrative fields (e.g., report transmission metadata, regulatory identifiers, high-cardinality product codes) were excluded from our visualization. Although important for regulatory tracking, those fields offer limited analytical value for understanding missingness patterns and would add considerable visual and computational complexity. Similarly, narrative and clinical text fields were omitted due to their unstructured nature and inconsistent reporting across databases.

Second, IRVINE’s scalability is constrained by data size. As dataset volume and complexity grow, the time required to preprocess data and generate missingness summaries also increases, which may limit the system’s responsiveness for real-time interaction with very large datasets. Third, although participants were able to complete analytical tasks effectively, the evaluation did not involve in-depth domain-specific reasoning tasks typical of expert pharmacovigilance workflows. Future studies could assess IRVINE’s support for more complex regulatory decision-making scenarios requiring advanced domain knowledge.

Looking ahead, this work underscores the importance of integrating interactive visualization into pharmacovigilance workflows. Future enhancements could incorporate real-time data feeds from additional regulatory sources and extend the visualization framework to other data integrity dimensions such as duplicate reports or data inconsistencies. Ultimately, IRVINE lays a foundation for more reliable, interpretable, and user-centered data quality assessment in global pharmacovigilance systems.