VICTORIOUS: A Visual Analytics System for Scoping Review of Document Sets

Abstract

Scoping review is an iterative knowledge synthesis methodology concerned with broad questions about the nature of a research subject. The increasingly large number of published documents in scholarly domains poses challenges in conducting scoping reviews. Despite attempts to address these challenges, the specific step of sensemaking in the context of scoping reviews is seldom addressed. We address sensemaking of a curated document collection by developing a VIsual analytiCs sysTem for scOping RevIew of dOcUment Sets (VICTORIOUS). Using known methods within the machine learning community, we propose and develop six modules within VICTORIOUS: Map, Summary, Skim, SemJump, BiblioNetwork, and Compare. To demonstrate the utility of VICTORIOUS, we describe three usage scenarios. We conclude by a qualitative comparison of VICTORIOUS and other available systems. While existing systems leave their users with singular information items regarding a document set and gaining an aggregated assessment in a scoping review is often a challenge, VICTORIOUS shows promise for making sense of documents in a scoping review process.

1. Introduction

An unprecedented rise in the number of published research articles has led to increasing interest in knowledge synthesis methodologies. Systematic reviews, meta-ethnography, rapid reviews, narrative reviews, realist reviews, and scoping reviews are among the well-known knowledge synthesis methodologies [1,2,3,4,5]. Scoping reviews are often used to understand the nature of an emerging or new discipline and for preliminary assessments of scholarly writings.

Scholarly writings are a specialized type of literature, considering their topics, intended audience, content, and presentation methods. The primary reason for their publication is to share knowledge and findings with peer scholars. From peer-reviewed journal articles, conference proceedings, and books to webpages, we refer to all such scholarly works only as documents.

An increasingly significant number of documents are being published every day. Only in the Sciences and Engineering, U. S. National Science Foundation cites a figure of ~3.34 million articles published in 2022 [6]. Compared to a decade earlier, this number of publications has increased by ~70%. Elsevier’s abstract and citation database, Scopus, indexes ~3 million new records across all academic disciplines every year, with a growth rate of ~6% annually [7]. This means at least 8000 peer-reviewed articles are being published daily across the world.

The open science movement, with an ever-increasing push towards Open Access publications, has made a breakthrough in recent years. Despite being a high-level statistic, the numbers above also indicate a considerable increase in both the scope and number of articles published within the scholarly enterprise. While critics raise concerns over quality control and financial sustainability [8], proponents advocate for even wider dissemination of scholarly material. With this growth in the size and scope of documents, identifying the leading edge, filtering out the low-quality research, and keeping up with the latest developments in each field seem increasingly difficult tasks.

To handle this complexity, many databases, document indices, search engines, and generative artificial intelligence (AI) models have been developed. We refer to these as document management systems (DMSs). By leveraging searchable tags, keywords, highlights, titles, author names, and quotations, DMSs like Semantic Scholar, Google Scholar, and Microsoft Academic serve as crawler-based search engines.

Scoping reviews are composed of a sequence of steps with the ultimate goal of identifying the nature of the literature/subject under study [9,10]. Exact definitions of these steps vary depending on the context to which a scoping review is applied, as documented by Munn et al. [10]. Generally, these steps are traversed iteratively and consist of:

• identifying or fine-tuning the research question;
• identifying relevant documents;
• selecting documents that are to be examined further;
• sensemaking and classifying documents;
• summarizing results.

Typically, researchers use a DMS only for the initial steps (1–3) of a scoping review. Afterwards, they focus on “making sense” of the curated documents. In this sensemaking stage, tasks such as comparison, knowledge acquisition, aggregation, navigation, analysis or evaluation are focused on [9,11,12,13,14]. Beyond simple querying/searching of academic databases or DMSs, each document of interest calls for in-depth analysis and contextual understanding. Existing approaches (Section 2) address only a limited number of tasks in the sensemaking process and do not fully support the iterative fine-tuning needs.

As part of a larger research project on DMS and research assistance tools, this paper is concerned with making sense of a curated document set in the context of scoping reviews. We introduce VICTORIOUS, a VIsual analytiCs sysTem for scOping RevIew of dOcUment Sets. The system’s main objective is to help in making sense of a set of documents in the context of scoping reviews. In this paper, we view sensemaking as iterative probing of a document set to help in classification and assessment of each document’s relevance. Given the iterative nature of a scoping review, the sensemaking step enables ideation about, validation, or verification of the hitherto gathered results of the review process.

We assume that search-based querying approaches or exploratory searches using a DMS are performed before using VICTORIOUS. Once this search is performed and a tentative set of documents is curated, we propose VICTORIOUS to help in making sense of the set, classification, and relevancy assessment of each document. In particular, the research questions that we address in this paper are:

• What tasks does the sensemaking step of a scoping review include?
• What visual analytics modules could be developed to support these tasks?
• How can these modules be composed to help a scoping review process?

The rest of this document is structured in the following manner. In Section 2, we investigate the defining concepts for this work and survey the existing works related to DMSs and various visual analytics systems developed for scoping reviews. In Section 3, we give details of our design of VICTORIOUS. In Section 4, we demonstrate the usefulness of our system via a scenario. Finally, we conclude and discuss the results in Section 5.

2. Background

In this section, we provide definitional descriptions and a brief survey of the concepts foundational to this work. We look at:

• knowledge synthesis methodologies and scoping reviews;
• existing visual analytics approaches in knowledge synthesis.

2.1. Knowledge Synthesis Methodologies

Chalmers et al. [15] in their seminal work provide a brief history of knowledge synthesis methodologies. Starting from within medical literature, structured reviews of studies began to appear in the 1970s. This led to the emergence of research groups such as Cochrane and the Joanna Briggs Institute (JBI) in the 1990s. Increasingly, methodologically structured reviews have become dominant in synthesizing evidence and establishing scholarly claims. While traditionally systematic reviews appeared only in healthcare literature and have been the primary method of knowledge synthesis, we have recently seen the emergence of methodologically structured reviews, specifically scoping reviews, in non-healthcare literature dealing with different knowledge constructs [10]. We construe other niche/novel review types, e.g., meta-ethnography (qualitative assessments), rapid reviews (omitting certain components of a systematic review), narrative reviews (based on expert’s opinion), and realist reviews (scoping review with more focus on sensemaking), as special cases for the two broad categories of systematic and scoping reviews.

In this section, we briefly introduce our survey of systematic and scoping review methods and provide a comparison to outline their differences.

Systematic Reviews Compared with Scoping Reviews

A systematic review investigates a body of knowledge within a domain regarding a specific question and with a pre-defined rigorous methodology for every step. According to the Cochrane handbook for Systematic Reviews of Interventions [16], a review process that uses a formal, explicit approach in gathering evidence and is designed to minimize bias should be called a systematic review. In contrast, a scoping review considers the scope of a body of knowledge and identifies key concepts within it. Researchers conduct scoping reviews on an emerging body of literature to understand its nature and possible knowledge gaps. In doing so, they answer questions about the relevancy of a tentative systematic review on that body of knowledge.

Having an explicitly structured and rigorous methodology is definitional to a systematic review and its results are the pillars of evidence-based scholarship. In contrast, an agreed upon, formal definition of the methodology and framework of a scoping review is still to be attained. A summary of goals offered by Munn et al. [10] for both of these review methodologies is provided in

Table 1. A comparison between the goals of a systematic and a scoping Review. Each focus area explains the primary direction of the goal.

Focus Area	Systematic Review Goals	Scoping Review Goals
Broad research questions	Classify existing body of knowledge regarding a particular research question	Characterization of research methodologies used within the literature
Particular research questions	Identify and investigate conflicting results	Clarify key concepts within the literature
Evidence	Uncover emerging evidence for a hypothesis	Identify a typology of evidence
Knowledge gaps	Identify and inform future research possibilities	Identify possible knowledge gaps within a domain
Outcome	Typology(ies) of existing research	Assessment of relevancy for a future systematic review

.

If the goal of a review is to address the feasibility, meaningfulness, and/or effectiveness of a specific method, a systematic review is likely the most valid approach. In contrast, when instead of precise questions, an identification of key concepts is sought after, a scoping review is preferred.

Table 2. A comparison between traditional systematic and scoping review methodologies offered by Munn et al. [10].

	Traditional Literature Reviews	Systematic Reviews	Scoping Reviews
Rigorous review protocol	No	Yes	Yes (some)
Explicit, transparent, peer-reviewed search strategy	No	Yes	Yes
Standardized data extraction forms	No	Yes	Yes
Critical appraisal (risk of bias assessment)	No	Yes	No
Synthesis and generation of summaries	No	Yes	No

summarizes various aspects of each of the traditional, systematic, and scoping review methods. Exact definitions of the steps to conduct a scoping review vary within the literature [10]. By examining foundational works by Arskey and O’Malley [9], Levac et al. [17], and the methodological working group of JBI [18,19], we summarize these steps in the following order.

Since scoping review is an iterative process, these steps are to be performed iteratively to fine-tune the outcome:

• identifying and fine-tuning the research question;
• identifying relevant documents;
• selecting documents that are to be examined further;
• sensemaking and classifying documents;
• summarizing results.

Support for steps 1–3 is usually provided by the existing DMSs and various existing search engines. An iterative process of fine-tuning results by looping between steps 3 and 4 is highly context-dependent and complex by nature [10]. Sensemaking of a document requires an in-depth analysis of that document in the context of the larger document collection. Hence, for step 3, individual documents must be drilled down to just as well as being viewed in the context of the broader collection. In our review of the literature, supporting steps 3 and 4 (i.e., iterative selection and fine-tuning of the selection of documents by sensemaking from and classifying them) is seldom addressed. To properly characterize these steps, we need to further examine the tasks that a researcher conducts while going through the steps.

2.2. Visual Analytics for Sensemaking

Sensemaking is a general term that suggests the active processing of information to achieve understanding. It can be considered as a continuous effort to understand connections among entities such as people, places, and/or events to anticipate their direction and act accordingly. Sensemaking activities do not always have a clear beginning and/or end. Instead, they are usually characterized as ill-structured and open-ended activities where potentially conflicting pieces of information should be gathered and understood.

Russel et al. [13] consider sensemaking to be a process of searching for a representation and answering task-specific questions by encoding data in that representation. More recently, and in the context of large language models, sensemaking is viewed as the activity used to explore topics without adding further complexity [20]. This definition consists of two essential components: representations and task-specific questions. As such, sensemaking activities involve establishing some goals, discovering the structure and type of involved informational items, generating required questions, and organizing corresponding answers to the raised questions.

Such conceptualization assumes an existing space of information in which each piece/part has a role to play in the sensemaking process. By extension, the sensemaking process can be viewed as consisting of four main sub-processes:

• gathering information;
• encoding the information in a new representation;
• gaining insight through manipulation of the new representation;
• generating knowledge based on the resulted information.

Consequently, we can infer two key features for sensemaking processes:

• the dynamic flow of data, i.e., initially found informational items might be removed/reformed during the process (referred to as “dynamic knowledge extraction”);
• the iterative nature of the sensemaking process, in which the sub-processes can come with many back loops and occur one after the other to further polish the formed mental model (referred to as “Multi-resolution sensemaking”).

The combination of automated analysis techniques with interactive representations (i.e., visualizations) constructs visual analytics systems (VASs), enabling effective understanding, reasoning, and decision making based on complex and large datasets [21]. In the context of scientific documents, there has been an increase in the development of VASs in recent years that support the search and analysis of documents. Various data types can be extracted from a collection of documents ranging from the textual content, which is the central component and encodes information in a natural language form, to the metadata of each document and citation-based inter-document connections. Many of the current VASs only focus on a subset of these data types or activities during the sensemaking process.

To cite an example, ref. [22] only visualizes each document’s derived topics and concepts in a 2-D visualization canvas. Therefore, textual content, figures and tables, bibliographic information and other important information of the documents should be extracted from other sources. This lack of ability to penetrate the inner layers of information may result in gaps between users’ mental models and external representations [23]. Encoding a large subset of information derived from a collection of scientific documents may result in researchers’ inability to navigate properly through the visualizations and answer the information needs during the sensemaking process. Reading textual content of documents, discovering semantics-based relationships, comparing documents, and articulating hypotheses about the studied scientific documents are time-consuming and challenging activities to be supported by VASs.

2.2.1. Existing Approaches

With the open science movement on the rise, several major projects in knowledge synthesis and sensemaking have been undertaken in previous years. All assume open and unlimited access to scholarly content. We catalogue the major projects in Table 3.

When conducting a survey of these projects, it can be noticed that most of them do not offer reusable results to their users. Oftentimes, the results are reported to be not as promising and only a partial prototype of these systems has been delivered [24]. Also, except for Covid-on-the-Web, all have been only developed preliminarily and abandoned in favor of a competitor VAS or text-mining system that offers ease of use to users.

Table 3. Recent major projects in knowledge synthesis.

Initiative	Aim/Objective
OpenMinTed [25]	“[T]o create an open, service-oriented Infrastructure for Text and Data Mining (TDM) of scientific and scholarly content”
Visa TM [26]	Serves as the core knowledge extraction component for OpenMinted by integrating thesauri, ontologies, and semantic resources from multiple domains.
ISTEX [27]	Constitutes corpora of scientific publications; provides tools for research communities to explore curated corpora subsets.
LODEX [28]	Formulating queries on Lined Open Data, to be used in ISTEX.
Covid-on-the-Web [29]	Provides access, extraction, and querying of knowledge from coronavirus-related literature.

Table 3. Recent major projects in knowledge synthesis.

Initiative	Aim/Objective
OpenMinTed [25]	“[T]o create an open, service-oriented Infrastructure for Text and Data Mining (TDM) of scientific and scholarly content”
Visa TM [26]	Serves as the core knowledge extraction component for OpenMinted by integrating thesauri, ontologies, and semantic resources from multiple domains.
ISTEX [27]	Constitutes corpora of scientific publications; provides tools for research communities to explore curated corpora subsets.
LODEX [28]	Formulating queries on Lined Open Data, to be used in ISTEX.
Covid-on-the-Web [29]	Provides access, extraction, and querying of knowledge from coronavirus-related literature.

Due to their unmanageable scope, most of these projects suffer from incoherent implementation and delivery. Covid-on-the-Web, despite covering a large archive of scholarly publications, has managed to sustain itself and deliver promising results. We can assume that such success is due to the limited scope of its focus.

Similar to Covid-on-the-Web, most novel approaches for encoding scientific information integrate visualization techniques with natural language processing (NLP) techniques. Encoding textual content of scientific documents can be used to support different activities ranging from finding and comparing documents to extracting patterns, relations, or individual data items from each document in the context of a collected document set.

More specifically, in

Table 4. A catalogue of existing VASs that focus on knowledge synthesis methodologies.

Visual Analytics System	Visualization Module(s)	Year
A visual analytics environment for navigating large conceptual models by leveraging generative artificial intelligence [30]	Conceptual models extracted from a document	2024
PaperMage [31]	Text annotation in and extraction from individual documents	2023
ISSA [24]	Topical inter-document relationships; individual document metadata	2022
Geo-quantities [32]	Numerical properties from individual documents	2021
An Aneka-based system for COVID-19 datasets [33]	Aggregate statistics of a document archive	2020
Topic modeling for systematic review of visual analytics in incomplete longitudinal behavioral trial data [34]	Topic modeling; longitudinal information of a document	2020
An interactive visual analytics system for incremental classification based on semi-supervised topic modeling [35]	Intra-document topic modeling	2019
Progressive learning of topic modeling parameters [36]	Intra-document topic modeling; reinforcement learning	2018
VISTopic [37]	Intra-document hierarchical topic modeling	2017
A system for cross-domain topic mining [38]	Topic modeling/mining	2016
Rexplore [39]	Topic modeling/similarity	2013
Maps of computer science [40]	Intra-document topic modeling; bibliographic information modeling	2013
CyBiS [41]	Bibliographic information	2011
ParallelTopics [42]	Topic modeling	2011

, we provide a brief survey of VASs that have been developed in relation to knowledge synthesis. We make note of each system’s focal point in its core visualization module.

What these systems have in common is that their aim is to provide a single unifying framework for all knowledge tasks. Their differences include their respective data types, data processing pipelines, interaction techniques, and visualization components. Furthermore, and as delineated in Table 4, we observe that most systems focus either heavily on (a) archive-level, aggregate analytics or (b) individual-level, particular insights of documents. This compartmentalized approach to knowledge synthesis often creates a knowledge gap, leading to a decontextualization or blunting of the research question.

2.2.2. Visualization/Design Principles

Information visualization literature is insightful in finding design principles for a VAS. In

Table 5. A summary of widely used information visualization principles.

Principle	Summarized Definition	Key Focus
Sensemaking Loop	Iterative process of interacting with data to derive meaning.	Iterative interactions
Schneiderman’s Mantra	“Overview first, zoom and filter, then details on demand”	Structured and disciplined exploration
Focus + Context	Simultaneous visualization of a focused data item and its context	Contextualizing details
Details on Demand	Providing details only when requested	Preventing information overload

, we summarize some of the widely used principles. Heterogenous datasets and complex sensemaking tasks leave visualization designers wanting to utilize a conjunction of principles such as Focus + Context, Sensemaking Loop, Details on Demand, and Schneiderman’s Mantra. Using multiple coordinated views with the aim of helping users interactively change their perspectives on both aggregated and individualized insights has been a widely adopted technique. Also, this technique has traditionally helped with the dynamic flow and iterative nature of the sensemaking process [43,44].

2.3. Automated Analysis in Research

Recent scholarship suggests an increase in the use of tools enhanced by various AI/ML models for the broad activity of research [45,46,47]. For simplicity, we refer to these approaches as the use of automated analysis in research. This approach has been subject to multiple concerns from scholars. Depending on accuracy and biases of each specific tool/model, its usage in research is encouraged or discouraged and recommendations with various degrees of human oversight are offered. Here, we briefly survey these issues from recent works.

While the recent debate on the use of automated analyses is still in its nascent stage, most works tend to acknowledge the efficiency gain as a result of such use [48,49,50,51]. At the same time, a great deal of attention has been put on the biases and weaknesses of these models and a greater degree of human oversight is called for [48,51,52,53,54,55]. The literature suggests that an answer to the following question determines the degree of human oversight needed/called for: “to what extent can the results of an automated analysis be validated and verified?”

Offering mathematical certainty, formal verifications of the output of statistical models are not always possible. In most cases, benchmarks are used to indicate a threshold of trust. We note that the opaque, “black-box” nature and haziness of the utility of statistical models used in various automated analyses are the cause of such skepticism.

Another area of concern highlighted in recent debates is intellectual property issues and copyrights. We note that these concerns are twofold:

• Data ownership [56,57]: Who gets to own proprietary data? The “thorny issue of data ownership” [58] is more emphasized when the automated analysis is outsourced to another entity, often (though not always) through the usage of a commercially licensed API. This dimension focuses on the tension between commercial AI service providers wanting to use input data for model training and users seeking to maintain confidentiality rights.
• Intellectual property [48,54,55]: Who should be credited with the produced results, if anyone? This dimension is concerned with academic integrity and plagiarism concerns, with broader ramifications on the very nature of academic inquiry and epistemology.

As inconclusive as these debates seem, the rapidity of development in the fields of AI and AI-adjacent is a factor that should not be overlooked [54,55]. We note that apart from data ownership, which is entirely dependent on the architecture of the data-processing pipeline, all the identified aspects of this debate are dependent on the nature of these AI models and how well they might respond to various user needs, including giving proper credit to the sources used in producing results. Future works may accentuate or tone down some of these concerns.

3. Proposed System

Here, we first describe the overall design and architecture of our system, VICTORIOUS. In the latter part of this section, we introduce and describe in detail the six key modules: Map, Summary, Skim, SemJump, BiblioNetwork, and Compare.

Developed using a server–client architecture, VICTORIOUS is composed of two main components: Application Programming Interface (API) and User Interface (UI). The API component is responsible for responding to and populating the requests and queries from the UI component. In the following section, we briefly describe the details of API and its Document Processing Pipeline (DPP).

3.1. Document Processing Pipeline (DPP)

The starting point in VICTORIOUS is document selection and upload. By selecting and uploading PDF documents, as seen in Figure 1, the user triggers DPP. Once the process starts, the unstructured PDFs are used to extract structured data, e.g., title and abstract. Each document is vectorized to enable numerical operations. Important features (dimensions) of the vectors are combined to reduce dimensionality and form the most important derived features. Finally, clusters of similar documents are identified. In this section, we briefly provide details on each step. Figure 2 represents a summary of the steps performed within DPP:

• extracting text from PDF files (Grobid [59]);
• vectorizing documents (AllenAI-Specter [60]);
• reducing dimensionality of the vectors (PCA);
• clustering of the reduced dimensions (K-Means);
• retrieving relevant information for each module (using various API calls).

3.1.1. Extracting Text

Due to the flexibility of PDF typesetting, PDF documents come with numerous layouts in which the position of headings, footings, texts on side margins, tables, figures, and other components of a document are determined arbitrarily. This makes text extraction a hard problem to solve using traditional methods.

Some studies have addressed this problem by adopting solutions such as rule-based methods, machine learning models, or various heuristics. Open-source tools like pdftotext3 [61], PDFMiner [62], and Apache PDFBox [63] are some of the widely used tools and libraries to convert PDF documents into text or structured XML/HTML formats. However, they often fail to make a clear distinction between the main body of the document, e.g., the text of each subsection of the document and its redundant parts, e.g., headers and page numbers in PDF documents.

After considering various tools, we decided to use Grobid, which uses ML models in each part of PDF documents to extract, parse, and re-structure them into structured XML/TEI encoded data. These models include the “full-text” model that attempts to identify the body of the document, e.g., paragraphs, section titles, figures, and tables. An example output of Grobid is shown in Figure 3.

Once the uploaded documents are passed through the Grobid tool, an XML file containing the structured content of the PDF is created for each. Sentences and paragraphs inside each section of the document are separated and can be accessed directly in the XML file. This XML file is then used to respond to later user queries sent from User Interface.

3.1.2. Vectorizing Documents

Using the textual content stored in XML files, we then vectorize the content of the documents. This is necessary to achieve an embedding space and enable numerical operations on the textual content. Simply put, the goal of this step is to generate numerical features for our textual content.

We classify existing approaches in word-sentence-document vectorization into two main categories: task-dependent and task-agnostic. Task-dependent techniques require fine-tuning measures for every dataset for which they are being used. This complicates the vectorization process while producing more accurate features. Task-agnostic approaches generally result in pretrained models from a large enough dataset that can be used without a need for fine-tuning with sufficient accuracy in benchmarks.

Given the nature of our dataset and the task of scoping review, we use Scientific Paper Embeddings using the Citation-informed TrnsformERs (SPECTER) model by AllenAI [60] for vectorization. This is a special purpose transformer designed for and trained on scientific documents. SPECTER uses a combination of title and abstract, in conjunction with a large corpus of citations for vectorization. It is specifically designed to produce similar vectors for documents that share common citations and different from those that do not. This is a novel technique and idea that shows significant improvement compared to others and has promising results in benchmarks. Using SPECTER, we end up with a vector for each document in a large, common embedding space. Using this technique, the documents with commonalities in both topics and citations will be closer to each other. The result of this step is the production of features for documents, based on their textual content and citation links.

3.1.3. Reducing Dimensionality

Once each document is represented by a vector in a high-dimensional embedding space, to better make sense of the complexity of this space, we need to reduce dimensionality and extract the most important dimensions. While a plurality of dimensionality reduction mechanisms exists, for the purposes of this work and for simplicity, we perform this by running a Principal Component Analysis (PCA) on the obtained features for documents. In doing so, we derive components through a linear combination of the most important features in the original vectors. Our criterion for the number of components is that they must explain almost all (>99%) of the variance in the underlying data. This threshold is experientially set and, according to the articles we have, gives us the best results.

3.1.4. Clustering Documents

Using the derived components from the last step, we use the K-Means algorithm to cluster the documents and find 2–5 centroids. To do so, and to compare the quality of centroids in each round of clustering, we use the well-known silhouette score for each clustering attempt. The clustering attempt that maximizes the obtained silhouette score is the one that produces for us our centroids. We refer to these centroids as our topics. These topics are representative of the clusters of documents and their subsequent keywords which are included in our uploaded documents.

3.2. User Interface

Once the documents are uploaded to the API component and are passed through DPP, each is represented by a color-coded rectangle on Initial Screen, as shown in Figure 4. The columns represent themes. Themes are identified by the K-Means analysis performed on all the textual data of all documents (Step 4 of DPP). By identifying 2–5 topics within the uploaded collection, we identify and group documents that bear closer semantic relationships. By default, we sort documents on this screen in chronological order, according to their publication dates (see Figure 4). The ascending or descending order of display can be changed. Also, the two criteria of (a) number of bibliographic out-links and (b) alphabetical titles could also be selected as the sorting criteria, as can be seen in Figure 5 and Figure 6.

Although visualization of scientific information mainly faces the challenge of a large number of scientific documents and information overload [64], to afford rapidity of information exploration and the sensemaking process in general, we visualize all the processed scientific information in one visualization canvas. Performing numerous sub-tasks in one scrolling window to make sense of a collection of scientific documents might also be challenging and lead to potential loss of context. Therefore, we adopt a fisheye-like view [65] visualization technique to enlarge specific regions of interest, suppress other regions, and maintain the global structure of the visual marks to avoid losing context, a pane called Sliding Pane.

As in Figure 5, we view documents with a fisheye effect in our Initial Screen. A draggable pane uses the following formulae to dilate specific documents in the canvas:

$$\# of Enlarged Documents={\left({\displaystyle \frac{Sliding Pane Height}{Canvas Height}}\right)}^2\times \# of Documents$$

$$Document Rectangle Height={\displaystyle \frac{Canvas Height-\#of Documents}{2\times \# of Documents-\# of Enlarged Documents}}$$

$$Enlarged Document Rectangle Height={\left({\displaystyle \frac{Canvas Height}{Sliding Pane Height}}\right)}^2\times Document Rectangle Height$$

Users can use the controller bar on the left side of Sliding Pane to drag it over all the canvas in vertical directions and change its height to cover more or fewer document rectangles in the lens. An indicator on the controller bar also displays the proportion of documents covered by the pane, as shown in Figure 6.

Once the documents are loaded to this screen, users can change the order of display using the various controls provided in Initial Screen, as seen in Figure 7 and Figure 8. Also, users can start the task of scoping review by interacting with the documents through the six modules we have developed. Each module represents a different aspect of the documents and their relationships with each other. In the following section, we describe the design of these modules in terms of the visualizations and interactions afforded by each.

3.3. The Six Modules of VICTORIOUS

Given the five iterative steps to conduct a scoping review (Section 2.1), we introduce six interactive visualizations. We refer to these as the six modules of VICTORIOUS. Each module is designed to assist the user, in tandem with the other five, in conducting the iterative steps of a scoping review.

We employ the metaphor of a lens for these modules. Like superimposing different lenses over objects to view them with a varying degree of detail, we employ lenses to look at documents with different focus points and varying details. Once documents are loaded to Initial Screen, scrolling the mouse wheel results in Sliding Pane switching among the six modules. When a document under any one of the modules is selected, the document rectangles covered by Sliding Pane will display corresponding details of the document according to the selected module; these modules act as lenses on the pane and display different information from the document under examination.

As shown in Figure 9, we have provided six lenses to cover different information items of documents and properly support the rapid and efficient sensemaking of the Scoping Review process. In the rest of this section, we explain details of each lens. In each of the following subsections, we first introduce the general goal of the module, followed by some details on how it functions. Then, we state the rationale of the module design and its contribution to scoping reviews.

3.3.1. Map

Map provides an overview of the metadata of each document in the context of all uploaded documents. This is the default lens of VICTORIOUS.

Once a document is selected (Figure 10), further information about the document is visible. In addition to the title, publisher, and publication year, this module provides a list of the authors, keywords, and number of out-link references mentioned in the document.

To represent the publication date of the selected document within the context of the uploaded documents, we use a bar that encodes the period of publications from among the uploaded documents and a single arrow to mark the year of publication for the selected document. Similarly, to contextualize the number of citations in a document, we use two concentric circles. The outer circle denotes the maximum number of out-links that exist among the uploaded documents and the inner circle denotes the selected document’s number of out-links.

This module contextualizes each document within the larger picture of all uploaded documents. By providing concise information on authors, references, date, publication venue, and keywords, this module helps with assessing the relevance of a given document rapidly.

3.3.2. Summary

The Summary lens displays a condensed representation of a document to support outlining its content rapidly and forming hypotheses.

Upon selecting a document (Figure 11), as an alternative to its abstract, we provide a method for collecting the most important sentences of the document by counting their word frequencies. Users can choose between the abstract and the most important sentences, as they see fit, to quickly outline the selected document.

Once a document is parsed in DPP, each sentence receives a score based on the frequency of each word of that sentence in the whole document. The sentences are then sorted according to this score. A higher score for a sentence means it embodies more keywords. The default view on this module is to show the top 50% of the most important sentences. There are two more values from which users can choose: top 10% or 90%. This is used to decrease or increase the context of the sentences, as shown in Figure 11. Upon using this module, hypotheses can be formed about a given document.

This module affords quick assessment of the textual content of each document by the user who has domain knowledge of the documents and wants to rapidly assess the relevance of the document without unnecessarily drilling into them.

3.3.3. Skim

Skim provides a contextualized view of the content of a document, including most important sentences, keywords, and outline of a given document to support drilling into the content of the document and verification of hypotheses formed using Summary.

Upon selecting a document (Figure 12), distinctly formatted textual content reflective of keywords and most important sentences is displayed. Users can change the formatting of keywords, sentences, and even view the actual PDF document or only an outline of it in place.

Pre-attentive visual attributes (e.g., size, hue, speed, direction) are difficult to ignore and mostly unaffected by the high load of information. Therefore, in this module, we encode each sentence visually with a specific font size according to each sentence’s importance score. Likewise, we distinguish keywords by using a different color. Also, compression ratio, which determines the font size ratio of the most important sentence to the least important one, is utilized to provide a noticeable contrast in the textual content. The default value for compression ratio is 2, which can be changed to 1.5 or 3 from the side panel.

This lens affords more in-depth assessment of the textual content and the hypotheses that are already formed or are being formed. Once ideation and hypothesis forming starts, this module can be used to quickly bring about their verifications.

3.3.4. Semantic Jump (SemJump)

The Semantic Jump (SemJump) lens provides a restructuring option of the documents according to a search query. Through two modalities, users can either search for a term or a sentence within the selected document or search across all uploaded documents.

When SemJump is opened with a document (Figure 13), sentences of the document are displayed in an interactable format. Selecting a sentence will result in a list of semantically similar sentences compiled and displayed. This is performed by calculating a cosine similarity metric between the selected sentence and other sentences in the document and displaying results in a descending order on the interface. This is the most computationally intensive process in VICTORIOUS with a time complexity of $O\left(n^d\right)$, where $n$ is the average number of sentences within documents and $d$ is the number of documents uploaded.

Similarly, through the second modality of this module, also based on a cosine similarity metric, users can direct the module to compile the list of semantically similar sentences from across all the uploaded documents. Instead of sentences, users can also perform a semantic search with keywords in both modalities.

Sentences are visually distinguishable upon mouse-over. This visual cue suggests intractability. Upon selecting a sentence, a semantic search request is sent to API and a compiled list of related sentences is returned. Using the sidebar, users can choose between two modes of semantic search: (1) within the selected document or (2) across all uploaded documents. Also, using the sidebar, keyword search can be activated.

This lens supports non-linear reading of documents. By identifying semantic units from other parts of a given document or other documents, users can quickly drill down into a document and assess it with regards to a specific search query. Also, by finding these semantic units of interest, this lens can shift the emphasis to parts of a document that may not be properly identified in less structured literature.

3.3.5. Bibliographic Connections (BiblioConnections)

The Bibliographic Connections (BiblioConnections) lens allows charting the network of citations within the documents. Once activated, this module can be used as a lens over all the documents to show any citations to or from the documents that are uploaded, as displayed in Figure 14.

When a document is selected under BiblioConnections, as seen in Figure 15, the connections of this document are highlighted and others are dimmed.

Once the documents are uploaded, bibliographic connections are extracted using Grobid. These connections are then stored in a metadata file that is used in the module.

This lens helps in identifying the bibliographic and logical connections that exist among the uploaded documents. Most/least cited and/or most/least used documents can be identified using this module. Also, foundational works within a body of knowledge can be quickly identified by a quick examination of the bibliographic connections. When used in conjunction with the sorting options offered in Initial Screen, the efficiency of usage is further increased, given the ordering of documents based on citations.

3.3.6. Compare

The Compare lens provides parallel reading of and cross-examination of queries in two documents simultaneously. By selecting two documents from Initial Screen (Figure 16), a new window is opened (Figure 17), within which the selected documents are restructured and put alongside each other for a more in-depth analysis.

Within this new window (Figure 17), users have four main options: (1) reading content of the documents side by side; (2) searching for a query statement/keyword within both and juxtaposing results for quick comparison, similar to SemJump; (3) juxtaposing semantically similar sentences in both documents, similar to Skim; and (4) comparing the keywords and word clouds of both documents, similar to Map.

Like Skim, this module affords in-depth analysis of the textual content and the hypotheses that are already formed or being formed (Figure 18). Also, users can restructure documents, like SemJump (Figure 17).

Unlike the other lenses, the Compare lens draws from both the meta-data that are extracted from documents in upload time, and the queries that can be searched for in run-time from within them. This module supports drilling down on and comparing the documents. In this way, it assists the task of scoping review.

4. Usage Scenarios

In this section, we describe usage scenarios to demonstrate how VICTORIOUS can assist researchers in handling their research tasks involving a high number of documents. We qualitatively examine the functionality of VICTORIOUS under three different scenarios, in three different research domains. In each of the scenarios, we assume one researcher as the user of VICTORIOUS who has cursory knowledge in the domain and has gathered a large body of documents, i.e., published articles in the given domain.

Each scenario focuses on some aspects of VICTORIOUS. Focal points are not necessarily mutually exclusive and can overlap. In each scenario, the user starts by uploading the curated document set to VICTORIOUS. In the following subsections, we identify the scope of each scenario first, followed by the pursued goals. Afterwards, we provide more details on the usage of VICTORIOUS through its interactions and lenses. For brevity, routine procedures, e.g., uploading and module switching, are not explained. We report metrics related to document clustering of each scenario in

Table 6. DPP clustering metrics summary of usage scenarios.

Scenario	Document Count	Topics Identified (K in K-Means)	Reduced Dimensions Count (PCA)	K: Silhouette Score Per K
A (Section 4.1)	112	2	40	2: 0.21172 3: 0.11393 4: 0.10214 5: 0.09852 6: 0.10178
B (Section 4.2)	115	3	94	2: 0.06488 3: 0.06674 4: 0.06490 5: 0.06032 6: 0.05822
C (Section 4.3)	40	3	28	2: 0.05757 3: 0.11113 4: 0.10935 5: 0.10833 6: 0.06936

Bold and underlined indicates the best Silhouette score, and by extension, the best K value.

.

4.1. A: Trends in AI Chatbots

By running search queries on IEEE Xplore digital library for conference papers and journal articles containing “chatbot” in their titles and published from 2008 to 2021, we gather a collection of 112 documents and downloaded associated PDF files. We are interested in answering the following questions based on this collection:

Q1.

How popular have chatbots been in scholarly studies?

Q2.

What are the existing definitions proposed for chatbots?

Q3.

What are the main differences among the publications related to chatbots?

Q4.

Who are the notable researchers in this domain, and how different are their perspectives about chatbots?

As seen in Figure 19, VICTORIOUS identifies two main topics within these 112 documents: (1) “chatbot using”, which contains documents related to the implementation of a chatbot in a specific language or an area, and (2) “chatbot”, which contains documents on the design considerations, challenges, methodologies, analysis, and reviews of chatbots. This demonstrates that the collected documents cover similar topics and the clustering component in DPP has only recognized two clusters, addressing Q3. We report metrics related to document clustering of each scenario in Table 6. We report Silhouette scores for each number of topics suggested in DPP. In this scenario, the best candidate for the number of clusters found in the documents is 2.

We also discover that a higher number of documents were published in the period of 2019–2021 than the period of 2008–2019, which shows the increase in attention towards this domain in the latter part of this period, addressing Q1.

One common approach to identify key documents and authors is to rely on the bibliographic connections. As shown in Figure 20, The BiblioConnections module helps to discover foundational works by visually identifying the most cited documents. We order documents based on their received citations and pick the top documents for further analysis, addressing Q4. Then, we use the Skim module to drill down on foundational documents and obtain definitional concepts from them, as shown in Figure 21, addressing Q2.

4.2. B: Evolution of AI-Adjacent Research in Recent CHI Proceedings

For this scenario, we focus on AI within Human–Computer Interaction (HCI) circles and want to understand the recent trends of AI-adjacent discourse in this domain. We select 115 published papers from the proceedings of the Association for Computing Machinery’s (ACM) Conference on Human Factors in Computing Systems (CHI) in the 2020–2024 period. These are a subset of all published papers related to AI, published under various sessions of the conference over this period. Specifically, we are interested in addressing the following questions:

Q1.

What are the main themes covered in the documents?

Q2.

What is the contribution of influential works to each respective theme?

Q3.

What has been the application of AI in healthcare-related domains?

Q4.

How has the subject of “fairness” in predictive analysis been taken into consideration within healthcare-related applications of AI?

We observe Initial Screen with three main columns, as shown in Figure 22, indicating three main themes among the topics: “generative”, “conversational”, and “ai”. These topics are derived based on the best Silhouette score reported in Table 6. By selecting one of the themes, we can drill down the individual keywords that form each theme. We hypothesize that the main difference among the documents is their approach to AI and its applications. Here, VICTORIOUS has identified a group of documents that focus on generative AI, another on conversational AI, and yet a third category involving keywords like “decision making”, “automation”, “transparency”, and “fairness”.

Similar to 4.1, to address Q2, we use the BiblioConnections module. This results in Figure 23. Using these bibliographic connections, we select one of the influential works in the “generative” (green) and another document from “conversational” (orange) that cites the green document.

We now drill down on both documents by activating the Map module on them. The result is shown in Figure 24. Here, we see the metadata on each document, including authors, the number of references used, relative recency of the documents, venue of publication, and, most importantly, the contribution of each to their respective themes in the form of a word cloud.

By using the Map module, we further drill down on documents to find the ones that are healthcare-related. In searching for such papers under the theme of “ai”, we find one that is titled “A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment”. We drill down on this document using the Summary module to read its abstract, as shown in Figure 25. Also, by using the Skim module on the same document, we can skim through parts of the document and get a better grasp of its relative worth in the review process, as shown in Figure 26. Thus, we address Q3.

To address Q4, we consider the previously found paper. We then use the Map module to find another document related to “fairness”. Then, we use the Compare module to juxtapose these documents and perform a search using the term “predictive analysis” within them. The results are visible in Figure 27.

4.3. C: Neighborhood Attention Transformers

Neighborhood Attention Transformers (NATs) are a new and trending architecture for neural networks that are primarily designed to improve efficiency of the attention mechanism in processing sequential data by transformers. Their primary use is in vision applications. In this scenario, we use the Connected Papers [66] website to look for papers related to NATs. We query the website with this title and receive a list of 40 references. We then retrieve these 40 PDF files from their publishers to use them in VICTORIOUS. In reviewing these documents via VICTORIOUS, we are mainly interested in addressing the following questions:

Q1.

How can the ideas used in each paper be traced back to an original source?

Q2.

How do Vision Transformers (ViTs) use the self-attention mechanism?

Q3.

Which documents harness self-attention in their proposed architectures?

To begin with, we examine the keywords created for each theme, as shown in Figure 28. We observe that “self attention” and “local self attention” appear in the orange theme. As such, we theorize that documents related to the self-attention mechanism will be found under this theme. By looking at the clustering metrics reported at Table 6, we observe that the best Silhouette score is obtained when the documents are clustered into three topics.

We switch to the BiblioConnections lens and find the most influential document in the orange column, as displayed in Figure 29. Next, we drill down on the most influential paper in this collection to examine its content and see if the hypothesis we made earlier holds true, as shown in Figure 30.

As such, we observe that this paper discusses the self-attention mechanism, and, thus, we validate our hypothesis, addressing Q1 and Q2. Once we gain sufficient grasp of the content of this work, we switch to the SemJump module and use it to find semantically relevant documents and address Q3. This helps us in identifying relevant parts of these papers, without complete familiarity with the internal structure of these documents. Also, we observe that the usage of SemJump module in this scenario resulted in a much faster response than other scenarios. The result of these steps is shown in Figure 31 and Figure 32.

A complete and formal usability assessment of VICTORIOUS lies beyond the scope of this paper. Instead, we propose a quasi-experimental mixed methods study to evaluate its effectiveness in supporting users during scoping reviews. This approach would leverage VICTORIOUS’s six distinct modules, assigning different combinations to user groups to examine their impact on performance and comprehension through both quantitative and qualitative measures. Specifically, we recommend allocating the Map, Bibliographic Connections, and SemJump modules to one group, and the Summary, Skim, and Compare modules to another. This design enables an evaluation of how each module set enhances users’ understanding of the research question and the broader research context. Alternative module combinations could also be explored to investigate their individual and combined effects. Furthermore, VICTORIOUS could be integrated with another system to facilitate a comprehensive systematic review, with module assignments tailored to the functionalities of the combined system. To assess the intervention’s effectiveness, users’ comprehension of the literature and the research question should be measured before and after engaging with the assigned modules.

5. Summary and Conclusions

In this paper, we have discussed what users need when making sense of a curated document set in the context of scoping reviews. We analyzed the steps needed in a scoping review process and presented a sample of existing systems that researchers use for performing such reviews. To support the scoping review process, we drew on the latest research to develop a set of visual analytics modules. Through a review of the existing literature, we identified five iterative steps necessary for any given scoping review process. The steps included (1) identifying and fine-tuning the research question, (2) identifying relevant documents, (3) selecting documents that are to be examined further, (4) making sense of and classifying the documents, and (5) summarizing the results.

We noted that although document management systems can provide support for curating an initial set of documents, they focus either on extracting archive-level, aggregate analytics or document-level, particular insights from the documents. This gap within existing systems leads to a lack of support for (1) contextualization while maintaining focus on the research question, or (2) fine-tuning the research question while maintaining the overall context of the research domain.

To address these shortcomings, we have developed VICTORIOUS (a VIsual analytiCs sysTem for scOping RevIew of dOcUment Sets). To support the five steps discussed above, we designed and incorporated six modules in VICTORIOUS. Users upload their documents to VICTORIOUS. Then, VICTORIOUS presents a conceptual framework, in the form of a topic model extracted from the documents. After this, VICTORIOUS’s six modules assist the iterative process of a scoping review of the uploaded documents.

With the use of a Focus + Context design principle, VICTORIOUS displays aggregate-level information in the Initial Screen. This screen acts as a conceptual frame and further prompts the user to drill down on the provided information and use a combination of modules to further the task of scoping review, hence putting the user in the sensemaking loop.

To demonstrate the utility and effectiveness of VICTORIOUS and its modules, we presented three usage scenarios. We noted that displaying all documents together supported with a fisheye view can enable researchers to discover specific patterns within the document set with minimum interactions. Furthermore, interactively expanded modules can enable researchers to access latent layers of information within scientific documents when required, without noticeable added time complexity. In this way, we managed to both avoid high density in our visualizations and to utilize a mechanism to encode more information items in our visualizations.

We observed that a large enough (more than 110 documents) collection can have negative impacts on the responsiveness of the SemJump module. This is because through one of its modalities, a sentence from one document must be compared with sentences from all other documents. Given the algorithmic complexity of cross-comparing sentences across all documents, saving the results as a pre-processing step can improve user experience. Also, training a language model on the uploaded document set and using it as a sentence cross-comparator at runtime seems a promising upgrade to overcome the observed computational limitation.

Given VICTORIOUS’s standalone architecture, on-site storage, and lack of dependence on external AI models/APIs, ethical considerations on data ownership or losing data confidentiality are not pertinent to this work. Likewise, the automated analysis methods used in VICTORIOUS are not generative and are only used as conceptual guides. Hence, they are not conducive to compromises on academic integrity or plagiarism. In addition, because of the iterative nature of scoping reviews, VICTORIOUS’s various modules prompt the user to constantly cross-check various hypotheses against the underlying documents themselves, maximizing user oversight and mitigating possible biases of the system. Existing approaches focus on singular aspects of each document to reveal them. They do not enable their users to aggregate singular information items from each document. Covid-on-the-Web, ISTEX, and PaperMage, as examples reviewed in Section 2, offer only a limited window to the underlying documents, leaving their users in need of further analysis to conclude the scoping review process. In comparison, our scenarios indicate that, by utilizing a dynamic knowledge extraction methodology and a multi-resolution sensemaking approach, VICTORIOUS holds promise in guiding and assisting researchers in a scoping review process.

VICTORIOUS starts its DPP by direct user upload. Integration with other reference and/or library management software (e.g., Zotero or Mendeley) is beyond the scope of this paper. For future improvements, such integrations could be considered. This might improve its usability, as importing documents to/from these widely used software programs can open the way to widespread uses of VICTORIOUS. Similarly, exploration of various machine learning methods and AI benchmarks on the outcome of the modules is also beyond the scope of this work. To overcome some of the shortcomings of the system, as future research, further studies can be conducted on novel, more efficient methods of gathering semantically similar units as part of the SemJump module.