Topic Modeling for Faster Literature Screening Using Transformer-Based Embeddings

Abstract

Systematic reviews are a powerful tool to summarize the existing evidence in medical literature. However, identifying relevant articles is difficult, and this typically involves structured searches with keyword-based strategies, followed by the painstaking manual selection of relevant evidence. A.I. may help investigators, for example, through topic modeling, i.e., algorithms that can understand the content of a text. We applied BERTopic, a transformer-based topic-modeling algorithm, to two datasets consisting of 6137 and 5309 articles, respectively, used in recently published systematic reviews on peri-implantitis and bone regeneration. We extracted the title of each article, encoded it into embeddings, and input it into BERTopic, which then rapidly identified 14 and 22 topic clusters, respectively, and it automatically created labels describing the content of these groups based on their semantics. For both datasets, BERTopic uncovered a variable number of articles unrelated to the query, which accounted for up to 30% of the dataset—achieving a sensitivity of up to 0.79 and a specificity of at least 0.99. These articles could have been discarded from the screening, reducing the workload of investigators. Our results suggest that adding a topic-modeling step to the screening process could potentially save working hours for researchers involved in systematic reviews of the literature.

1. Introduction

The advancement of technology and the internet have ushered in a new era of information sharing that is significantly transforming how research is conducted at every stage [1]. Scholars and researchers can now effortlessly publish a vast array of data, reviews, and opinions across globally accessible platforms through new publishing models that simplify and accelerate the sharing of data and knowledge. However, publishing not only serves to disseminate knowledge for furthering research but also involves complex dynamics related to career progression and securing grant funding [2,3]. As a result, an overwhelming number of publications covering diverse topics are published every day, making it challenging to efficiently identify relevant papers amidst a sea of sometimes only tangentially related literature [4]. The challenge becomes particularly critical when conducting systematic reviews.

Systematic reviews are rigorous and comprehensive examinations of the existing literature to answer specific research questions [5] that rely on experimental—and sometimes observational—studies, with the purpose of gathering all the existing evidence on a given medical condition and, usually, its therapy. To conduct a successful systematic review, researchers must consider a wide range of sources and databases to ensure comprehensive coverage of the relevant literature [5,6] and to make sure to identify all the pertinent data, minimizing the risk of missing relevant evidence. For this purpose, researchers usually rely on established library repositories to search for scientific articles [7]. These databases, such as Medline, are commonly searched using specific keywords that may appear in the article, e.g., in its title or abstract. Though this approach is fast and robust, it may fail to capture relevant articles if they use different wordings or synonyms [8]. It has long been recognized that multiple searches with complex syntax, which requires advanced query construction skills, are often needed to narrow down the search space [9]. Moreover, as the lexicon may be ambiguous, this approach usually yields large numbers of publications that do not match the exact focus of the query—or are even off topic—and forces researchers to manually sift through the query results to handpick the papers they need [10]. Considerable research has previously focused on developing and testing search filters designed to refine results by targeting specific criteria [11,12,13,14,15].

Automation has great potential to improve literature searches and expedite systematic reviews [16]. Recent advancements in natural language processing (NLP) and machine learning have demonstrated the possibility to automate or assist several tasks within the systematic review process [17,18,19]. Innovations in this area have led to the development of software like Abstractr, ASReviews, EPPI-reviewer, and RobotSearch, which utilizes convolutional neural network architectures to identify RCTs [20,21]. Large language models are promising tools to automate some aspects of systematic reviews by enhancing literature retrieval through semantic understanding and the contextual analysis of search terms [22,23]. An essential component for achieving semantic understanding is embeddings—numerical representations that encode word or even sentence meaning—which are key in NLP for capturing complex relationships between words and sentences using special architectures known as transformers [24]. Embeddings based on transformer architectures have proven very effective in several NLP tasks, and many efforts have been devoted to developing better embeddings for specific tasks, also in the biomedical field, including topic modeling [25].

Topic modeling involves identifying the main theme of unlabeled documents [26]. This approach can be valuable in understanding complex scientific literature corpora by automatically organizing and categorizing large sets of publications based on their topics [27]. One of the latest examples among the topic-modeling algorithms is BERTopic—an advanced algorithm developed by Grootendorst in 2022—which harnesses the power of the embeddings obtained from BERT, a well-known transformer architecture [28]. BERTopic can segment a dataset of text documents—an operation commonly known as clustering—by their semantic content and extract a series of representative keywords, which BERTopic then concatenates to create a topic label for each cluster [29]. Compared to previous topic-modeling algorithms, transformer-based architectures such as BERTopic capture contextual meanings much more effectively, especially on smaller amounts of data, which, in turn, results in more coherent and interpretable topics [30]. These new methods are being increasingly applied to the analysis of scientific literature [31,32,33,34].

Based on the available data in the literature, we assumed that topic modeling could also be used as an intermediate step within the literature screening process to filter out undesired papers that may have been retrieved with a conventional keyword-based search so to clean the dataset before investigators manually screen it. To this purpose, algorithms providing a contextual interpretation of words and sentences appeared as the only viable option to improve the results of keyword-based searches, because they might be able to correctly interpret the meaning of ambiguous expressions depending on their context of use. BERTopic, therefore, appeared as a valuable choice to pursue this goal.

The main aim of the present paper is thus to explore how BERTopic can be applied effectively on datasets of the scientific literature and identify relevant papers for systematic reviews. To do that, we used two datasets that were recently used for two published systematic reviews in the dental field. We analyzed them separately with BERTopic, identified topic clusters within these two corpora, and investigated whether individual non-relevant topic clusters could have been discarded from the screening without negatively affecting the outcome of the review.

The approach we propose might prove helpful in filtering out non relevant articles from further assessment, thereby enhancing the speed and efficiency of the search process. While topic modeling has been applied in various fields, to the best of our knowledge, our approach is unique in leveraging BERTopic as an intermediate step to segment scientific datasets for systematic reviews in the biomedical (dental) field and filter out irrelevant papers retrieved by conventional keyword-based searches.

2. Materials and Methods

2.1. Datasets

Our analysis focused on two separate datasets that had been previously used to identify articles of interest (henceforth, “target articles”) for two published systematic reviews in the dental field [35,36]. The two datasets contained a list of scientific articles on peri-implantitis and bone regeneration, respectively. The authors of the systematic reviews had created these datasets through specific searches conducted across literature databases, and comprised bibliographic information about the articles, including authors, title, abstract, journal with publication date, plus keywords.

The first dataset included 6137 articles on the treatment of peri-implantitis and was generated through a state of the art and well-detailed keyword-based search across several databases, including Medline and Embase [36]. In that systematic review, the investigators eventually identified 24 target articles that answered the following focused questions (FQ):

FQ1: In patients with peri-implantitis, what is the efficacy of different bone reconstructive therapies compared to access flap surgery (AFS) in terms of pocket reduction and change in bleeding and suppuration on probing (BOP and SOP), at a minimum of 12 months of follow-up?

FQ2: In patients with peri-implantitis, what is the long-term (≥12 months) performance of reconstructive therapies in terms of pocket reduction and change in BOP/SOP?

The second database had been used as a basis for another published systematic review by Calciolari et al. on bone augmentation techniques [35]. The dataset used for this work comprised 5309 articles obtained through a systematic literature search to address the following FQs:

FQ1: In patients receiving GBR simultaneous to implant placement, what is the impact of biomaterials (membranes, grafts, bioactive factors) on the stability of peri-implant bone levels as assessed through 2D or 3D radiographs in RCTs/CCTs with ≥ 12 months of follow-up?

FQ2: In patients receiving GBR simultaneous to implant placement, what is the impact of biomaterials (membranes, grafts, bioactive factors) on bone defect dimension (width and/or height) changes as evaluated at re-assessment procedures performed at ≥4 months post GBR in RCTs/CCTs?

We manually screened these two datasets, searching for off-topic articles (henceforth labelled OffTA). We adopted a broad definition of OffTAs as those articles that did not focus on dentistry, e.g., penile prostheses or breast implants. Pre-clinical studies were not considered OffTAs unless they were investigating areas that were clearly related to fields other than dentistry. So, for instance, a report on cellular behavior in an in vitro setting would not necessarily be considered an OffTA, but a pre-clinical investigation on a fracture model in rodent would. All the papers that investigated areas of dentistry (or applicable to dentistry) were considered on-topic articles (OnTAs) for both datasets. It may be argued that when it comes to peri-implantitis, orthopedic implants may be closer to dental implants than, e.g., orthognatic surgery (which is related to dentistry), or that orthopedic research articles may be more relevant to bone regeneration of the alveolar ridges than many dental-related research areas, but as the goal of our investigation was to filter out papers from the dataset to make systematic reviews faster, we worked under the assumption that it would be safer to discard articles that focused on different clinical areas than dentistry, and that would not increase the risk of losing important pieces of evidence for both datasets.

Upon inspection, the first dataset was composed of 3810 OnTAs (i.e., dentistry-related) and 2327 OffTAs (38%), while the second dataset appeared to include only 814 OffTAs or 15% of the total.

2.2. Purpose of the Study

The purpose of the study was to investigate whether running BERTopic, a topic-modeling algorithm, on these two datasets to segment them into topic clusters could make subsequent screening faster, by identifying groups of articles constituted only (or prevalently) of OffTAs—henceforth designated as off-topic groups or OffTGs—that could be safely discarded to narrow down the corpus.

2.3. Data Analysis

The data were analyzed using Google Colab Pro notebook powered by Python 3.10.12 [37] and running on T4 GPUs [38], which provide the acceleration required to handle embeddings efficiently.

The analysis of the publications was conducted on their titles, based on the assumption that titles are a summary of the content of a paper and are thus representative of their topic [39,40]. The datasets did not need to undergo any preprocessing other than removing entries when titles were missing. Unlike previous publications [41], we did not deem it necessary to lowercase the titles, nor to remove stopwords, to rely on BERTopic’s capability to produce contextual embeddings. Unlike bag-of-words approaches, there is a consensus that stopwords may actually improve sentence encoding by providing further context using transformer architectures [42].

Embeddings in NLP are dense vectors that represent the semantics of words in a multidimensional space [43]. Unlike older algorithms [44], bidirectional encoder representations from transformers (BERT) understands the context and creates unique word embeddings based on their usage in different contexts [24,45]. BERTopic operates through several stages, including transformer embedding models, dimensionality reduction, clustering, and cluster tagging using cTF-IDF [28], which are summarized in Figure 1.

The first step is the creation of the embeddings. To carry that out, we chose the Huggingface’s ‘all-mpnet-base-v2’ model.

The embeddings from this model are too large for efficient clustering and must therefore be reduced. As several dimensionality reduction algorithms are available, we used uniform manifold approximation and projection (UMAP), which has been shown to be very effective in preserving the topological structure of data [46]. We empirically decided to reduce all-mpnet-base-v2’s 768-dimension embeddings to 5-dimension embeddings for subsequent processing and to 2-dimensions for visualization. The reduced embeddings were then clustered with hierarchical density-based spatial clustering of applications with noise (HDBSCAN) [47], and cTf-Idf was applied to extract topic keywords in each cluster. Unlike Tf-Idf [48], cTf-Idf adjusts the weight based on the term frequency within a cluster of documents rather than within an individual document [49].

More specifically, we decided to use the following set-up:

−

UMAP metric: cosine distance (default setting);

−

size of the neighborhood: 15;

−

number of components: 5;

−

HDBSCAN clustering metric: Euclidean (default setting);

−

minimum cluster size: 50.

The number of components was determined empirically as a compromise between preserving the richness of information of the original embedding and making it easier for HDBSCAN to cluster them. The settings for the size of the neighborhood and the minimum cluster size were determined empirically through a grid search, as described in Figure 2. To enhance processing speed, we used the cuML GPU-based implementation of UMAP and HDBSCAN [50]. In addition to BERTopic’s default representation model, we adopted KeyBERT, a more recent algorithm, to improve keyword extraction [51].

BERTopic output is not always straightforward. The way BERTopic labels a topic is by taking the main 4 keywords that describe it and joining them together. This default labeling is efficient but its result is admittedly often obscure. Fortunately, BERTopic allows the integration of other algorithms to create better representations of topics, including large language models (LLM). An LLM is an A.I. algorithm that can generate human-like responses [52] and can create more comprehensive and representative descriptions of topics, i.e., better labels. We decided to use the freely available OpenHermes-2.5-Mistral large language model [53]. LLMs are typically—as their name suggests—very large and require vast computer resources, thus reduced versions have been elaborated [54] that are commonly referred to as quantized LLMs [55]. We opted for the OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q4_K_M.gguf quantization, available for download on Huggingface.com (accessed on 3 March 2024).

LLMs need a prompt from the users [56] to generate a response, and we set the following prompt:

““Q:

I have a topic that contains the following documents:

[DOCUMENTS]

The topic is described by the following keywords: ‘[KEYWORDS]’.

Based on the above information, can you give a short label of the topic of at most 5 words?

A:

“““

So, to briefly summarize it, BERTopic worked as we described by clustering the embeddings and a series of keywords that describe these clusters. The LLM then took the keywords of each topic and generated a sentence that described and captured the essence of these keywords. So eventually, every cluster of documents had a little sentence label as a descriptor, which was much more readable and immediate to human users.

We used BERTopic’s inbuilt functions and the matplotlib [57] and seaborn libraries [58] for data visualization. The Datamapplot library [59] was used for effective cluster visualization.

To measure the performance of the algorithm, we used specificity, sensitivity, and F1 scores. Sensitivity measures the proportion of actual positives that are correctly identified by a test and is usually calculated as

$$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{P}\right)}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{P}\right)+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{F}\mathrm{N}\right)}}$$

Specificity measures the proportion of actual negatives that are correctly identified and is calculated as follows:

$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{N}\right)}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{N}\right)+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{F}\mathrm{P}\right)}}$$

The F1 score is the harmonic mean of precision and sensitivity, also commonly referred to as recall, in this context. If precision is calculated as

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{P}\right)}{\mathrm{T}\mathrm{r}\mathrm{u}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{T}\mathrm{P}\right)+\mathrm{F}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{s} \left(\mathrm{F}\mathrm{P}\right)}},$$

then F1 score is calculated as

$$\mathrm{F}1\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$$

To this purpose, we considered OffTAs as positive and OnTAs as negative. OffTAs clustered within a OffTG were considered true positive, while OnTAs clustered in an OffTG were considered false positives.

3. Results and Discussion

3.1. Peri-Implantitis Dataset Analysis

Running BERTopic on a Colab notebook, after loading the necessary libraries and packages, required a few seconds for such a small dataset. BERTopic successfully identified several topics based on the dataset titles, but the exact number of topics varied depending on algorithm parameters. The main steps that users can control in BERTopic are summarized in Figure 1:

(1)

Creating the embeddings,

(2)

Reducing the embeddings’ dimensions.

(3)

Clustering the embeddings,

(4)

Labelling the embeddings.

To create sentence embeddings from the titles, we decided to use the all-mpnet-base-v2 model, which has been pre-trained on a large dataset and has proven to be effective also on academic titles [60]. While slower than smaller models, this did not significantly impact computation time with a dataset of just over 6000 titles.

To improve clustering efficiency, we reduced the dimensions of our initial embeddings using UMAP. This algorithm allows for a thorough customization of its parameters, including the granularity of the topological structure that it aims to preserve during dimensionality reduction through the “number of neighbors” parameter. A major advantage of HDBSCAN is that it does not require pre-determining the number of clusters (e.g., unlike K-means algorithms), although, as a result, it tends to cluster unclear documents in a null group, which is identified by the −1 label. HDBSCAN can be customized through several parameters too, including the minimum acceptable size for a cluster. The last step consists of finding a label for each cluster that corresponds to its topic. For that purpose, we have chosen two representation models: a large language model, to create human-like labels, and KeyBERT, to generate keywords that are characteristic to the topic. Figure 2 shows that by reducing the minimum size of the clusters, BERTopic was able to find more topics in our dataset. This is expected, as niche topics consisting of only a few articles may be overlooked if the threshold is too high. At the same time, changing the sensitivity of UMAP through its number of neighbors parameter altered the slope of the curve. The choice of the best settings is a subjective decision, which depends on the purpose of the investigators; in our case, we wanted to have enough granularity to isolate unrelated topics while keeping the number of topics small enough to be easily manageable and several orders of magnitude smaller than the dataset itself to make it a convenient step in the workflow. Therefore, we empirically determined the optimal number of neighbors to be 15 and the minimum cluster size to be 50. These settings generated 14 topics (

Table 1. The list of topics identified by BERTopic, in order of size, for the peri-implantitis dataset. Non-dental topics are highlighted in bold. The full list can be found as Supplementary Material (Table S1).

Topic	Count	LLM
−1	357	Treating Implant Infections
0	3733	Peri-Implant Bone Study
1	587	Valves and Stents in Coronary Arteries
2	232	Intraocular Lens Inflammation
3	192	Breast Reconstruction and Implants
4	174	Parkinson’s Disease and Deep Brain Stimulation
5	144	Sinus Floor Elevation
6	137	Photodynamic Therapy for Peri-implantitis Treatment
7	132	Implant-retained Mandibular Overdentures
8	127	Orbital Reconstruction Implants
9	96	Cochlear Implantation
10	89	Cervical Fusion and Disc Disease
11	76	Serous Borderline Ovary Tumors.
12	61	Zirconia Implants and Abutments

). Table 1 lists the topics identified by the algorithm, from the biggest one to the smallest one. The topic list includes the ‘−1’ unclassified documents cluster, where all the unclassified papers are supposed be allocated.

Interestingly, the algorithm agreed that the non-classified (−1 cluster) articles were still mostly centered on implants (and implant infections) and assigned them the label “Treating Implant Infections”. This null topic cluster was quite small (n = 357). As expected, BERTopic identified several topics related to peri-implantitis (e.g., topic #0 which included the majority of the papers (n = 3733) or topic #6) but also more broadly related to implant dentistry (e.g., topics #5, #7, and #12).

However, this dataset also included at least 8 completely unrelated OffTGs, some of them quite conspicuous in size, such as the following:

#1 Valves and Stents in Coronary Arteries (n = 587), e.g., “Carotid-subclavian bypass grafting with polytetrafluoroethylene grafts for symptomatic subclavian artery stenosis or occlusion: a 20-year experience” [61];

#2 Intraocular Lens Inflammation (n = 232), e.g., “Double-masked, placebo-controlled evaluation of loteprednol etabonate 0.5 for postoperative inflammation” [62];

#5 Parkinson’s Disease and Deep Brain Stimulation (n = 174), e.g., “Three-dimensional space fluid-attenuated inversion recovery at t to improve subthalamic nucleus lead placement for deep brain stimulation in Parkinson’s disease: from preclinical to clinical studies” [63];

#16 Cochlear Implantation (n = 96), e.g., “Online support group users’ perceptions and experiences of bone-anchored hearing aids (bahas): a qualitative study” [64].

A closer examination of the topic list suggests that more topics could be considered unrelated to the query, albeit dental-related. Some of them are small niche topics, such as #12 Zirconia Implants and Abutments (n = 61), some are larger, such as topic #5 Sinus Floor Elevation (n = 144); although it is thematically related to implants, nothing in its keyword descriptors mentions peri-implant disease:

[‘sinus floor’, ‘sinus elevation’, ‘osteotome sinus’, ‘sinus augmentation’, ‘sinus surgery’, ‘maxillary sinus’, ‘sinus implants’, ‘sinus lift’, ‘transcrestal sinus’, ‘eluting sinus’]

Similarly to identifying OffTAs, the decision to discard OffTGs based on their descriptors is subjective, and there is a degree of risk in discarding dentistry- or implant-related topics. The domain knowledge of the investigator is likely the main factor affecting where to place the threshold, i.e., to decide which topics to retain and which ones to discard. The algorithm we employed makes no assumption on the relevance of the identified topics but merely segments the dataset into groups and labels them. It is up to the investigators to decide whether a topic is relevant to the theme of the query. We adopted a cautious approach by retaining all dental-related topics, and we thus decided to discard only those topics that were unrelated to dentistry as a whole to avoid risking losing relevant articles.

To better understand how the titles of the dataset were semantically distributed, we reduced the embedding dimensionality that we used for topic modeling from 5 down to 2 dimensions, so that each title could be represented as a data point in a scatter plot (Figure 3). Closer points correspond to articles whose titles have closer meaning and, therefore. topic, while farther points represent articles that belong to less closely related topics, including frank OffTAs. As can be easily noticed, the conceptual space for this dataset of scientific publications is not homogeneous, but there are several areas of higher density, which constitute the individual topics, and some topics appear isolated. Unsurprisingly, OffTAs tend to be distributed peripherally, as a satellite constellation of articles with looser association to the core of the dataset, which appears in the middle of the plot and includes most of the dental-related articles (Figure 3).

Even when considering only the topics highlighted in Table 1 that are grossly unrelated to the purpose of the systematic review (i.e., unrelated to dentistry in general), they alone account for about 30% of the papers in the dataset (or 1856 papers out of the 6137 titles that had to be manually screened at the time the systematic review was performed).

As this dataset has already been published, we also already knew which articles had been selected for the review. As

Table 2. The list of the target articles identified in Donos et al. systematic review on peri-implantitis [36]. No article was clustered in any of the unrelated topic groups.

Authors	Topic	Reference
Andersen, Heidi, Aass, Anne Merete and Wohlfahrt, Johan Caspar	#0 Peri-Implant Bone Study	[65]
Jepsen, K., Jepsen, S., Laine, M. L., Anssari Moin, D., Pilloni, A., Zeza, B., Sanz, M., Ortiz-Vigon, A., Roos-Jansaker, A. M. and Renvert, S.	#0 Peri-Implant Bone Study	[66]
Wohlfahrt, Johan Caspar, Lyngstadaas, Stale Petter, Ronold, Hans Jacob, Saxegaard, Erik, Ellingsen, Jan Eirik, Karlsson, Stig and Aass, Anne Merete	#0 Peri-Implant Bone Study	[67]
Emanuel, Noam, Machtei, Eli E., Reichart, Malka and Shapira, Lior	#0 Peri-Implant Bone Study	[68]
Renvert, Stefan, Giovannoli, Jean-Louis, Roos-Jansaker, Ann-Marie and Rinke, Sven	#0 Peri-Implant Bone Study	[69]
Isehed, C., Holmlund, A., Renvert, S., Svenson, B., Johansson, I. and Lundberg, P.	#0 Peri-Implant Bone Study	[70]
Isehed, C., Svenson, B., Lundberg, P. and Holmlund, A.	#0 Peri-Implant Bone Study	[71]
Renvert, Stefan, Roos-Jansaker, Ann-Marie and Persson, Gosta Rutger	#0 Peri-Implant Bone Study	[72]
Nct	#0 Peri-Implant Bone Study	[73]
Froum, Stuart J., Froum, Scott H. and Rosen, Paul S.	#0 Peri-Implant Bone Study	[74]
Gonzalez Regueiro, Iria, Martinez Rodriguez, Natalia, Barona Dorado, Cristina, Sanz-Sanchez, Ignacio, Montero, Eduardo, Ata-Ali, Javier, Duarte, Fernando and Martinez-Gonzalez, Jose Maria	#0 Peri-Implant Bone Study	[75]
Isler, S.C., Soysal, F., Ceyhanli, T., Bakirarar, B. and Unsal, B.	#0 Peri-Implant Bone Study	[76]
La Monaca, Gerardo, Pranno, Nicola, Annibali, Susanna, Cristalli, Maria Paola and Polimeni, Antonella	#0 Peri-Implant Bone Study	[77]
Mercado, Faustino, Hamlet, Stephen and Ivanovski, Saso	#0 Peri-Implant Bone Study	[78]
Polymeri, Angeliki, Anssari-Moin, David, van der Horst, Joyce, Wismeijer, Daniel, Laine, Marja L. and Loos, Bruno G.	#0 Peri-Implant Bone Study	[79]
Roccuzzo, Mario, Gaudioso, Luigi, Lungo, Marco and Dalmasso, Paola	#0 Peri-Implant Bone Study	[80]
Roccuzzo, Mario, Mirra, Davide, Pittoni, Dario, Ramieri, Guglielmo and Roccuzzo, Andrea	#0 Peri-Implant Bone Study	[81]
Isrctn	#0 Peri-Implant Bone Study	[82]
Aghazadeh, A., Rutger Persson, G. and Renvert, S.	#0 Peri-Implant Bone Study	[83]
Aghazadeh, A., Persson, R.G. and Renvert, S.	#0 Peri-Implant Bone Study	[84]
Nct	#6 Photodynamic Therapy for Peri-implantitis Treatment	[85]
Roos-Jansaker, Ann-Marie, Renvert, Helena, Lindahl, Christel and Renvert, Stefan	#0 Peri-Implant Bone Study	[86]
Roos-Jansaker, Ann-Marie, Lindahl, Christel, Persson, G. Rutger and Renvert, Stefan	#0 Peri-Implant Bone Study	[87]
Roos-Jansaker, Ann-Marie, Persson, Gosta Rutger, Lindahl, Christel and Renvert, Stefan	#0 Peri-Implant Bone Study	[88]

shows, the target articles that corresponded to the query were not contained in any of the OffTGs. OffTGs (and the OffTAs contained in them) removal during screening would thus not have negatively affected the review.

When examining the allocation of the target articles (Figure 4), it is apparent that the majority were allocated to the #0 group (n = 23), followed by group #6 Photodynamic Therapy for Peri-implantitis Treatment (n = 1).

This supports the idea that BERTopic working on all-mpnet-base-v2 embeddings is robust enough to discriminate not only OffTGs from dental-related topics but, within dental topics, what the peri-implantitis topics are.

We then further analyzed the dataset and found that six OnTAs were misclassified into OffTGs, specifically in topic #8 Porous Polyethylene Orbital Reconstruction; the first misclassified paper was about orthognatic surgery and so not directly related to implants:

• “Long-term evaluation of the use of coralline hydroxyapatite in orthognathic surgery” [89]

The remaining five papers focused on craniofacial surgery and thus understandably closer to orbital surgery than to dental-related topics:

• “Timing of cranial reconstruction after cranioplasty infections: are we ready for a re-thinking? A comparative analysis of delayed versus immediate cranioplasty after debridement in a series of 48 patients” [90]
• “HTR^® polymer facial implants: A five-year clinical experience” [91]
• “Gore-Tex chin implants: a review of 324 cases” [92]
• “The application of alloplastic materials for augmentation in cosmetic facial surgery” [93]
• “Japanese National Questionnaire Survey in 2018 on Complications Related to Cranial Implants in Neurosurgery” [94]

In this case, it can be argued that BERTopic did not significantly misclassify these papers and discarding them would not have jeopardized the systematic review. Additionally, we identified 469 OffTAs (i.e., unrelated to dentistry) that had not been clustered in the OffTGs but had been allocated to dental topics. This classification thus yields a specificity = 0.99, a sensitivity = 0.79, and F1 score = 0.88.

3.2. Bone Augmentation Dataset Analysis

We again applied BERTopic on the titles of the articles of the second dataset and tuned the clustering parameters as previously described to obtain an approachable number of topics. As shown in Figure 5, in general, decreasing the minimum size of the acceptable clusters in the HDBSCAN algorithm again increased the number of topics identified by BERTopic.

At the same time, adjusting the UMAP parameters allows BERTopic to better recognize local features in the distribution of the data points. We maintained the same parameters as for the first dataset, which yielded 22 topics.

Table 3. The list of topics identified by BERTopic, in order of size, for the bone augmentation dataset. Non-dental topics are highlighted in bold. The full data are found in Supplementary Table S2.

Topic	Count	LLM
−1	1515	Dental Implant Studies
0	727	Dental Implants in Edentulous Patients
1	692	Sinus Floor Elevation
2	296	Hip Arthroplasty
3	265	Alveolar Ridge Augmentation Techniques
4	225	Titanium Implants Surface Acid-Etching Osse
5	167	Ridge Preservation Bone Allograft
6	147	Bone Grafting for Dental Implants
7	146	Soft Tissue Augmentation for Dental Implants
8	140	Peri-Implantitis Treatment
9	138	Implants in Fresh Extraction Sockets
10	117	Ventricular Assist Devices
11	102	Platelet-Rich Fibrin Effects on Dental Implants
12	93	Cervical and Lumbar Fusion Studies
13	89	Bone Regeneration Recombinant Human BMP
14	81	Hydroxyapatite-coated Dental Implant Studies
15	73	Periodontal defect treatment
16	71	Collagen Membranes for Guided Bone Regeneration
17	64	Implant-retained Mandibular Overdentures
18	57	Antibiotic Prophylaxis for Dental Implants
19	54	Bone Anchored Hearing Implant
20	50	Orbital and Retinal Implants

lists the topics identified in this dataset.

The topics were visually inspected, and based on their LLM description and their keyword descriptors, we assessed that most topics were, as expected, related to bone regeneration (e.g., topic #3 Alveolar Ridge Augmentation Techniques), ridge preservation (e.g., topic #5 Ridge Preservation Bone Allograft), or otherwise implant-related (e.g., topic #9 Implants in Fresh Extraction Sockets).

However, a careful inspection of the dataset also revealed five OffTGs that appeared completely unrelated to the dental field and that altogether totaled 610 OffTAs (Table 3, bold). These topics included the following:

#2 Hip Arthroplasty (n = 296), e.g., “Timing of tibial tubercle osteotomy in two-stage revision of infected total knee arthroplasty does not affect union and reinfection rate. A systematic review” [95]

#12 Cervical and Lumbar Fusion Studies (n = 93), e.g., “A Long-Term Follow-up, Multicenter, Comparative Study of the Radiologic, and Clinical Results between a CaO-SiO2-P2O5-B2O3Bioactive Glass Ceramics (BGS-7) Intervertebral Spacer and Titanium Cage in 1-Level Posterior Lumbar Interbody Fusion” [96]

#10 Ventricular Assist Devices (n = 117), e.g., “Heart transplantation of patients with ventricular assist devices: impact of normothermic ex-vivo preservation using organ care system compared with cold storage” [97]

The dataset also contained one topic that, albeit related to dentistry, was not centered on implant dentistry or bone regeneration, i.e., topic #15 Periodontal defect treatment, and that could thus be potentially discarded but was retained following a conservative attitude, as explained above.

When we plotted the dimensionally reduced embeddings for these five selected OffTGs, these were again mostly located at the periphery of the scatter plot (Figure 6), at some semantic distance from the bulk of the data points. Overall, the bone augmentation dataset contained a smaller proportion of OffTAs compared to the peri-implantitis dataset, because our analysis identified only 11% of papers that could be safely excluded from further consideration (632 OffTAs out of 5309 articles).

We then assessed the allocation of the 36 target articles that were identified by the manual screening in the systematic review. No target article had been allocated to any of the OffTGs, indicating that their removal before manual screening would not have impacted the review outcome (

Table 4. The list of the target articles identified in Donos et al. systematic review on bone regeneration [35]. No article was clustered in any of the OffTGs.

Authors	Topic	Reference
Naenni, N., Stucki, L., Hüsler, J., Schneider, D., Hämmerle, C.H., Jung, R.E. and Thoma, D.S.	#16 Collagen Membranes for Guided Bone Regeneration	[98]
Basler, T., Naenni, N., Schneider, D., Hämmerle, C.H., Jung, R.E. and Thoma, D.S.	#16 Collagen Membranes for Guided Bone Regeneration	[99]
Mau, J.L., Grodin, E., Lin, J.J., Chen, M.C.J., Ho, C.H. and Cochran, D.,	#6 Bone Grafting for Dental Implants	[100]
Annen, B.M., Ramel, C.F., Hammerle, C.H.F. and Jung, R.E.	#16 Collagen Membranes for Guided Bone Regeneration	[101]
Naenni, N., Schneider, D., Jung, R.E., Hüsler, J., Hämmerle, C.H. and Thoma, D.S.	#16 Collagen Membranes for Guided Bone Regeneration	[102]
Lee, J.H., Lee, J.S., Baek, W.S., Lim, H.C., Cha, J.K., Choi, S.H. and Jung, U.W.	#16 Collagen Membranes for Guided Bone Regeneration	[103]
Lee, J.H., Park, S.H., Kim, D.H. and Jung, U.W.	#16 Collagen Membranes for Guided Bone Regeneration	[104]
Becker, J., Al-Nawas, B., Klein, M.O., Schliephake, H., Terheyden, H. and Schwarz, F.	-1 Dental Implants Studies	[105]
Schwarz, F., Schmucker, A. and Becker, J.	#16 Collagen Membranes for Guided Bone Regeneration	[106]
Schwarz, F., Hegewald, A., Sahm, N. and Becker, J.	#16 Collagen Membranes for Guided Bone Regeneration	[107]
Schwarz, F., Sahm, N. and Becker, J.	#6 Bone Grafting for Dental Implants	[108]
Benic, G.I., Eisner, B.M., Jung, R.E., Basler, T., Schneider, D. and Hämmerle, C.H.	#6 Bone Grafting for Dental Implants	[109]
Carpio, L., Loza, J., Lynch, S. and Genco, R.	#6 Bone Grafting for Dental Implants	[110]
Deesricharoenkiat, N., Jasilynn, P., Chuenchompoonut, V., Mattheos, N. and Thunyakitpisal, P.	#6 Bone Grafting for Dental Implants	[111]
Jung, R.E., Glauser, R., Schärer, P., Hämmerle, C.H., Sailer, H.F. and Weber, F.E.	-1 Dental Implant Studies	[112]
Jung, R.E., Windisch, S.I., Eggenschwiler, A.M., Thoma, D.S., Weber, F.E. and Hämmerle, C.H.	#6 Bone Grafting for Dental Implants	[113]
Jung, R.E., Kovacs, M.N., Thoma, D.S. and Hämmerle, C.H.	#13 Bone Regeneration Recombinant Human BMP	[114]
Jung, R.E., Hälg, G.A., Thoma, D.S. and Hämmerle, C.H.	#16 Collagen Membranes for Guided Bone Regeneration	[115]
Ramel, C.F., Wismeijer, D.A., F Hämmerle, C.H. and Jung, R.E.	#16 Collagen Membranes for Guided Bone Regeneration	[116]
Jung, R.E., Benic, G.I., Scherrer, D. and Hämmerle, C.H.	#7 Soft Tissue Augmentation for Dental Implants.	[117]
Jung, R.E., Mihatovic, I., Cordaro, L., Windisch, P., Friedmann, A., Blanco Carrion, J., Sanz Sanchez, I., Hallman, M., Quirynen, M. and Hammerle, C.H.	#16 Collagen Membranes for Guided Bone Regeneration	[118]
Benic, G.I., Bienz, S.P., Song, Y.W., Cha, J.K., Hämmerle, C.H., Jung, U.W. and Jung, R.E.	-1 Dental Implant Studies	[119]
Lee, D.W., Kim, K.T., Joo, Y.S., Yoo, M.K., Yu, J.A. and Ryu, J.J.	-1 Dental Implant Studies	[120]
Mattout, P., Nowzari, H. and Mattout, C.	#6 Bone Grafting for Dental Implants	[121]
Merli, M., Moscatelli, M., Mariotti, G., Pagliaro, U., Raffaelli, E. and Nieri, M.	-1 Dental Implant Studies	[122]
Merli, M., Moscatelli, M., Mariotti, G., Pagliaro, U., Raffaelli, E. and Nieri, M.	-1 Dental Implant Studies	[123]
Park, S.H., Lee, K.W., Oh, T.J., Misch, C.E., Shotwell, J. and Wang, H.L.	#16 Collagen Membranes for Guided Bone Regeneration	[124]
Schneider, D., Weber, F.E., Grunder, U., Andreoni, C., Burkhardt, R. and Jung, R.E.	#16 Collagen Membranes for Guided Bone Regeneration	[125]
Temmerman, A., Cortellini, S., Van Dessel, J., De Greef, A., Jacobs, R., Dhondt, R., Teughels, W. and Quirynen, M.	-1 Dental Implant Studies	[126]
Simion, M., Misitano, U., Gionso, L. and Salvato, A.	#16 Collagen Membranes for Guided Bone Regeneration	[127]
Urban, I.A., Wessing, B., Alández, N., Meloni, S., González-Martin, O., Polizzi, G., Sanz-Sanchez, I., Montero, E. and Zechner, W.	#16 Collagen Membranes for Guided Bone Regeneration	[128]
Wessing, B., Urban, I., Montero, E., Zechner, W., Hof, M., Alandez Chamorro, J., Alandez Martin, N., Polizzi, G., Meloni, S. and Sanz, M.	#16 Collagen Membranes for Guided Bone Regeneration	[129]
Van Assche, N., Michels, S., Naert, I. and Quirynen, M.	#6 Bone Grafting for Dental Implants	[130]
Veis, A.A., Tsirlis, A.T. and Parisis, N.A.	-1 Dental Implant Studies	[131]
Wen, S.C., Fu, J.H. and Wang, H.L.	-1 Dental Implant Studies	[132]
Tsai, Y.L., Tsao, J.P., Wang, C.L., Grodin, E., Lin, J.J., Chen, C.J., Ho, C.H., Cochran, D. and Mau, J.L.P.	#0 Dental Implants in Edentulous Patients	[133]

).

Interestingly, most target papers belonged to either group #16 Collagen Membranes for Guided Bone Regeneration (16 papers out of 36 target articles), group #6 Bone Grafting for Dental Implants (8 target articles), or group -1 (9 target articles) (Figure 7). This raises an interesting point for consideration as out of 5309 articles, 45% of the target articles were found in topic #16, which contained only 71 papers, and all the 36 articles were contained in six topic groups, which contained 2695 articles (noticeably, topic -1 alone contained 1515 papers).

We then manually screened the corpus again and found that no (dental) OnTA had been clustered in the OffTGs, yielding a classification specificity = 1. Upon inspection, we also found 204 OffTAs (i.e., unrelated to dentistry) that were not clustered in the OffTGs. The recall (or sensitivity) for this task is thus 0.74, and the F1 score for this classification task is 0.85, in line with the results for the first dataset. This indicates that additional papers could have been filtered out from the dataset, but it also confirms that no real dental paper (i.e., a potential target article for a systematic review) was inadvertently lost due to misclassification.

The proposed workflow for article screening is summarized in Figure 8.

It could be argued that alternative, faster, and less resource-intensive topic-modeling protocols could be implemented for the same purpose. We ran both latent Dirichlet allocation (LDA) and latent semantic analysis (LSA) on the titles of both datasets, setting the number of topics to 14 and 22, respectively, to match the topics identified by BERTopic. The performance of these algorithms can be found in Supplementary Table S3. To run both LDA and LSA, the titles had to be pre-processed, including stopword removal. While these approaches use bag-of-words mechanics, making them potentially less capable of capturing semantic nuances, their specificity was overall acceptable, i.e., they tended not to discard on-topic articles, making them potentially safe for use in this context. Their sensitivity was, however, very low when compared to BERTopic, which means that they failed to identify many off-topic articles, making them thus less effective in filtering non-relevant articles, as we propose.

Overall, high performing topic-modeling algorithms, such as BERTopic, create the opportunity to segment whole datasets of articles retrieved from online databases and literature repositories into clusters labelled by their topic, and this is turning out very useful to quickly understand the topic landscape of whole science fields [31,32,34].

Some of the topics BERTopic identified in our datasets are clearly unrelated to the matter at hand and can potentially and, based on our data, safely be removed and excluded from further screening, saving a variable amount of time to investigators. The first dataset, on peri-implantitis, was more heterogeneous and eight OffTGs were identified, which contained about 30% of the total number of papers in the dataset. It can be assumed that their exclusion would have significantly impacted the total manual screening time. The second dataset, on bone regeneration, was cleaner, and the five clearly unrelated OffTGs that BERTopic identified were smaller. It can be therefore assumed that cleaner, more focused datasets will benefit from this semantic filtering less than broader and more heterogeneous datasets. At the same time, it can also be hypothesized that the availability of these protocols of semantic filtering could affect the way keyword-based searches are conducted, relaxing the need for more stringent queries and allowing for broader, more heterogeneous datasets that can be filtered using semantic-based algorithms before manual screening. This stands in contrast to the previous approaches to literature searches, which have mostly relied on complex search strategies to minimize the retrieval of heterogeneous data [134,135,136].

We used the all-mpnet-base-v2 model for our work, which is a general model trained on a very wide corpus of texts. BERTopic, however, is independent of the embedding model, and as new models become available, these can be used to improve the effectiveness of the approach. Specific models, trained on corpora targeting specific science areas, could also be used, if the investigators deem it necessary, to better capture the possible meaning nuances of certain science niches.

One could argue that abstracts could provide more information on topic and context that just titles, and topic modeling should therefore be rather conducted on abstracts or a title + abstract combination. Of course, abstracts arguably provide more details on the content of a manuscript and work as well as or even better than titles to capture the theme of a report. Our choice of using titles for the topic-modeling analysis was rather based on the performance of the proposed algorithm. Abstracts are considerably longer than titles, and processing them takes much longer than analyzing titles, even if using hardware acceleration. One of the purposes of our work was to prove the feasibility of semantic title screening as an acceptable compromise to filter out off-topic articles prior to screening. Our approach, which relies on titles alone, could be easily scaled up to much bigger datasets, paving the way to changes in the paradigms of how literature searches are managed as a whole.

In fact, our brief analysis also suggests that some topics might even be positively selected to conduct a more restricted and focused screening, with some caveats. In our situation, it was easy to retrospectively identify the topics where the target articles were contained but implementing that with a new dataset would not be as straightforward, because many topics would be about closely related areas, and excluding them would be risky. As an example of this, considering a hypothetical adoption of BERTopic in the pre-screening phase of the bone regeneration dataset for a systematic review, topic #16 Collagen Membranes for Guided Bone Regeneration could have been an easy pick for further assessment, but at this stage, there would have been no rationale to safely exclude, e.g., the quite large (n = 692) #1 Sinus Floor Elevation group by just looking at its LLM descriptor or its keywords.

One final aspect to consider is the knowledge and language barrier that this procedure still poses to many life science investigators. At the present moment, applying this and similar algorithms using command line interfaces still requires a degree of coding literacy that may discourage many investigators in biology and medicine, although unjustifiably. We encourage investigators to become familiar with coding interfaces such as Google Colab or Jupyter notebooks, as they often provide an affordable alternative to proprietary software and offer a quick way to deploy, customize, and maximize the potential of new algorithms. Given the rapid advancements in machine learning and artificial intelligence, relying solely on point-and-click user interfaces may be a risky choice. Investing time in learning basic Python syntax may be a more sustainable approach to keep up with the current technological revolution. To assist those interested in trying out the algorithm used in this study, we have made a simplified version of code available as a ipynb notebook in the Supplementary Materials.

4. Conclusions

Our analysis focused on two separate datasets, and used BERTopic, a popular topic-modeling algorithm, to identify sets of articles to discard from datasets of the biomedical literature prior to manual screening to expedite evidence identification in systematic reviews. Taken together, our data show that encoding article titles using the all-mpnet-base-v2 model, followed by semantic clustering with BERTopic, is an inexpensive and quick way to categorize articles into topics and have an effective overview of the dataset’s semantic structure. This procedure has sufficient granularity to identify article groups that can be safely removed from the dataset before it is processed by the investigators during the reliable but slow and labor-intensive process of manual inspection. The number of topics unrelated to the query varies according to the query itself, the keywords used to conduct it, and the database used to create the dataset, but in one of the two datasets we used, we were able to filter out more than 1800 articles, or 30% of the dataset. Furthermore, we also observed that the target articles that had been actually identified for the systematic reviews tended to be found in few clusters, This suggests that semantic search can help investigators not only identify unrelated articles for exclusion but also focus manual inspection on a smaller, relevant subset of the dataset, with faster processing.

The key highlights of our study can be listed as follows:

• We applied BERTopic to two datasets of biomedical literature (dentistry).
• BERTopic identified a significant number of papers unrelated to the query.
• Off-topic papers constituted up to 30% of the initial dataset.
• BERTopic effectively filtered out off-topic papers from datasets, saving review time.
• The excluded topic clusters contained no relevant manuscripts, ensuring safe exclusion.