An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases

Kasaraneni Purna Prakash; Kasaraneni HimaJyothi; Salini Rosaline; Yellapragada Venkata Pavan Kumar; Gogulamudi Pradeep Reddy; Naveen Mukkapati

doi:10.3390/publications14020034

Abstract

Bibliometric data from various databases are crucial for exploring research trends through a bibliometric analysis. Usually, deduplicating records and merging several citation index databases for bibliometric research is tedious, particularly when dealing with larger datasets. Although several manual and automatic merging processes are available in the academic literature, some key issues were identified during the implementation of existing merging processes. To address such issues, this paper proposes an open-source preprocessing pipeline developed using R programming for a simple merging of bibliometric data collected from multiple databases. This open-source reproducible preprocessing pipeline precompiles and deduplicates records based on a Digital Object Identifier (DOI). To implement this proposed research work, bibliometric data are considered from Scopus, Web of Science and Lens databases. The key outcomes of this research work are identifying multiple DOIs and Titles, standardizing the DOIs, and deduplicating records to obtain a merged dataset without noisy data. This enables researchers to conduct an effective bibliometric analysis.

Keywords:

bibliometric data; bibliographic database; Lens; merging data; Scopus; Web of Science

1. Introduction

Access to research data and resources is essential for maintaining openness, reproducibility, and transparency of science (Nosek et al., 2015). This paper contributes to the preprocessing and development of bibliometric data used for bibliometric analysis. The statistical and mathematical methods are utilized in bibliometric analysis (Groos & Pritchard, 1969). This analysis is prevalent in the present research world, facilitating a comprehensive understanding of trends, identifying the top journals, affiliations and countries that contributed to the research work and finding the top authors on a particular research topic. Records from a single citation database may not provide a sufficiently comprehensive bibliometric analysis, and consequently, the results may not be reliable (Fang et al., 2025). Hence, it is important to consider multiple databases for gathering bibliometric data.

Merging of articles from multiple databases is significant for several reasons.

Merging of databases provides comprehensive academic literature coverage without missing the most relevant and important articles in the concerned research field.
The search results will be improved in the merged dataset by identifying the duplicates and multiple DOIs.
The merged dataset enables a comprehensive analysis that provides valuable insights to the researchers to generate innovative and quality research.
Accessing the comprehensive and accurate dataset enables data-driven decision-making in the research evaluation.
Automatic merging of data from multiple databases will save the time and effort of the researchers, enabling them to spend more time on providing quality research work.

Several data mining tools are available for implementing this process in the current digital era (Shi et al., 2011). Some popular tools for merging and analyzing data from multiple databases include VOSviewer, CiteSpace, Gephi, Tableau, R packages and Python libraries (Donthu et al., 2021; Kara et al., 2025). From these, R was used for data analysis with the flexible use of available libraries (Berga et al., 2025). R is an ideal programming environment for statistical computation and data analysis. R was created in the early 1990s by Robert Gentleman and Ross Ihaka to make sophisticated data processing easier and produce simple, adaptable visualizations. Its capabilities and the wide range of packages it offers have made it more and more popular over time among statistics, data science, and bioinformatics (Giorgi et al., 2022). The “Bibliometrix” package in R provides a comprehensive toolkit for bibliometric and scientometric analysis, ideal for quantitative research. Furthermore, R packages were used to merge the articles from multiple datasets (Chansanam & Li, 2025). These datasets help to generate a merged dataset, necessary for generating data value (Gong et al., 2023). Even large datasets can be merged by dealing with duplicate records and multiple DOIs. A merged dataset without noisy data provides a basis for data analysis and generates valuable insights.

2. Literature Review, Problem Statement and Proposed Solution

2.1. Literature Review

Bibliometric analysis can generate accurate results that are based on the quality of the data. One critical part of data quality is identifying and rectifying the issues of missing or inaccurate data in the dataset (Gong et al., 2023). These deficiencies can be handled by applying data cleaning to improve the quality of the dataset. Combining information from multiple databases is crucial for bibliometric research, in which the unique characteristics of each database may affect the outcomes of the analysis (Echchakoui, 2020). Only when Web of Science (WoS) (Clarivate, 2026) and Scopus (Elsevier, 2026) databases provide representative coverage of the scientific activities assessed are analyses based on these databases considered valid (Vera-Baceta et al., 2019). Currently, the consideration of other databases along with Scopus and WoS is essential for bibliometric analysis. The Turgel and Chernova (2024) study mentioned that besides Scopus and WoS, the alternative databases such as Lens, Dimensions, Wizdom.ai, AMiner, and OpenAlex provide essential data to identify key trends in the scientific development. Dimensions is an abstract and citation database of research grants that connects the grants to the publications. It was launched in 2018 and is headquartered in London, United Kingdom. Wizdom.ai is an AI-driven, free, and comprehensive research management tool useful for searching and managing publications. Its former name was Colwiz. AMiner is an AI-powered search engine that enhances research discovery by analyzing millions of papers. It uses natural language processing to do the intended tasks. OpenAlex is an academic database with a multidisciplinary nature that contains more than 256 million research works. These works comprise publications, grants, and datasets. It has advanced search features, citation tracking, and tools for visualization. The structural and functional heterogeneity of various scientometric databases, arising from their diverse contexts and source coverage, results in distinct content. However, the quality of the content generated from open scientometric databases is questionable due to their incapability and low entry barriers. Manual processes were discussed to merge the records from different databases (Singh et al., 2021; Visser et al., 2021; Zhu & Liu, 2020). The manual integration of two databases is challenging due to different tag fields; researchers who use multiple databases for bibliometric analysis examine them independently. For instance, the tag field “Cited-By” in Scopus represents author citations, and similarly, in WoS it is represented as “Cited-Reference”. Therefore, it is challenging to combine two databases to perform original analysis, particularly when the databases are huge. It is difficult for the researcher even to retrieve publications from multiple databases using the same “.bib” file format (Echchakoui, 2020). Some studies (Caputo & Kargina, 2022; Kasaraneni & Rosaline, 2024) mentioned an automatic process for merging the records from multiple databases. These studies attempted to solve the tag fields’ comparison in different databases and remove duplicates from the merged dataset. Not only on tag fields, but also relying on the Digital Object Identifier (DOI) has limitations during the merging, even though it is a unique identifier (Culbert et al., 2025). These limitations include the records with duplicate DOIs and no DOIs, which need to be excluded. Vieira and Leta (2024) highlighted that they differ by subject in the Scopus and WoS databases. The proposed research work aimed to obtain a merged dataset by solving issues such as records with duplicate DOIs, multiple DOIs or no DOIs. Some literature works on merging databases are provided in Table 1.

Table 1. Some previous studies on the merging of bibliometric databases.

2.2. Problem Statement

This study identified various issues during database merging using the process available in earlier research works.

A bibliometrix package “mergeDbSources” was already available for merging the databases (Bibliometrix, 2020). It was identified that the articles under ISI were only recorded in the merged dataset by using this package, which may cause the omission of important articles from other databases.
The code available in Kasaraneni and Rosaline (2024) was executed to merge the articles from the Scopus and WoS databases. The duplicate records were identified based on the DOI. An issue was identified during the removal of duplicate records. It was observed that different databases use either uppercase or lowercase letters to represent data in the DOI column. This issue may impact the duplicate records identification.
Another issue was identified in the datasets resulting from the keyword search. Multiple DOIs and Titles were generated in the same record, which creates inconsistency in the bibliometric data.

2.3. Proposed Solution

This paper discusses various R functions to solve the problems highlighted in Section 2.1. These functions facilitate the conversion of DOIs to lowercase and remove multiple DOIs when they are identified in the same record. The proposed research work utilizes common metadata fields provided in Table 2 to deduplicate the records in the considered databases. This proposed work gives equal priority to all databases while merging the records.

Table 2. Common metadata fields available in different databases.

The other sections of the paper include data collection in Section 3, methodology of the proposed research in Section 4, rationale of the proposed process in Section 5, results and discussion in Section 6, and conclusions in Section 7.

3. Data Collection

The bibliometric data from the Scopus, WoS and Lens databases were considered for the proposed research work. Elsevier launched the abstract and citation database Scopus in 2004. Four numerical metrics, such as SJR (SCImago Journal Rank), h-Index, CiteScore, and SNIP (Source Normalized Impact per Paper), are used to evaluate the quality of each journal in the Scopus database once a year. In a similar vein, the WoS is regarded as the top publisher-independent citation database globally. A group of databases encompassing subjects including biology (Biosis), computers (Inspec), social sciences, science and humanities (Core Collection) makes up this Clarivate-managed database collection (Kasaraneni & Rosaline, 2024). The third database selected is ‘Lens’, a free global platform that enables the discovery of both scholarly (255 million) and patent (127 million) records (Lens, 2026). The Lens allows for visualizing the top publications and geographical distribution of the researchers. It offers information in various graphical formats such as bar charts, global maps, heat maps and more. It also enables custom grouping by integrating values of many aspects as a combined visualization (Turgel & Chernova, 2024).

From the selected databases, namely Scopus, WoS, and Lens, articles were searched on 11-JULY-2024 using the desired keywords. It is recommended to use the advanced search features available in the databases. These allow the knowledge base access for searching by applying Boolean operators and keywords to complete the required search (Caputo & Kargina, 2022). As a case study, the keywords considered to extract the bibliometric data from Scopus, WoS, and Lens included “Brain tumor detection” OR “Brain tumor classification”. The search strategy is presented in Table 3. As discussed in Tranfield et al. (2003), the deliberately chosen and validated keywords were considered for the process after the search. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) model discussed in Moher et al. (2009) recommends determining and achieving refinement stages to ensure data reliability and uniformity. The stages of this model are identification, screening, admissibility, and inclusion. During the identification, the count of publications was 1138 from the Scopus database, 839 from the WoS database and 868 from the Lens database. During the screening, publications in the English language and the document type “articles” only were retained, and other publications were excluded. The coverage of years was also limited to 2013–2023. During admissibility, publications in the related areas of “Brain tumor detection” OR “Brain tumor classification” are retained, and publications in the unrelated areas are excluded. During the inclusion, 1138 records from Scopus, 839 records from WoS, and 868 records from Lens were considered in “.bib” file format. The “Bibliometrix” (Aria & Cuccurullo, 2017) library was imported to obtain “.xlsx” files from “.bib” files. These “.xlsx” files were used to conduct the proposed research work.

Table 3. The search strategy in various databases.

4. Methodology

Initially, a mergeDbSources function developed in Bibliometrix (2020) was implemented on the considered data. The algorithmic steps of the implemented mergeDBSources function are given in Table 4. This function provided only ISI-indexed records in the merged file, which can result in losing the important records from other databases. Hence, to address this problem, the step-by-step approach for merging the articles from multiple databases is shown in Figure 1. The process begins with importing all the required libraries in the R programming environment. The datasets collected from multiple databases are read. From these datasets, the columns with the common names are retrieved and saved as sub-datasets. These sub-datasets are combined to represent a single file. To deal with DOI case issues, the available DOIs are standardized by converting them into lowercase. By using these standardized DOIs, the duplicate records can be removed and the distinct values retained. Identify the records with multiple DOIs and/or titles and remove them to have a merged dataset. The detailed algorithmic steps of the proposed methodology are given in Table 5.

Table 4. Steps involved in mergeDBSources function.

Figure 1. Workflow of the merging process.

Table 5. Steps involved in the proposed work.

5. Rationale of the Proposed Research Work

Several issues are identified while merging the data from multiple databases. The proposed work is focused to solve such issues and simplify the merging process by the development of an open-source reproducible preprocessing pipeline.

Issue 1: By using the mergeDbSources function, only Institute for Scientific Information (ISI) records (fetched from WoS database) are prioritized. To avoid this consequence, the authors have developed an open-source reproducible preprocessing pipeline to get records from all databases with equal priority.
Issue 2: DOIs are recorded in small and capital cases in multiple databases. All DOIs are converted to records in small cases for better identification of duplicate records.
Issue 3: The other issue recognized is that multiple DOIs are identified in one record, which may affect the identification of duplicates. Hence, the records with multiple DOIs are identified and eliminated from the merged dataset.

6. Results and Discussion

After the implementation of mergeDBSources, it has been observed that it provided only the ISI database articles in the final merged dataset instead of considering the records from Scopus and Lens, which is shown in Figure 2 and Figure 3. From these figures, it is evident that the dataset contains the records indexed only with ISI after the merging. To better understand the overall dataset, the records at the beginning are shown in Figure 2, and the records at the end are shown in Figure 3. In total, there are 2084 records in the merged file. The details for Figure 2 and Figure 3 are available in the folder named “Results_with_mergeDBSources_function” in the Supplementary Materials. Duplicate DOIs with different cases are shown in Figure 4. This case difference of the same DOI will affect the deduplication of records in the dataset. In Figure 4, the DOIs, 10.12694/SCPE.V21i1.1600 and 10.12694/scpe.v21i1.1600 have different cases. The detailed presentation of the same DOIs with different cases is presented in Figure 5 and Figure 6. The uppercase DOI from Scopus is shown in Figure 5. The lowercase DOI from WoS is shown in Figure 6.

Figure 2. Records indexed only with ISI in the merged dataset (beginning of the dataset).

Figure 3. Records indexed only with ISI in the merged dataset (end of the dataset).

Figure 4. DOIs in different cases.

Figure 5. DOI in uppercase in Scopus.

Figure 6. DOI in lowercase in WoS.

While observing the dataset during merging process, multiple DOIs are identified in some records as shown in Figure 7, the column titled “DI” has multiple DOIs in each row. For example, “10.1007/s13755-022-00203-w 10.1002/ima.22830” are recorded in a single row, which leads to inconsistency. Accordingly, multiple titles are also present in the dataset, depending on the multiple DOIs. In Figure 8, for example, the column titled “TI” is observed with two different titles “BTC-FCNN: FAST CONVOLUTION NEURAL NETWORK FOR MULTI-CLASS BRAIN TUMOR CLASSIFICATION A MODIFIED THRESHOLD SCORE-BASED MULTILEVEL THRESHOLDING SEGMENTATION TECHNIQUE FOR BRAIN MAGNETIC RESONANCE IMAGES USING OPPOSITION-BASED LEARNING HYBRID RICE OPTIMIZATION ALGORITHM” in the same record. Once multiple DOIs are identified, they are removed from the dataset because they are invalid for comparing records from different databases, depending on the DOI. Along with the multiple DOIs, multiple titles are also removed. Hence, a merged dataset without noisy data is available as presented in Figure 9. The details for Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 are available in the folder named “Results_brain_tumor_data” in the Supplementary Materials.

Figure 7. Multiple DOIs.

Figure 8. Multiple DOIs and TIs.

Figure 9. Merged dataset without noisy data.

The summary of quantitative details for the present data (brain tumor) is provided in Table 6. The details for Table 6 are available in the folders named “Source_Brain Tumor” and “Results_brain_tumor_data” in the Supplementary Materials. To validate the performance of the proposed methodology, a new large set of data on the topic “Parkinson’s disease” is considered. The summary of quantitative details for these data is provided in Table 7. The details for Table 7 are available in the folders named “Source_Data_Parkinson’s Disease” and “Results_Parkinsons_disease_data” in the Supplementary Materials. From these tables, it is to be noted that the processing time for both the present data and a new large set of data is in seconds, which reflects the good performance of the proposed preprocessing pipeline. In the processing time, the start and end timings are considered based on the system time and the difference between these timings provides the actual processing time taken to complete the proposed preprocessing. In Table 6, the time difference observed is 2.199827 s, and in Table 7, the time difference observed is 21.31339 s.

Table 6. Summary of quantitative details for present data (brain tumor).

Table 7. Summary of quantitative details for a new large set of data (Parkinson’s disease).

During the preprocessing of a new large set of data, it is observed that there are only multiple DOIs, and there are no multiple titles. Although there are no multiple titles, it still creates ambiguity in choosing the right DOI for a particular record. Hence, those records are also to be removed to avoid the noise in the data. It is evident from the two cases considered that there are fewer records with multiple DOIs, and the deletion of such records will not impact the size of the data in further analysis. Final key observations from the preprocessing are as follows.

Some records contain both multiple DOIs and multiple titles, which is very important to focus on.
Some records contain multiple DOIs but no multiple titles.

7. Conclusions

This paper addresses various issues related to the merging process of documents from multiple databases. It presents an effortless and accurate process to merge datasets from Scopus, WOS and Lens. In addition to these, researchers can include other databases if their columns are matched. This merging process is scalable and can handle large volumes of data in less time.

The key outcomes of this research work are mentioned as follows:

While performing the merging process, the proposed preprocessing pipeline removes duplicates from all the databases with equal priority by solving the issue of getting only ISI-indexed records.
DOIs are presented in different cases of alphabets in different databases. This case difference will affect the searching of duplicate records. Hence, to solve this issue, a function is implemented to convert all DOIs to lowercase.
While merging the records from multiple databases, multiple DOIs and titles were identified and removed from the final merged dataset.
Finally, a merged dataset without the noisy data is available for conducting the required analyses.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/publications14020034/s1. Folder: Results_with_mergeDBSources_function; Folder: Source_Brain Tumor; Folder: Results_brain_tumor_data; Folder: Source_Data_Parkinson’s Disease; Folder: Results_Parkinsons_disease_data.

Author Contributions

Conceptualization, K.H., S.R. and Y.V.P.K.; methodology, K.P.P., G.P.R., and S.R.; software, N.M.; validation, Y.V.P.K.; formal analysis, K.H. and S.R.; investigation, K.P.P. and K.H.; resources, N.M.; data curation, G.P.R. and N.M.; writing—original draft preparation, K.H. and S.R.; writing—review and editing, K.P.P. and G.P.R.; visualization, G.P.R. and K.H.; supervision, Y.V.P.K.; project administration, Y.V.P.K.; funding acquisition, K.P.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The source files of the datasets and the obtained result datasets are provided as Supplementary Files.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:

DOI	Digital Object Identifier
ISI	Institute for Scientific Information
PRISMA	Preferred Reporting Items for Systematic Reviews and Meta-Analyses
SJR	SCImago Journal Rank
SNIP	Source Normalized Impact per Paper
WoS	Web of Science

References

Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4), 959–975. [Google Scholar] [CrossRef] [Scilit]
Berga, M., Coelho, P., & Ochman, A. (2025). Top 21 data mining tools. Available online: https://www.imaginarycloud.com/blog/data-mining-tools (accessed on 21 June 2025).
Bibliometrix. (2020). Comprehensive science mapping analysis. Available online: https://www.bibliometrix.org/ (accessed on 17 March 2026).
Caputo, A., & Kargina, M. (2022). A user-friendly method to merge Scopus and Web of Science data during bibliometric analysis. Journal of Marketing Analytics, 10(1), 82–88. [Google Scholar] [CrossRef] [Scilit]
Chansanam, W., & Li, C. (2025). KKU-BiblioMerge: A novel tool for multi-database integration in bibliometric analysis. Iberoamerican Journal of Science Measurement and Communication, 5(1), 1–16. [Google Scholar] [CrossRef] [Scilit]
Clarivate. (2026). Web of science database. Available online: https://access.clarivate.com/login?app=wos (accessed on 17 May 2026).
Culbert, J. H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M., Donner, P., & Mayr, P. (2025). Reference coverage analysis of OpenAlex compared to Web of Science and Scopus. Scientometrics, 130(4), 2475–2492. [Google Scholar] [CrossRef] [Scilit]
Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133, 285–296. [Google Scholar] [CrossRef] [Scilit]
Echchakoui, S. (2020). Why and how to merge Scopus and Web of Science during bibliometric analysis: The case of sales force literature from 1912 to 2019. Journal of Marketing Analytics, 8(3), 165–184. [Google Scholar] [CrossRef] [Scilit]
Elsevier. (2026). Scopus database. Available online: https://www.scopus.com/pages/home (accessed on 17 May 2026).
Fang, Y., Li, X., Mohtar, T. M., & Chekima, B. (2025). A bibliometric analysis of work–family balance: Trends, themes, and future directions (2000–2024). Cogent Business & Management, 12(1), 2541041. [Google Scholar] [CrossRef] [Scilit]
Giorgi, F. M., Ceraolo, C., & Mercatelli, D. (2022). The R language: An engine for bioinformatics and data science. Life, 12(5), 648. [Google Scholar] [CrossRef] [Scilit]
Gong, Y., Liu, G., Xue, Y., Li, R., & Meng, L. (2023). A survey on dataset quality in machine learning. Information and Software Technology, 162, 107268. [Google Scholar] [CrossRef] [Scilit]
Groos, O. V., & Pritchard, A. (1969). Documentation notes. Journal of Documentation, 25(4), 344–349. [Google Scholar] [CrossRef] [Scilit]
Kara, B. C., Şahin, A., & Dirsehan, T. (2025). BibexPy: Harmonizing the bibliometric symphony of Scopus and Web of Science. SoftwareX, 30, 102098. [Google Scholar] [CrossRef] [Scilit]
Kasaraneni, H., & Rosaline, S. (2024). Automatic merging of Scopus and Web of Science data for simplified and effective bibliometric analysis. Annals of Data Science, 11(3), 785–802. [Google Scholar] [CrossRef] [Scilit]
Lens. (2026). Lens database. Available online: https://www.lens.org/ (accessed on 17 May 2026).
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. [Google Scholar] [CrossRef] [Scilit]
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. [Google Scholar] [CrossRef] [Scilit]
Shi, Y., Tian, Y., Kou, G., Peng, Y., & Li, J. (2011). Optimization based data mining: Theory and applications. Springer. [Google Scholar] [CrossRef] [Scilit]
Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021). The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics, 126(6), 5113–5142. [Google Scholar] [CrossRef] [Scilit]
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222. [Google Scholar] [CrossRef] [Scilit]
Turgel, I. D., & Chernova, O. A. (2024). Open science alternatives to Scopus and the Web of Science: A case study in regional resilience. Publications, 12(4), 43. [Google Scholar] [CrossRef] [Scilit]
Vera-Baceta, M.-A., Thelwall, M., & Kousha, K. (2019). Web of Science and Scopus language coverage. Scientometrics, 121(3), 1803–1813. [Google Scholar] [CrossRef] [Scilit]
Vieira, G. A., & Leta, J. (2024). biblioverlap: An R package for document matching across bibliographic datasets. Scientometrics, 129(7), 4513–4527. [Google Scholar] [CrossRef] [Scilit]
Visser, M., Van Eck, N. J., & Waltman, L. (2021). Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. Quantitative Science Studies, 2(1), 20–41. [Google Scholar] [CrossRef] [Scilit]
Zhu, J., & Liu, W. (2020). A tale of two databases: The use of Web of Science and Scopus in academic papers. arXiv, arXiv:2002.02608. [Google Scholar] [CrossRef] [Scilit]