Next Article in Journal
Literature Search Query in Academic Databases: Artificial Intelligence Think Tank Guideline for Literature Reviews
Previous Article in Journal
Scientific Production in Global Mental Health: A Meta-Research Study of Income-Stratified Trends, Gaps, and Health Metrics Impact
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Publications
Volume 14
Issue 2
10.3390/publications14020034
publications-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases

by
Kasaraneni Purna Prakash
1,
Kasaraneni HimaJyothi
2,
Salini Rosaline
3,
Yellapragada Venkata Pavan Kumar
4,*,
Gogulamudi Pradeep Reddy
5 and
Naveen Mukkapati
6
1
Department of Computer Science and Engineering, Siddhartha Academy of Higher Education, Deemed to be University, Vijayawada 520007, Andhra Pradesh, India
2
Department of Business Administration, Andhra Loyola College (Autonomous), Vijayawada 520008, Andhra Pradesh, India
3
Department of Management Studies, Netaji Subhas University of Technology, Dwarka Sector-3, New Delhi 110078, Delhi, India
4
School of Electronics Engineering, VIT-AP University, Amaravati 522241, Andhra Pradesh, India
5
Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, Karnataka, India
6
Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur 522502, Andhra Pradesh, India
*
Author to whom correspondence should be addressed.
Publications 2026, 14(2), 34; https://doi.org/10.3390/publications14020034
Submission received: 12 April 2026 / Revised: 21 May 2026 / Accepted: 26 May 2026 / Published: 1 June 2026
Browse Figures
Versions Notes

Abstract

Bibliometric data from various databases are crucial for exploring research trends through a bibliometric analysis. Usually, deduplicating records and merging several citation index databases for bibliometric research is tedious, particularly when dealing with larger datasets. Although several manual and automatic merging processes are available in the academic literature, some key issues were identified during the implementation of existing merging processes. To address such issues, this paper proposes an open-source preprocessing pipeline developed using R programming for a simple merging of bibliometric data collected from multiple databases. This open-source reproducible preprocessing pipeline precompiles and deduplicates records based on a Digital Object Identifier (DOI). To implement this proposed research work, bibliometric data are considered from Scopus, Web of Science and Lens databases. The key outcomes of this research work are identifying multiple DOIs and Titles, standardizing the DOIs, and deduplicating records to obtain a merged dataset without noisy data. This enables researchers to conduct an effective bibliometric analysis.

1. Introduction

Access to research data and resources is essential for maintaining openness, reproducibility, and transparency of science (Nosek et al., 2015). This paper contributes to the preprocessing and development of bibliometric data used for bibliometric analysis. The statistical and mathematical methods are utilized in bibliometric analysis (Groos & Pritchard, 1969). This analysis is prevalent in the present research world, facilitating a comprehensive understanding of trends, identifying the top journals, affiliations and countries that contributed to the research work and finding the top authors on a particular research topic. Records from a single citation database may not provide a sufficiently comprehensive bibliometric analysis, and consequently, the results may not be reliable (Fang et al., 2025). Hence, it is important to consider multiple databases for gathering bibliometric data.
Merging of articles from multiple databases is significant for several reasons.
  • Merging of databases provides comprehensive academic literature coverage without missing the most relevant and important articles in the concerned research field.
  • The search results will be improved in the merged dataset by identifying the duplicates and multiple DOIs.
  • The merged dataset enables a comprehensive analysis that provides valuable insights to the researchers to generate innovative and quality research.
  • Accessing the comprehensive and accurate dataset enables data-driven decision-making in the research evaluation.
  • Automatic merging of data from multiple databases will save the time and effort of the researchers, enabling them to spend more time on providing quality research work.
Several data mining tools are available for implementing this process in the current digital era (Shi et al., 2011). Some popular tools for merging and analyzing data from multiple databases include VOSviewer, CiteSpace, Gephi, Tableau, R packages and Python libraries (Donthu et al., 2021; Kara et al., 2025). From these, R was used for data analysis with the flexible use of available libraries (Berga et al., 2025). R is an ideal programming environment for statistical computation and data analysis. R was created in the early 1990s by Robert Gentleman and Ross Ihaka to make sophisticated data processing easier and produce simple, adaptable visualizations. Its capabilities and the wide range of packages it offers have made it more and more popular over time among statistics, data science, and bioinformatics (Giorgi et al., 2022). The “Bibliometrix” package in R provides a comprehensive toolkit for bibliometric and scientometric analysis, ideal for quantitative research. Furthermore, R packages were used to merge the articles from multiple datasets (Chansanam & Li, 2025). These datasets help to generate a merged dataset, necessary for generating data value (Gong et al., 2023). Even large datasets can be merged by dealing with duplicate records and multiple DOIs. A merged dataset without noisy data provides a basis for data analysis and generates valuable insights.

2. Literature Review, Problem Statement and Proposed Solution

2.1. Literature Review

Bibliometric analysis can generate accurate results that are based on the quality of the data. One critical part of data quality is identifying and rectifying the issues of missing or inaccurate data in the dataset (Gong et al., 2023). These deficiencies can be handled by applying data cleaning to improve the quality of the dataset. Combining information from multiple databases is crucial for bibliometric research, in which the unique characteristics of each database may affect the outcomes of the analysis (Echchakoui, 2020). Only when Web of Science (WoS) (Clarivate, 2026) and Scopus (Elsevier, 2026) databases provide representative coverage of the scientific activities assessed are analyses based on these databases considered valid (Vera-Baceta et al., 2019). Currently, the consideration of other databases along with Scopus and WoS is essential for bibliometric analysis. The Turgel and Chernova (2024) study mentioned that besides Scopus and WoS, the alternative databases such as Lens, Dimensions, Wizdom.ai, AMiner, and OpenAlex provide essential data to identify key trends in the scientific development. Dimensions is an abstract and citation database of research grants that connects the grants to the publications. It was launched in 2018 and is headquartered in London, United Kingdom. Wizdom.ai is an AI-driven, free, and comprehensive research management tool useful for searching and managing publications. Its former name was Colwiz. AMiner is an AI-powered search engine that enhances research discovery by analyzing millions of papers. It uses natural language processing to do the intended tasks. OpenAlex is an academic database with a multidisciplinary nature that contains more than 256 million research works. These works comprise publications, grants, and datasets. It has advanced search features, citation tracking, and tools for visualization. The structural and functional heterogeneity of various scientometric databases, arising from their diverse contexts and source coverage, results in distinct content. However, the quality of the content generated from open scientometric databases is questionable due to their incapability and low entry barriers. Manual processes were discussed to merge the records from different databases (Singh et al., 2021; Visser et al., 2021; Zhu & Liu, 2020). The manual integration of two databases is challenging due to different tag fields; researchers who use multiple databases for bibliometric analysis examine them independently. For instance, the tag field “Cited-By” in Scopus represents author citations, and similarly, in WoS it is represented as “Cited-Reference”. Therefore, it is challenging to combine two databases to perform original analysis, particularly when the databases are huge. It is difficult for the researcher even to retrieve publications from multiple databases using the same “.bib” file format (Echchakoui, 2020). Some studies (Caputo & Kargina, 2022; Kasaraneni & Rosaline, 2024) mentioned an automatic process for merging the records from multiple databases. These studies attempted to solve the tag fields’ comparison in different databases and remove duplicates from the merged dataset. Not only on tag fields, but also relying on the Digital Object Identifier (DOI) has limitations during the merging, even though it is a unique identifier (Culbert et al., 2025). These limitations include the records with duplicate DOIs and no DOIs, which need to be excluded. Vieira and Leta (2024) highlighted that they differ by subject in the Scopus and WoS databases. The proposed research work aimed to obtain a merged dataset by solving issues such as records with duplicate DOIs, multiple DOIs or no DOIs. Some literature works on merging databases are provided in Table 1.

2.2. Problem Statement

This study identified various issues during database merging using the process available in earlier research works.
  • A bibliometrix package “mergeDbSources” was already available for merging the databases (Bibliometrix, 2020). It was identified that the articles under ISI were only recorded in the merged dataset by using this package, which may cause the omission of important articles from other databases.
  • The code available in Kasaraneni and Rosaline (2024) was executed to merge the articles from the Scopus and WoS databases. The duplicate records were identified based on the DOI. An issue was identified during the removal of duplicate records. It was observed that different databases use either uppercase or lowercase letters to represent data in the DOI column. This issue may impact the duplicate records identification.
  • Another issue was identified in the datasets resulting from the keyword search. Multiple DOIs and Titles were generated in the same record, which creates inconsistency in the bibliometric data.

2.3. Proposed Solution

This paper discusses various R functions to solve the problems highlighted in Section 2.1. These functions facilitate the conversion of DOIs to lowercase and remove multiple DOIs when they are identified in the same record. The proposed research work utilizes common metadata fields provided in Table 2 to deduplicate the records in the considered databases. This proposed work gives equal priority to all databases while merging the records.
The other sections of the paper include data collection in Section 3, methodology of the proposed research in Section 4, rationale of the proposed process in Section 5, results and discussion in Section 6, and conclusions in Section 7.

3. Data Collection

The bibliometric data from the Scopus, WoS and Lens databases were considered for the proposed research work. Elsevier launched the abstract and citation database Scopus in 2004. Four numerical metrics, such as SJR (SCImago Journal Rank), h-Index, CiteScore, and SNIP (Source Normalized Impact per Paper), are used to evaluate the quality of each journal in the Scopus database once a year. In a similar vein, the WoS is regarded as the top publisher-independent citation database globally. A group of databases encompassing subjects including biology (Biosis), computers (Inspec), social sciences, science and humanities (Core Collection) makes up this Clarivate-managed database collection (Kasaraneni & Rosaline, 2024). The third database selected is ‘Lens’, a free global platform that enables the discovery of both scholarly (255 million) and patent (127 million) records (Lens, 2026). The Lens allows for visualizing the top publications and geographical distribution of the researchers. It offers information in various graphical formats such as bar charts, global maps, heat maps and more. It also enables custom grouping by integrating values of many aspects as a combined visualization (Turgel & Chernova, 2024).
From the selected databases, namely Scopus, WoS, and Lens, articles were searched on 11-JULY-2024 using the desired keywords. It is recommended to use the advanced search features available in the databases. These allow the knowledge base access for searching by applying Boolean operators and keywords to complete the required search (Caputo & Kargina, 2022). As a case study, the keywords considered to extract the bibliometric data from Scopus, WoS, and Lens included “Brain tumor detection” OR “Brain tumor classification”. The search strategy is presented in Table 3. As discussed in Tranfield et al. (2003), the deliberately chosen and validated keywords were considered for the process after the search. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) model discussed in Moher et al. (2009) recommends determining and achieving refinement stages to ensure data reliability and uniformity. The stages of this model are identification, screening, admissibility, and inclusion. During the identification, the count of publications was 1138 from the Scopus database, 839 from the WoS database and 868 from the Lens database. During the screening, publications in the English language and the document type “articles” only were retained, and other publications were excluded. The coverage of years was also limited to 2013–2023. During admissibility, publications in the related areas of “Brain tumor detection” OR “Brain tumor classification” are retained, and publications in the unrelated areas are excluded. During the inclusion, 1138 records from Scopus, 839 records from WoS, and 868 records from Lens were considered in “.bib” file format. The “Bibliometrix” (Aria & Cuccurullo, 2017) library was imported to obtain “.xlsx” files from “.bib” files. These “.xlsx” files were used to conduct the proposed research work.

4. Methodology

Initially, a mergeDbSources function developed in Bibliometrix (2020) was implemented on the considered data. The algorithmic steps of the implemented mergeDBSources function are given in Table 4. This function provided only ISI-indexed records in the merged file, which can result in losing the important records from other databases. Hence, to address this problem, the step-by-step approach for merging the articles from multiple databases is shown in Figure 1. The process begins with importing all the required libraries in the R programming environment. The datasets collected from multiple databases are read. From these datasets, the columns with the common names are retrieved and saved as sub-datasets. These sub-datasets are combined to represent a single file. To deal with DOI case issues, the available DOIs are standardized by converting them into lowercase. By using these standardized DOIs, the duplicate records can be removed and the distinct values retained. Identify the records with multiple DOIs and/or titles and remove them to have a merged dataset. The detailed algorithmic steps of the proposed methodology are given in Table 5.

5. Rationale of the Proposed Research Work

Several issues are identified while merging the data from multiple databases. The proposed work is focused to solve such issues and simplify the merging process by the development of an open-source reproducible preprocessing pipeline.
  • Issue 1: By using the mergeDbSources function, only Institute for Scientific Information (ISI) records (fetched from WoS database) are prioritized. To avoid this consequence, the authors have developed an open-source reproducible preprocessing pipeline to get records from all databases with equal priority.
  • Issue 2: DOIs are recorded in small and capital cases in multiple databases. All DOIs are converted to records in small cases for better identification of duplicate records.
  • Issue 3: The other issue recognized is that multiple DOIs are identified in one record, which may affect the identification of duplicates. Hence, the records with multiple DOIs are identified and eliminated from the merged dataset.

6. Results and Discussion

After the implementation of mergeDBSources, it has been observed that it provided only the ISI database articles in the final merged dataset instead of considering the records from Scopus and Lens, which is shown in Figure 2 and Figure 3. From these figures, it is evident that the dataset contains the records indexed only with ISI after the merging. To better understand the overall dataset, the records at the beginning are shown in Figure 2, and the records at the end are shown in Figure 3. In total, there are 2084 records in the merged file. The details for Figure 2 and Figure 3 are available in the folder named “Results_with_mergeDBSources_function” in the Supplementary Materials. Duplicate DOIs with different cases are shown in Figure 4. This case difference of the same DOI will affect the deduplication of records in the dataset. In Figure 4, the DOIs, 10.12694/SCPE.V21i1.1600 and 10.12694/scpe.v21i1.1600 have different cases. The detailed presentation of the same DOIs with different cases is presented in Figure 5 and Figure 6. The uppercase DOI from Scopus is shown in Figure 5. The lowercase DOI from WoS is shown in Figure 6.
While observing the dataset during merging process, multiple DOIs are identified in some records as shown in Figure 7, the column titled “DI” has multiple DOIs in each row. For example, “10.1007/s13755-022-00203-w 10.1002/ima.22830” are recorded in a single row, which leads to inconsistency. Accordingly, multiple titles are also present in the dataset, depending on the multiple DOIs. In Figure 8, for example, the column titled “TI” is observed with two different titles “BTC-FCNN: FAST CONVOLUTION NEURAL NETWORK FOR MULTI-CLASS BRAIN TUMOR CLASSIFICATION A MODIFIED THRESHOLD SCORE-BASED MULTILEVEL THRESHOLDING SEGMENTATION TECHNIQUE FOR BRAIN MAGNETIC RESONANCE IMAGES USING OPPOSITION-BASED LEARNING HYBRID RICE OPTIMIZATION ALGORITHM” in the same record. Once multiple DOIs are identified, they are removed from the dataset because they are invalid for comparing records from different databases, depending on the DOI. Along with the multiple DOIs, multiple titles are also removed. Hence, a merged dataset without noisy data is available as presented in Figure 9. The details for Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 are available in the folder named “Results_brain_tumor_data” in the Supplementary Materials.
The summary of quantitative details for the present data (brain tumor) is provided in Table 6. The details for Table 6 are available in the folders named “Source_Brain Tumor” and “Results_brain_tumor_data” in the Supplementary Materials. To validate the performance of the proposed methodology, a new large set of data on the topic “Parkinson’s disease” is considered. The summary of quantitative details for these data is provided in Table 7. The details for Table 7 are available in the folders named “Source_Data_Parkinson’s Disease” and “Results_Parkinsons_disease_data” in the Supplementary Materials. From these tables, it is to be noted that the processing time for both the present data and a new large set of data is in seconds, which reflects the good performance of the proposed preprocessing pipeline. In the processing time, the start and end timings are considered based on the system time and the difference between these timings provides the actual processing time taken to complete the proposed preprocessing. In Table 6, the time difference observed is 2.199827 s, and in Table 7, the time difference observed is 21.31339 s.
During the preprocessing of a new large set of data, it is observed that there are only multiple DOIs, and there are no multiple titles. Although there are no multiple titles, it still creates ambiguity in choosing the right DOI for a particular record. Hence, those records are also to be removed to avoid the noise in the data. It is evident from the two cases considered that there are fewer records with multiple DOIs, and the deletion of such records will not impact the size of the data in further analysis. Final key observations from the preprocessing are as follows.
  • Some records contain both multiple DOIs and multiple titles, which is very important to focus on.
  • Some records contain multiple DOIs but no multiple titles.

7. Conclusions

This paper addresses various issues related to the merging process of documents from multiple databases. It presents an effortless and accurate process to merge datasets from Scopus, WOS and Lens. In addition to these, researchers can include other databases if their columns are matched. This merging process is scalable and can handle large volumes of data in less time.
The key outcomes of this research work are mentioned as follows:
  • While performing the merging process, the proposed preprocessing pipeline removes duplicates from all the databases with equal priority by solving the issue of getting only ISI-indexed records.
  • DOIs are presented in different cases of alphabets in different databases. This case difference will affect the searching of duplicate records. Hence, to solve this issue, a function is implemented to convert all DOIs to lowercase.
  • While merging the records from multiple databases, multiple DOIs and titles were identified and removed from the final merged dataset.
    Finally, a merged dataset without the noisy data is available for conducting the required analyses.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/publications14020034/s1. Folder: Results_with_mergeDBSources_function; Folder: Source_Brain Tumor; Folder: Results_brain_tumor_data; Folder: Source_Data_Parkinson’s Disease; Folder: Results_Parkinsons_disease_data.

Author Contributions

Conceptualization, K.H., S.R. and Y.V.P.K.; methodology, K.P.P., G.P.R., and S.R.; software, N.M.; validation, Y.V.P.K.; formal analysis, K.H. and S.R.; investigation, K.P.P. and K.H.; resources, N.M.; data curation, G.P.R. and N.M.; writing—original draft preparation, K.H. and S.R.; writing—review and editing, K.P.P. and G.P.R.; visualization, G.P.R. and K.H.; supervision, Y.V.P.K.; project administration, Y.V.P.K.; funding acquisition, K.P.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The source files of the datasets and the obtained result datasets are provided as Supplementary Files.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
DOIDigital Object Identifier
ISIInstitute for Scientific Information
PRISMAPreferred Reporting Items for Systematic Reviews and Meta-Analyses
SJRSCImago Journal Rank
SNIPSource Normalized Impact per Paper
WoSWeb of Science

References

  1. Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4), 959–975. [Google Scholar] [CrossRef] [Scilit]
  2. Berga, M., Coelho, P., & Ochman, A. (2025). Top 21 data mining tools. Available online: https://www.imaginarycloud.com/blog/data-mining-tools (accessed on 21 June 2025).
  3. Bibliometrix. (2020). Comprehensive science mapping analysis. Available online: https://www.bibliometrix.org/ (accessed on 17 March 2026).
  4. Caputo, A., & Kargina, M. (2022). A user-friendly method to merge Scopus and Web of Science data during bibliometric analysis. Journal of Marketing Analytics, 10(1), 82–88. [Google Scholar] [CrossRef] [Scilit]
  5. Chansanam, W., & Li, C. (2025). KKU-BiblioMerge: A novel tool for multi-database integration in bibliometric analysis. Iberoamerican Journal of Science Measurement and Communication, 5(1), 1–16. [Google Scholar] [CrossRef] [Scilit]
  6. Clarivate. (2026). Web of science database. Available online: https://access.clarivate.com/login?app=wos (accessed on 17 May 2026).
  7. Culbert, J. H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M., Donner, P., & Mayr, P. (2025). Reference coverage analysis of OpenAlex compared to Web of Science and Scopus. Scientometrics, 130(4), 2475–2492. [Google Scholar] [CrossRef] [Scilit]
  8. Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., & Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of Business Research, 133, 285–296. [Google Scholar] [CrossRef] [Scilit]
  9. Echchakoui, S. (2020). Why and how to merge Scopus and Web of Science during bibliometric analysis: The case of sales force literature from 1912 to 2019. Journal of Marketing Analytics, 8(3), 165–184. [Google Scholar] [CrossRef] [Scilit]
  10. Elsevier. (2026). Scopus database. Available online: https://www.scopus.com/pages/home (accessed on 17 May 2026).
  11. Fang, Y., Li, X., Mohtar, T. M., & Chekima, B. (2025). A bibliometric analysis of work–family balance: Trends, themes, and future directions (2000–2024). Cogent Business & Management, 12(1), 2541041. [Google Scholar] [CrossRef] [Scilit]
  12. Giorgi, F. M., Ceraolo, C., & Mercatelli, D. (2022). The R language: An engine for bioinformatics and data science. Life, 12(5), 648. [Google Scholar] [CrossRef] [Scilit]
  13. Gong, Y., Liu, G., Xue, Y., Li, R., & Meng, L. (2023). A survey on dataset quality in machine learning. Information and Software Technology, 162, 107268. [Google Scholar] [CrossRef] [Scilit]
  14. Groos, O. V., & Pritchard, A. (1969). Documentation notes. Journal of Documentation, 25(4), 344–349. [Google Scholar] [CrossRef] [Scilit]
  15. Kara, B. C., Şahin, A., & Dirsehan, T. (2025). BibexPy: Harmonizing the bibliometric symphony of Scopus and Web of Science. SoftwareX, 30, 102098. [Google Scholar] [CrossRef] [Scilit]
  16. Kasaraneni, H., & Rosaline, S. (2024). Automatic merging of Scopus and Web of Science data for simplified and effective bibliometric analysis. Annals of Data Science, 11(3), 785–802. [Google Scholar] [CrossRef] [Scilit]
  17. Lens. (2026). Lens database. Available online: https://www.lens.org/ (accessed on 17 May 2026).
  18. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. [Google Scholar] [CrossRef] [Scilit]
  19. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. [Google Scholar] [CrossRef] [Scilit]
  20. Shi, Y., Tian, Y., Kou, G., Peng, Y., & Li, J. (2011). Optimization based data mining: Theory and applications. Springer. [Google Scholar] [CrossRef] [Scilit]
  21. Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021). The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics, 126(6), 5113–5142. [Google Scholar] [CrossRef] [Scilit]
  22. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207–222. [Google Scholar] [CrossRef] [Scilit]
  23. Turgel, I. D., & Chernova, O. A. (2024). Open science alternatives to Scopus and the Web of Science: A case study in regional resilience. Publications, 12(4), 43. [Google Scholar] [CrossRef] [Scilit]
  24. Vera-Baceta, M.-A., Thelwall, M., & Kousha, K. (2019). Web of Science and Scopus language coverage. Scientometrics, 121(3), 1803–1813. [Google Scholar] [CrossRef] [Scilit]
  25. Vieira, G. A., & Leta, J. (2024). biblioverlap: An R package for document matching across bibliographic datasets. Scientometrics, 129(7), 4513–4527. [Google Scholar] [CrossRef] [Scilit]
  26. Visser, M., Van Eck, N. J., & Waltman, L. (2021). Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. Quantitative Science Studies, 2(1), 20–41. [Google Scholar] [CrossRef] [Scilit]
  27. Zhu, J., & Liu, W. (2020). A tale of two databases: The use of Web of Science and Scopus in academic papers. arXiv, arXiv:2002.02608. [Google Scholar] [CrossRef] [Scilit]
Publications 14 00034 g001
Figure 1. Workflow of the merging process.
Figure 1. Workflow of the merging process.
Publications 14 00034 g001
Publications 14 00034 g002
Figure 2. Records indexed only with ISI in the merged dataset (beginning of the dataset).
Figure 2. Records indexed only with ISI in the merged dataset (beginning of the dataset).
Publications 14 00034 g002
Publications 14 00034 g003
Figure 3. Records indexed only with ISI in the merged dataset (end of the dataset).
Figure 3. Records indexed only with ISI in the merged dataset (end of the dataset).
Publications 14 00034 g003
Publications 14 00034 g004
Figure 4. DOIs in different cases.
Figure 4. DOIs in different cases.
Publications 14 00034 g004
Publications 14 00034 g005
Figure 5. DOI in uppercase in Scopus.
Figure 5. DOI in uppercase in Scopus.
Publications 14 00034 g005
Publications 14 00034 g006
Figure 6. DOI in lowercase in WoS.
Figure 6. DOI in lowercase in WoS.
Publications 14 00034 g006
Publications 14 00034 g007
Figure 7. Multiple DOIs.
Figure 7. Multiple DOIs.
Publications 14 00034 g007
Publications 14 00034 g008
Figure 8. Multiple DOIs and TIs.
Figure 8. Multiple DOIs and TIs.
Publications 14 00034 g008
Publications 14 00034 g009
Figure 9. Merged dataset without noisy data.
Figure 9. Merged dataset without noisy data.
Publications 14 00034 g009
Table 1. Some previous studies on the merging of bibliometric databases.
Table 1. Some previous studies on the merging of bibliometric databases.
ReferenceMerging
Process		Years
Considered		Databases
Considered
Visser et al. (2021)Manual2008–2017WoS, Scopus,
Crossref, Dimensions,
and Microsoft Academic
Singh et al. (2021)Manual2010–2018Scopus, WoS,
and Dimensions
Zhu and Liu (2020)Manual2004–2018Scopus, WoS
Caputo and Kargina (2022)Bibliometrix and Excel
(Automatic up to some extent)		Not availableScopus, WoS
Vera-Baceta et al. (2019)Advanced query in the database2018Scopus, WoS
Kasaraneni and Rosaline (2024)Automatic2022Scopus, WoS
Echchakoui (2020)Manual process,
Bibliometrix and Excel		1912–2019Scopus, WoS
Table 2. Common metadata fields available in different databases.
Table 2. Common metadata fields available in different databases.
SourceDOIDocument
Title		ISSNSourceDocument
Type		Authors’
Keywords		PublisherAuthors
ScopusDITISNSODTDEPUAU
WoSDITISNSODTDEPUAU
LensDITIISSNSODTDEPUAU
Table 3. The search strategy in various databases.
Table 3. The search strategy in various databases.
DatabaseSearch Strategy
WoS“brain tumor detection” OR “brain tumor classification” (All Fields)
and 2013 or 2014 or 2015 or 2016 or 2017 or 2018 or 2019 or 2020 or
2021 or 2022 or 2023 (Publication Years) and Article or Review Article
(Document Types) and English (Languages)
ScopusTITLE-ABS-KEY (“brain tumor detection” OR “brain tumor classification”)
AND PUBYEAR > 2012 AND PUBYEAR < 2024 AND (LIMIT-TO (DOCTYPE, “ar”)
OR LIMIT-TO (DOCTYPE, “re”) ) AND (LIMIT-TO (PUBSTAGE, “final”) )
AND (LIMIT-TO (SRCTYPE, “j”) ) AND (LIMIT-TO (LANGUAGE, “English”) )
LensScholarly Works (868) = (“brain tumor detection” OR “brain tumor classification”)
Filters: Year Published = (2013–2023) Publication Type = (journal article,
conference proceedings article) External ID Type = (DOI)
Table 4. Steps involved in mergeDBSources function.
Table 4. Steps involved in mergeDBSources function.
Steps
1. Import required R libraries to achieve the desired functionality.
list.of.packages ← c (“bibliometrix”, “tidyverse”, “readxl”, “writexl”, “openxlsx”)
2. Choose Scopus .bib file. read, convert, and save it as an Excel file
scopusbibFile ← file.choose ()
scopusbibFile ← convert2df (file = scopusbibFile, dbsource = ‘scopus’, format = “bibtex”)
scopus_ExcelFile ← write.xlsx (scopusbibFile, “scopus.xlsx”)
3. Choose WOS .bib file. read, convert, and save it as an Excel file
wosbibFile ← file.choose ()
wosbibFile ← convert2df (file = wosbibFile, dbsource = ‘isi’, format = “bibtex”)
wos_ExcelFile ← write.xlsx (wosbibFile, “wos.xlsx”)
4. Choose Lens .csv file. read, convert, and save it as an Excel file
lenscsvFile ← file.choose ()
lenscsvFile ← convert2df (file = lenscsvFile, dbsource = ‘lens’, format = “csv”)
lens_ExcelFile ← write.xlsx (lenscsvFile, “lens.xlsx”)
5. Merge the obtained three Excel files as a singe file (without duplicates) using mergeDBSources function.
file_merged ← mergeDbSources (scopusbibFile, wosbibFile, lenscsvFile, remove.duplicated = T)
write.xlsx (file_merged, “file_merged.xlsx”)
Table 5. Steps involved in the proposed work.
Table 5. Steps involved in the proposed work.
Steps
1. Import required R libraries to achieve the desired functionality.
list.of.packages ← c (“bibliometrix”, “tidyverse”, “readxl”, “writexl”, “openxlsx”, “stringr”, “janitor”)
2. Read Scopus, WoS, and Lens data Excel files and save them into the respective objects.
scopus_file ← file.choose ()
wos_file ← file.choose ()
lens_file ← file.choose ()
3. Retrieve column names from each file and save them into the respective objects.
cols_scopus ← colnames (scopus_file)
cols_wos ← colnames (wos_file)
cols_lens ← colnames (lens_file)
4. Retrieve common column names from these columns and save them in an object.
common_cols ← intersect (cols_scopus, cols_wos, cols_lens)
5. Select data from each file based on the common columns identified and save as a sub-dataset.
subdata_scopus ← select (scopus_file, common_cols)
subdata_wos ← select (wos_file, common_cols)
subdata_lens ← select (lens_file, common_cols)
6. Merge all sub-datasets.
merged_data ← rbind (rbind (subdata_scopus, subdata_wos), subdata_lens)
7. Convert DOIs to lowercase. The common column for DOI is DI.
This column name is used for case conversion.
merged_data$DI ← as.character (merged_data$DI)
merged_data$DI ← tolower (merged_data$DI)
8. Find the duplicate records based on DOI and remove them. Keep the distinct records.
# Duplicate records
dup_merged_data ← merged_data[duplicated (merged_data$DI), ]
cat (“No. of Duplicate Records =”, print (nrow (dup_merged_data)))
print (dup_merged_data)
write_xlsx (dup_merged_data, “Duplicates.xlsx”)
# Unique records
unique_data ← distinct (merged_data, DI, .keep_all = T)
write_xlsx (unique_data, “merged_data_after_deduplication.xlsx”)
9. Search for multiple DOIs and blanks in DI column using a pattern and remove the records
with multiple DOIs.
pattern_doi_alphabet ← c (“10.\\d4,9/[-._;()/:a-z0-9A-Z]+10.\\d4,9/[-._;()/:a-z0-9A-Z]+”)
# Searching for multiple DOIs
multiple_doi_alphabet ← multiple_doi_alphabet % > % remove_empty(“rows”)
write_xlsx (multiple_doi_alphabet, “multiple_doi_alphabet.xlsx”)
print (nrow (multiple_doi_alphabet))
pattern_doi_whitespace ← 10\\.[^\\ s]+\\ s+[^\\ s]+
multiple_doi_whitespace ← unique_data[str_detect (unique_data$DI, pattern_doi_whitespace), ]
multiple_doi_whitespace ← multiple_doi_whitespace % > % remove_empty(“rows”)
write_xlsx (multiple_doi_whitespace, “multiple_doi_whitespace.xlsx”)
print (nrow (multiple_doi_whitespace))
all_patterns ← paste (pattern_doi_alphabet, pattern_doi_whitespace, sep = “|”)
10. Save unique data after removing the records with multiple DOIs.
unique_data ← unique_data[!str_detect (unique_data$DI, all_patterns), ] write_xlsx (unique_data, “merged_data_after_removing_multiple_DOIs.xlsx”)
print (nrow (unique_data))
11. Tracking processing time.
end.time ← Sys.time ()
time.taken ← end.time − start.time
print (paste0 (“Start Time”, start.time))
print (paste0 (“End Time”, end.time))
print (time.taken)
Symbol # represents the comment for the statement.
Table 6. Summary of quantitative details for present data (brain tumor).
Table 6. Summary of quantitative details for present data (brain tumor).
DescriptionQuantitative Details
Number of records in Scopus file1138
Number of records in WOS file839
Number of records in Lens file868
Number of records in the merged file before deduplication2845
Number of duplicate records in the merged file910
Number of records in the merged file after deduplication1935
Number of records with multiple DOIs
(considering alphabets in DOI)		8
Number of records with multiple DOIs
(considering whitespace in DOI)		8
Number of records in the merged file after removing records
with multiple DOIs		1927
Total Processing time“Start Time 2026-05-17 06:08:07.319666”
“End Time 2026-05-17 06:08:09.519493”
Time difference of 2.199827 s
Table 7. Summary of quantitative details for a new large set of data (Parkinson’s disease).
Table 7. Summary of quantitative details for a new large set of data (Parkinson’s disease).
DescriptionQuantitative Details
Number of records in Scopus file19,999
Number of records in WOS file20,000
Number of records in Lens file10,000
Number of records in the merged file before deduplication49,999
Number of duplicate records in the merged file12,734
Number of records in the merged file after deduplication37,265
Number of records with multiple DOIs
(considering alphabets in DOI)		7
Number of records with multiple DOIs
(considering whitespace in DOI)		7
Number of records in the merged file after removing records
with multiple DOIs		37,258
Total processing time“Start Time 2026-05-17 06:22:16.413374”
“End Time 2026-05-17 06:22:37.726767”
Time difference of 21.31339 s
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Purna Prakash, K.; HimaJyothi, K.; Rosaline, S.; Venkata Pavan Kumar, Y.; Reddy, G.P.; Mukkapati, N. An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases. Publications 2026, 14, 34. https://doi.org/10.3390/publications14020034

AMA Style

Purna Prakash K, HimaJyothi K, Rosaline S, Venkata Pavan Kumar Y, Reddy GP, Mukkapati N. An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases. Publications. 2026; 14(2):34. https://doi.org/10.3390/publications14020034

Chicago/Turabian Style

Purna Prakash, Kasaraneni, Kasaraneni HimaJyothi, Salini Rosaline, Yellapragada Venkata Pavan Kumar, Gogulamudi Pradeep Reddy, and Naveen Mukkapati. 2026. "An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases" Publications 14, no. 2: 34. https://doi.org/10.3390/publications14020034

APA Style

Purna Prakash, K., HimaJyothi, K., Rosaline, S., Venkata Pavan Kumar, Y., Reddy, G. P., & Mukkapati, N. (2026). An Open-Source Reproducible Preprocessing Pipeline for Merging Bibliometric Data from Multiple Databases. Publications, 14(2), 34. https://doi.org/10.3390/publications14020034

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop