Towards Automated Meta-Analysis of Clinical Trials: An Overview

Abstract

Background: Nowadays, much research deals with the application of the automated meta-analysis of clinical trials through appropriate machine learning tools to extract the results that can then be applied in daily clinical practice. Methods: The author performed a systematic search of the literature from 27 September 2022–22 November 2022 in PUBMED, in the first 6 pages of Google Scholar and in the online catalog, the Systematic Review Toolbox. Moreover, a second search of the literature was performed from 7 January 2023–20 January 2023 in the first 10 pages of Google Scholar and in the Semantic Google Scholar. Results: 38 approaches in 39 articles met the criteria and were included in this overview. These articles describe in detail machine learning approaches, methods, and tools that have been or can potentially be applied to the meta-analysis of clinical trials. Nevertheless, while the other tasks of a systematic review have significantly developed, the automation of meta-analyses is still far from being able to significantly support and facilitate the work of researchers, freeing them from manual, difficult and time-consuming work. Conclusions: The evaluation of automated meta-analysis results is presented in some studies. Their approaches show positive and promising results.

1. Introduction

Today clinical trials are considered as an established experimental clinical tool suitable not only for evaluating the effectiveness of interventions, but also for supporting the conduct of an adequately designed systematic review [1]. In addition, meta-analysis is a systematic review of a focused topic in the literature that provides a quantitative estimate of the effect of a therapeutic intervention or exposure [2]. This effect is inferred from outputs usually from more than one previously published clinical trial. A meta-analysis is necessary for making correct medical decisions (such as prognosis, diagnosis, treatment, recording side effects in taking drugs, etc.). It is the prevailing method applied in clinical trials for generating qualitative and quantitative evidence and conclusions. Meta-analysis and synthesis of the results of clinical trials are gaining rapid momentum in the research to generate quantitative information [3]. Thus, clinical trials are at the forefront of clinical decision support.

In parallel, since in the present time the volume of clinical studies is increasing exponentially, automating their processing by applying machine learning (ML) is a great challenge and a dominant research topic.

The automation in the management of clinical studies refers to dealing with the individual processes related to the search, collection, selection, and extraction of results. In detail these tasks are the following: Design Systematic Search, Run Systematic Search, Deduplicate, Obtain full texts, Snowballing, Screen abstracts, Data extraction and Text Mining Tool, Automated bias assessments, Automated Meta-Analysis, Summarize/Synthesis of data (analysis), Write up, and Data Miner/Analysis of Data for General-Purpose [4].

More specifically, the meta-analysis is a systematic approach for understanding a phenomenon by analyzing the results of many previously published experimental studies. Unfortunately, the conventional meta-analysis requires a great deal of human effort, is labor-intensive, and vulnerable to human bias [2,5].

The task of the automated meta-analysis and synthesis of data is the part of their management in which the least research has been done in terms of their mechanical processing and automation [4].

The authors of many studies [6,7,8,9,10,11] demonstrated the feasibility and acceptance of semi-automated and automated meta-analyses yielding promising results. The outcomes of these studies suggest that automated meta-analysis through machine learning approaches reduces the time required for a meta-analysis without altering the expert confidence in methodological and scientific rigor. Moreover, these results suggest acceptance for risk assessment and improve the quality of reporting.

In this direction, the author of this article deals with the application of automated meta-analysis of clinical trials through appropriate machine learning tools to extract the results that can then be applied in daily clinical practice.

In addition, deep learning methods and tools as a subcategory of ML, are also included in this study. Deep learning based on learning data representations are part of the larger family of machine learning algorithms that use multiple layers to progressively extract higher-level features from the raw input [12].

The novelty of this overview is that until yet very few review articles have been published which describe all these mentioned frameworks, techniques, and tools alongside their applications in a complete and effective way in order to contribute to their further development and improvement.

Thus, initially, the author searched for relevant work and described it in detail below in Section 3.1.

More specifically, the author in this article performed an overview exploring the applied state-of-the-art ML methods, approaches, frameworks, and tools in automating the meta-analysis and synthesis of data extracted from clinical trials.

The main research questions were as follows:

• RQ1. What are the trends and key characteristics of studies showing automation in the meta-analysis and synthesis of clinical trial data.
• RQ2. What are the most common technologies, methods, tools, and software used in the meta-analysis and synthesis of data extracted from clinical trials.
• RQ3. What are the impacts that derive from the usage of the automation in the meta-analysis in clinical trials.
• RQ4. What are the challenges, guidelines, and obstacles to be addressed and what studies and research are proposed to achieve automation and maximum and reliable application of clinical trial results in daily medical practices.

The rest of this study is organized as follows: Section 2 discusses other relevant studies. Section 3 presents the materials and methods of this study. Section 4 summarizes the results. Section 5 discusses the key issues arising from this study. Section 6 concludes the study and presents future directions.

2. Related Work

There are many studies in the field of the management of studies and clinical trials and the extraction of their knowledge [4,13,14,15,16] but only a limited number deal with the automation of the meta-analysis task.

Wang et al. [12] conducted a review and assessment of 18 common deep learning frameworks and libraries (Caffe, Caffe2, Tensorflow, Theano including Keras Lasagnes and Blocks, MXNet, CNTK, Torch, PyTorch, Pylearn2, Scikit-learn, Matlab including MatconvNet Matlab deep learning and Deep learning toolbox, Chainer, Deeplearning4j) and introduced a large number of benchmarking data.

In order to provide a basis for comparing and selecting between software tools that support Systematic reviews, the authors of [17] performed a feature-by-feature comparison of Systematic reviews tools.

Finally, the Systematic Review Toolbox [9] is an online catalog of tools that support various tasks within the systematic review and wider evidence synthesis process. The updated version of the Systematic Review Toolbox was launched on 13 May 2022, with 235 software tools and 112 guidance documents included.

3. Materials and Methods

3.1. Study Design

In this study design the author used the overview approach [18]. An overview is a generic term used for “any summary of the literature” [19] that attempts to survey the literature and describe its characteristics. As such, it can be used for many different types of literature review, with differing degrees of systematicity. Overviews can provide a broad and often comprehensive summation of a topic area and, as such, have value for those coming to a subject for the first time [20]. They are also important in cases where either a subject is not yet mature and well-known enough to be treated with a thorough systematic review or there is not the necessary time to perform it.

Additionally, the forward and backward snowball method is used [21]. It has been proposed that in reviews of complex or heterogeneous evidence in the field of health services research, “snowball” methods of forward (citation) and backwards (reference) searching are powerful. This method allows researchers using the references and citations of an article to find specific literature on an issue quickly and easily.

3.2. Literature Search and Study Selection

The author performed a systematic search of the literature from 27 September 2022–22 November 2022 in PUBMED (http://www.pubmed.org, accessed on 27 December 2022), in the first 6 pages of Google Scholar and in the online catalog: Systematic Review Toolbox (http://www.systematicreviewtools.com/, accessed on 28 December 2022) using combinations of search strings (“automated meta-analysis” AND “trials”). Moreover, a second search of the literature was performed from 7 January 2023–20 January 2023 in the first 10 pages of Google Scholar and in the Semantic Google Scholar using combinations of search strings (“automated meta-analysis” OR “automatic meta-analysis”).

The author did not find records in clinical trials.gov, or in the COCHRANE library.

In addition, forward and reverse citation searches (snowball method) were performed for specific studies to ensure inclusion of the most relevant studies. Snowballing was undertaken, starting from the included citations and from the references of each article.

Restrictions are related to the language (only English articles are included).

In addition, studies involving tools and techniques for image management (e.g., [22,23,24,25,26,27]) were out of the scope of this study and excluded.

3.3. Data Screening

The data were screened in a two-stage review process (Figure 1) that the author performed, (a) initially excluding assignments based on the titles and their abstracts, and (b) then the remaining assignments were screened based on the reading of the full text of the article.

One researcher reviewed the articles.

3.4. Data Extraction and Analyses

The following data were extracted from the included studies:

• Bibliographic elements of the included articles:

  ■

  Name of the studies’ object

  ■

  Reference

  ■

  Title

  ■

  Year

  ■

  Author(s)

  ■

  Journal

• Characteristics of the studies’ object:

  ■

  Name studies’ object

  ■

  Domain

  ■

  Type

  ■

  (Not)Free/(Not)Open

  ■

  Source Code

  ■

  Method/Language

4. Results

There were 38 approaches in 39 articles that met the criteria and were included in this overview (

Table 1. The bibliographic elements of the included studies.

Name	Reference	Title	Year	Author(s)	Journal
A Logic of Meta-Analysis approach	[28]	Towards a Logic of Meta-Analysis	2020	Peñaloza, R	Proceedings of the International Conference on
Amamida R Package	[29]	Amanida: An R package for meta-analysis of metabolomics non-integral data	2022	Llambrich, Maria; Correig, Eudald; Gumà, Josep; Brezmes, Jesús; Cumeras, Raquel	Bioinformatics
Amazon SageMaker	[30,31]	Getting Started with Amazon SageMaker Studio: Learn to build end-to-end machine learning projects in the SageMaker machine learning IDE	2022	Hsieh, M	Packt Publishing Ltd.
Automated Meta-analysis of Biomedical Texts	[10]	Towards Automated Meta-analysis of Biomedical Texts in the Field of Cell-based Immunotherapy	2019	Devyatkin DA, Molodchenkov AI, Lukin AV et al.	Research and Methods
Caffe2	[32]	Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective	2018	Hazelwood, K; et al.	IEEE International Symposium on High Performance Computer Architecture
Causal Learning Perspective	[5]	Automated Meta-Analysis: A Causal Learning Perspective	2021	Cheng, L; Katz-Rogozhnikov, D A; Varshney, K R; others	arXiv preprint
CINeMA	[33]	CINeMA: An approach for assessing confidence in the results of a network	2020	Nikolakopoulou, Adriani; Higgins, Julian P T; Papakonstantinou, Theodoros; Chaimani, Anna; Del Giovane, Cinzia; Egger, Matthias; Salanti, Georgia	PLOS Medicine
DIAeT	[11]	Synthesizing evidence from clinical trials with dynamic interactive	2022	Sanchez-Graillet; Witte, Olivia; Grimm, Christian; Grautoff, Frank; Ell, Steffen; Cimiano, Basil; Philipp	J. Biomed. Semantics
dmetar	[34]	Doing Meta-Analysis with R: A Hands-On Guide	2021	Harrer, Mathias; Cuijpers, Pim; Furukawa, Toshi A; Ebert, David D	CRC Press
DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA)	[35]	Graphical enhancements to summary receiver operating characteristic plots to facilitate the analysis and reporting of meta-analysis of diagnostic test accuracy data	2021	Patel, Amit; Cooper, Nicola; Freeman, Suzanne; Sutton, Alex	Res Synth Methods
Keras	[36]	Introduction to keras. In Deep learning with Python	2017	Ketkar, Nikhil	Apress, Berkeley, CA
Meta-Essentials	[37]	Introduction, comparison, and validation of Meta-Essentials: A free and simple tool for meta-analysis	2017	Suurmond, Robert; van Rhee, Henk; Hak, Tony	Res Synth Methods
metafor	[38]	Conducting Meta-Analyses in R with the metafor Package	2010	Viechtbauer, Wolfgang	Journal of Statistical Software
MetaInsight	[39]	MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta	2019	Owen, Rhiannon K; Bradbury, Naomi; Xin, Yiqiao; Cooper, Nicola; Sutton, Alex	Res Synth Methods
metamisc	[40]	A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes	2019	Debray, Thomas Pa; Damen, Johanna Aag; Riley, Richard D; Snell, Kym; Reitsma, Johannes B; Hooft, Lotty; Collins, Gary S; Moons, Karel Gm	Stat. Methods Med. Res.
MetaXL	[41]	Advances in the meta-analysis of heterogeneous clinical trials I: The	2015	Doi, Suhail A R; Barendregt, Jan J; Khan, Shahjahan; Thalib, Lukman; Williams, Gail M	Contemp. Clin. Trials
Nested-Knowledge	[17]	Web-Based Software Tools for Systematic Literature Review in Medicine: A Review and Feature Analysis	2021	Cowie; Rahmatullah, Kathryn; Hardy, Asad; Holub, Nicole; Kallmes, Karl; Kevin	Nested Knowledge, Inc.
netmeta	[42]	Network Meta-Analysis using Frequentist Methods [R package netmeta version 0.9-8	2022	Rücker, Gerta; Krahn, Ulrike; König, Jochem; Efthimiou, Orestis; Davies, Annabel; Papakonstantinou, Theodoros; Schwarzer, Guido	CRAN package repository
OpenNN	[43]	Open NN: An Open Source Neural Networks C++ Library	2022	Lopez, Roberto	International Center for Numerical Methods in Engineering (CIMNE)
Pymeta	[44]	PyMeta	2018	Hongyong, Deng	PythonMeta Website
PythonMeta	[45]	PythonMeta 1.26	2018	Hongyong, Deng	PythonMeta Website
PyTorch	[46]	PyTorch	Not found	PyTorch–Linux Foundation
scikit-learn	[47]	scikit-learn	2016	Python Software Foundation	Python Software Foundation
ShinyMDE	[48]	ShinyMDE: Shiny tool for microarray meta-analysis for differentially expressed gene detection	2016	Shashirekha, H. L.; Wani, Agaz Hussain	HLS and team
Spark ML	[49]	Scaling Machine Learning with Spark: Distributed ML with MLlib, TensorFlow, and Pytorch	2023	Polak, A.	O’Reilly Media
TensorFlow	[50]	Learning TensorFlow: A Guide to Building Deep Learning Systems	2017	Hope, Tom; Resheff, Yehezkel S.; Lieder, Itay	O’Reilly Media
Torch	[51]	Torch7: A Matlab-like Environment for Machine Learning	2019	Collobert, Ronan; Kavukcuoglu, Koray; Farabet, Clement	Neural Information Processing Systems
Whyis	[52]	Developing Scientific Knowledge Graphs Using Whyis	2018	McCusker, J.P., Rashid, S.M., Agu, N., Bennett, K.P. and McGuinness, D.L.	SemSci
Comprehensive gene expression meta-analysis	[53]	A comprehensive gene expression meta-analysis identifies novel immune signatures in rheumatoid arthritis patients	2017	Afroz, S.; Giddaluru, J.; Vishwakarma, S.; Naz, S.; Khan, A.A.; Khan, N.	Frontiers in
NeuroSynth	[54]	Large-scale automated synthesis of human functional neuroimaging data	2011	Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TD	Nat. Methods
Text-mining the neurosynth corpus (NeuroSynth #2)	[55]	Text-mining the neurosynth corpus using deep boltzmann machines	2016	Monti R, Lorenz R, Leech R, Anagnostopoulos C, Montana G	2016 International Workshop on Pattern Recognition in Neuroimaging
Social brain (NeuroSynth #3)	[56]	The “social brain” is highly sensitive to the mere presence of social information: An automated meta-analysis and an independent study	2018	Tso, Ivy F; Rutherford, Saige; Fang, Yu; Angstadt, Mike; Taylor, Stephan F	PLoS One
MetaCyto	[57]	MetaCyto: A Tool for Automated Meta-analysis of Mass and Flow Cytometry Data	2018	Hu Z, Jujjavarapu C, Hughey JJ, Andorf S, Lee HC, Gherardini PF et al.	Cell Rep.
Automated meta-analysis of the ERP literature	[58]	Automated meta-analysis of the event-related potential (ERP) literature	2022	Donoghue T, Voytek B	Sci. Rep.
CancerMA	[59]	CancerMA: a web-based tool for automatic meta-analysis of public cancer microarray data	2012	Feichtinger J, McFarlane RJ, Larcombe LD	Database
CancerEST	[60]	CancerEST: a web-based tool for automatic meta-analysis of public EST data	2014	Feichtinger J, McFarlane RJ, Larcombe LD	Database
Research Method Classification	[61]	Research Method Classification with Deep Transfer Learning for Semi-Automatic Meta-Analysis of Information Systems Papers	2021	Anisienia A, Mueller RM, Kupfer A, Staake T	Proceedings of the Annual Hawaii International Conference on System Sciences
AUTOMETA	[62]	AUTOMETA: Automatic Meta-Analysis System Employing Natural Language Processing	2022	Mutinda FW, Yada S, Wakamiya S, Aramaki E	Stud. Health Technol. Inform.

).

These articles describe in detail ML approaches, methods, and tools that have been or can potentially be applied to the meta-analysis of clinical trials.

All articles in the review range from the years 2010–2023, most of which were identified during the years 2016–2022. (There were 3 articles in 2016, 4 in 2017, 6 in 2018, 4 in 2019, 2 in 2020, 5 in 2021, and 7 in 2022) (Table 1).

The dominant technologies used for the development and application of the automated meta-analysis are Python and R programming languages. Some studies also used Java, Excel, and either C++ or another version of it (i.e., C, ANSI C++, C++11). More rarely were found CUDA, Docker environment, Lua, and LuaJIT. In addition, some studies combined the use of several different technologies to achieve their goals (

Table 2. The characteristics of the studies’ objects.

Name	Domain	Type	(Not)Free (Not)Open	Source Code	Method/Language
A Logic of Meta-Analysis approach	General purpose	Approach	No need	Not supported	Not supported
Amamida R Package	Metabolomic studies	Package	Open source	(https://github.com/mariallr/amanida, accessed on 23 December 2022)	R package
Amazon SageMaker	General purpose	Tool	Not free	(https://aws.amazon.com/sagemaker/resources/, accessed on 23 December 2022)	Python
Automated Meta-analysis of Biomedical Texts	Biomedical	All	Not described	No need	MetaMap; Fasttext model; Eclat algorithm/Python package
Caffe2	General purpose	Framework	Open Source	(https://github.com/pytorch/pytorch, accessed on 22 December 2022)	Graph representation is shared among all backend implementation; C++ & Python API
Causal Learning Perspective	General purpose	Approach	No need	No need	Multiple Causal inference for automated Meta-Analysis (MCMA)
CINeMA	General purpose	Tool	Open source	(https://github.com/esm-ispm-unibe-ch/cinema, accessed on 25 December 2022)	Salanti approach; JavaScript, Docker, and R package
DIAeT	Evidence-based medicine (EBM)	Model/Method	Open source	(https://doi.org/10.5281/zenodo.5604516, accessed on 24 December 2022)	model Toulmin; Java
dmetar	General purpose	Package	Open source	(https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R, accessed on 25 December 2022; https://dmetar.protectlab.org/, accessed on 25 December 2022)	R package
DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA)	General purpose	Software	Open source	(https://github.com/CRSU-Apps/MetaDTA; https://crsu.shinyapps.io/dta_ma/, accessed on 25 December 2022)	R package
Keras	General purpose	Software	Open source	(https://keras.io/; https://github.com/keras-team/keras, accessed on 25 December 2022)	Python
Meta-Essentials	General purpose	Software	Open source	(https://www.erim.eur.nl/research-support/meta-essentials/download/, accessed on 26 December 2022; https://www.meta-essentials.com, accessed on 26 December 2022)	Excel files
metafor	General purpose	Software	Open source	(https://www.jstatsoft.org/article/view/v036i03, accessed on 27 December 2022)	R package
MetaInsight	General purpose	Web application	Not Open; Freely available	(https://crsu.shinyapps.io/metainsight, accessed on 22 December 2022)	Not described
metamisc	General purpose	Model/Method	Open source	(https://cran.r-project.org/web/packages/metamisc/index.html, accessed on 25 December 2022; https://github.com/smartdata-analysis-and-statistics/metamisc, accessed on 28 December 2022)	R package
MetaXL	Evidence-based medicine (EBM)	Software	Freely available	(http://www.epigear.com/index_files/metaxl.html, accessed on 27 December 2022)	Excel files
Nested-Knowledge	Evidence-based medicine (EBM)	Web application	Not free	(https://nested-knowledge.com/nest/qualitative/371, accessed on 26 December 2022)	Not described
netmeta	General purpose	Web application	Open source	(https://cran.r-project.org/web/packages/netmeta/index.html, accessed on 23 December 2022; https://github.com/guido-s/netmeta, accessed on 28 December 2022; https://link.springer.com/book/10.1007/978-3-319-21416-0, accessed on 26 December 2022; https://rdrr.io/cran/netmeta/src/R/netmeta.R, accessed on 26 December 2022)	R package
OpenNN	General purpose	Tool	Open source	(https://github.com/Artelnics/OpenNN, accessed on 27 December 2022; http://opennn.cimne.com/download.asp, accessed on 28 December 2022)	ANSI C++
Pymeta	Evidence-based medicine (EBM)	Tool	Not Open	(https://www.pymeta.com/, accessed on 28 December 2022	Python
PythonMeta	Evidence-based medicine (EBM)	Tool	Open source	https://pypi.org/project/PythonMeta/, accessed on 28 December 2022)	Python
PyTorch	Evidence-based medicine (EBM)	Tool	Open source	(https://github.com/pytorch/pytorch, accessed on 28 December 2022)	Python
scikit-learn	General purpose	Tool	Open source	(scikit-learn/scikit-learn: scikit-learn: machine learning in Python (github.com), accessed on 21 December 2022)	Python
ShinyMDE	genomics, molecular genetics	Tool	Not Open; Freely available	(https://hussain.shinyapps.io/App-1, accessed on 21 December 2022)	R package
Spark ML	General purpose	Tool	Open source	(https://github.com/apache/spark, accessed on 22 December 2022)	Java; Python; R
TensorFlow	General purpose	Tool	Open source	(https://github.com/tensorflow/tensorflow, accessed on 22 December 2022)	C++; Python
Torch	General purpose	Framework	Open source	(https://github.com/torch/torch7, accessed on 22 December 2022)	C++11; Lua; LuaJIT, C; CUDA and C++
Whyis	General purpose	All	Open	(https://whyis.readthedocs.io/en/latest/index.html, accessed on 22 December 2022; https://github.com/tetherless-world/whyis, accessed on 22 December 2022)	probabilistic knowledge graphs by using Stouffer’s Z-Method/ Python; Flask framework; Fuseki; SPARQL; Graph Store HTTP Protocol; FileDepot Python library
Comprehensive gene expression meta-analysis	Biomedical	Method	Open	No need	Weighted Z-method/ survcomp R package
NeuroSynth	Medical	Framework	Open	(https://github.com/neurosynth, accessed on 26 December 2022)	naïve Bayes classification
Text-mining the neurosynth corpus (NeuroSynth #2)	Medical	Method	No need	No need	unsupervised study/ Deep Boltzmann machines for text-mining
Social brain (NeuroSynth #3)	Medical	Method	No need	(http://neurosynth.org/analyses/terms/social/, accessed on 28 December 2022)	Regions Of Interest (ROIs) analysis
MetaCyto	Biomedical	Method	No need	(http://bioconductor.org/packages/release/bioc/html/MetaCyto.html, accessed on 28 December 2022)	clustering methods with a scanning method/R package
Automated meta-analysis of the ERP literature	Medical	Tool	Open	(https://erpscanr.github.io/, accessed on 28 December 2022; https://github.com/ERPscanr/ERPscanr, accessed on 28 December 2022)	text-mining and word co-occurrence analyses
CancerMA	Biomedical	Tool	Open	(http://www.cancerma.org.uk, accessed on 28 December 2022) (not found)	HTML/CSS; Twitter Bootstrapp; Javascript/jQuery; Perl; R package; Bioconductor package
CancerEST	Biomedical	Tool	Open	(http://www.cancerest.org.uk/help.html http://www.cancerest.org.uk, accessed on 28 December 2022) (not found)	HTML/CSS; Twitter Bootstrapp; Javascript/jQuery; Perl; R package; Bioconductor package
Research Method Classification	General purpose	Method	No need	No need	Support Vector Models
AUTOMETA	Medical	Approach	No need	No need	BERT-based model

).

The research for automation of meta-analyses in some studies is specialized to handle strictly specialized issues such as the biomedical domain (5 articles), evidence-based medicine (6 articles), genomics and molecular genetics (1 article), and medical (5 articles) and metabolomic domain (1 article) (Table 2).

According to the findings of this overview, the types of research most frequently encountered to achieve automated meta-analysis are: the development and implementation of appropriate tools (13 studies), the development and implementation of software (5 studies), and the development of appropriate models and methods (7 studies). More analytically this overview basically identified four types of applications related to supporting or developing an automated meta-analysis. These are the following:

A Framework or Tool: this category includes the development of an integrated framework or the development of a specific tool to support automated meta-analysis. Most of the studies included in this review fall into this category.

A Package or Software: this category includes the development of package software to support automated meta-analysis.

A Model, Method, or Approach: this category includes the development of models and/or methods in the field of automated meta-analysis.

A Web application: This category includes web-based applications that implement automated meta-analyses and are either already implemented or may potentially be implemented in the future in clinical studies as well.

Some of the applications may be included in more than one category. Moreover, some of them can be a complete implementation and include all of the above. More analytically, it is worth noting that 2 studies ([10,52]) fully and comprehensively deal with the topic under discussion here by presenting an integrated modeling and application framework (Table 2).

Below are briefly described the applications found in this overview as classified based on the above four categories.

4.1. Framework/Tool (Includes 16 Studies)

• Amazon SageMaker [30,31]

Description: Amazon SageMaker Studio is the first integrated development environment in the cloud for machine learning and is designed to integrate the following machine learning workflows: data preparation, feature engineering, statistical bias detection, automated machine learning, training, hosting, ML explainability, monitoring, and machine learning operations in one environment.

Features: The features available in Amazon SageMaker Studio include the following issues: build, train, and deploy machine learning models quickly using Amazon SageMaker; analyze, detect, and receive alerts relating to various business problems using machine learning algorithms and techniques; improve productivity by training and fine-tuning machine learning models in production.

Inputs: datasets; csv files; models.

Outputs: models; Python script; data flow; data.

• Caffe2 [32]

Caffe2 is Facebook’s in-house production framework for training and deploying large-scale machine learning models. Caffe2 is a deep learning framework that provides an easy and straightforward way for you to experiment with deep learning and leverage community contributions of new models and algorithms.

Features: Caffe2 focuses on several key features required by products: performance, cross-platform support, and coverage for fundamental machine learning algorithms and multi-layer perceptions. The design involves a modular approach, where a unified graph representation is shared among all backend implementations.

Inputs: Python and C++ files; models.

Outputs: everything.

• CINeMA [33]

Description: the Confidence in Network Meta-Analysis (CINeMA) approach is broadly based on the GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework, with several conceptual and semantic differences [5]. It covers the following domains: (i) within-study bias, (ii) reporting bias, (iii) indirectness, (iv) imprecision, (v) heterogeneity, and (vi) incoherence. The reviewer’s input is required at the study level. Then, CINeMA assigns judgments at three levels (no concerns, some concerns, or major concerns) to each domain. Judgments across domains can be summarized to obtain four levels of confidence (very low, low, moderate, or high) for each relative treatment effect.

Features: the CINeMA framework has been implemented in a freely available, user-friendly web application aiming to facilitate the evaluation of confidence in the results from network meta-analysis. The web application applies the Salanti approach and is programmed in JavaScript, uses Docker, and is linked with R; in particular, packages meta and netmeta are used, and an R package to calculate the contribution of studies in network meta-analysis treatment effects.

Inputs: csv files.

Outputs: outputs a downloadable report with a summary of the evaluations.

• OpenNN [43]

Description: OpenNN is a software library that implements neural networks, a major area of deep learning research.

Features: OpenNN includes: a multilayer perceptron software implementation; many examples; unit testing.

Inputs: C++ code.

Outputs: data; plots.

• Pymeta [44]

Description: Pymeta is an online meta-analysis tool, as a web-based application it is created and supported with PythonMeta, a Python package of meta-analysis.

Features: performs: combining effect measures (OR, RR, RD for count data and MD, SMD for continuous data); heterogeneity testing (the Q/Chi-square test); subgroup analysis; cumulative meta-analysis; and sensitivity analysis (one or two factors).

Inputs: Python code.

Outputs: data; plots; bar-lines.

• PythonMeta [45]

Description: PythonMeta package performs the meta-analysis on an open-access dataset from COCHRANE..

Features: meta-analysis package by and for the Python language. This module was designed to perform some evidence-based medicine (EBM) tasks, such as: combining effect measures (OR, RR, RD, MD, SMD), heterogeneity testing (the Q/Chi-squared test), subgroup analysis, and plots (forest, funnel, etc.).

Inputs: dataset from COCHRANE.

Outputs: data; plots.

• PyTorch [46]

Description: PyTorch is a deep learning research platform that provides maximum flexibility and speed.

Features: PyTorch is a library that consists of the following components: torch which is a Tensor library with strong GPU support; torch.autograd which is a tape-based automatic differentiation library; torch.jit which is a compilation stack; torch.nn which is a neural networks library deeply integrated with autograd designed for maximum flexibility; torch.multiprocessing which is Python multiprocessing useful for data loading and Hogwild training; torch.utils which is a DataLoader; and other utility functions.

Inputs: Python code.

Outputs: data; plots.

• scikit-learn [47]

Description: Scikit-learn is a Python library.

Features: it includes functions that are integral to the machine learning pipeline such as data preprocessing steps, data resampling techniques, evaluation parameters, and search interfaces for tuning/optimizing an algorithm’s performance.

Inputs: datasets.

Outputs: data; plots.

• ShinyMDE [48]

Description: ShinyMDE supports an automated meta-analysis of gene expression data facilitating screening and downloading the results.

The tool handles processed and raw data generated from the most widely used data platforms. In addition, the tool provides users with an option of choosing the method of their choice from the list for meta-analysis.

Features: ShinyMDE consists of a web interface, a standalone version to work remotely, and a database holding GPL files. The general workflow of the ShinyMDE system visualizes the steps of the meta-analysis, which is carried out automatically once a user submits the data and selects the necessary parameters.

Inputs: CSV and txt files.

Outputs: data; web.

• Spark ML [49]

Description: Spark is a unified analytics engine for large-scale data processing. The package spark.ml aims to provide a uniform set of high-level APIs that help users create and tune practical machine learning pipelines.

Features: Spark provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

Inputs: Java and Python code.

Outputs: data results; plots.

• TensorFlow [50]

Description: TensorFlow 2 is an end-to-end, open-source machine learning platform which operates as an infrastructure layer for differentiable programming.

Features: It combines the following key abilities: efficiently executing low-level tensor operations on the CPU, GPU, or TPU; computing the gradient of arbitrary differentiable expressions; scaling computation to many devices; and exporting programs (“graphs”) to external runtimes such as servers, browsers, and mobile and embedded devices.

Inputs: Python and C++ code.

Outputs: data; graphs; and plots.

• Torch [51]

Description: Torch is not in active development. The functionality provided by the C backend of Torch, which are the TH, THNN, THC, and THCUNN libraries, is actively extended and re-written in the ATen C++11 (which is a new version of C++) library (https://github.com/pytorch/pytorch/tree/master/aten, accessed on 30 December 2022). ATen exposes all operators you would expect from torch7, nn, cutorch, and cunn directly in C++11 and includes additional support for sparse tensors and distributed operations. Thus, Torch is the main package in Torch7 where data structures for multi-dimensional tensors and mathematical operations over these are defined. Moreover, it provides many utilities for accessing files, serializing objects of arbitrary types, and other useful utilities.

Features: Torch includes the following libraries: Tensor Library, File I/O Interface Library, and Useful Utilities. Moreover, Torch7 is a versatile numeric computing framework and machine learning library that extends Lua. Its goal is to provide a flexible environment to design and train learning machines. Flexibility is obtained via Lua, an extremely lightweight scripting language. Torch7 can easily be interfaced to third-party software thanks to Lua’s light interface.

Inputs: code; scripts.

Outputs: data results; plots.

• NeuroSynth [54]

Description: In this article the authors describe and validate an automated brain mapping framework that uses text mining, meta-analysis, and machine learning techniques to generate a large database of mappings between neural and cognitive states.

Features: in this article the authors describe and validate a framework for the brain mapping, NeuroSynth, that takes an instrumental step towards automated large-scale synthesis of the neuroimaging literature. NeuroSynth combines text mining, meta-analysis, and machine learning techniques (naïve Bayes classification) to generate probabilistic mappings between cognitive and neural states.

Inputs: data.

Outputs: data; plots.

• Automated meta-analysis of the ERP literature [58]

Description: event-related potentials (ERP) are a common signal of analysis in medicine experiments, with a large existing literature of ERP-related work. This work uses automated literature collection and the text-mining of research articles to summarize the ERP literature, examining patterns and associations within and between components.

Features: all code for this project is written in the Python programming language and uses the LISC [63] Python tool to collect and analyze scientific literature. The data is collected from Pubmed, a database of biomedical literature. From there, the authors use text-mining and word co-occurrence analyses to derive data-driven summaries for each ERP, as well as to compare across these profiles to summarize patterns across the literature.

Inputs: Python code.

Outputs: data; plots.

• CancerMA [59]

Description: CancerMA is an online, integrated bioinformatic pipeline for automated identification of novel candidate cancer markers/targets. CancerMA operates by means of meta-analyzing expression profiles of user-defined sets of biologically significant and related genes across a manually curated database of 80 publicly available cancer microarray datasets covering 13 cancer types. A simple-to-use web interface allows experts to initiate new analyses as well as to view and retrieve the meta-analysis results.

Features: CancerMA consists of a web interface, a set of pipelined analyses, and two relational databases, one holding the analysis data for each user and another one holding the gene annotation data.

Inputs: R code and data.

Outputs: data; plots.

• CancerEST [60]

Description: CancerEST was developed as a user-friendly and intuitive tool to compute cancer marker/target potential as well as to obtain comprehensive expression profiles and information about the tissue specificity for genes of interest to biologists/clinicians. The CancerEST web interface for viewing the analysis results consists of three sections: the overview, the information, and the results section. The overview section provides basic information about the submitted job and a brief explanation on how to interpret the results.

Features: CancerEST consists of a web interface, pipelined analyses, and three relational databases; one holding the analysis data, one holding the Unigene data, and another one holding the gene annotation data.

Inputs: R code and data.

Outputs: data; plots.

4.2. Package/Software (Includes 7 Studies)

• Amamida R Package [29]

Description: the Amanida R package allows a meta-analysis of metabolomics data, combining the results of different studies addressing the same question. The Amanida package contains a collection of functions for computing a meta-analysis in R only using significance and effect size. It covers the lack of data provided on metabolomic studies. Amanida also computes qualitative meta-analysis.

Features: Amanida is a meta-analysis approach using only the most reported statistical parameters in this field: P-value and fold-change. The P-values are combined via Fisher’s method and fold-changes by averaging, both weighted by the study size.

Inputs: supported files are csv, xls/xlsx, and txt.

Outputs: the Amanida package includes several visualization options: a volcano plot for quantitative results, a vote plot for total regulation behaviors for each compound, and an explore plot of the vote-counting results.

• dmetar [34]

Description: the dmetar package using the meta, metafor, netmeta, and meta-SEM packages as a base is provided as a companion to the R package to support more functions that improve the workflow of a meta-analysis.

Features: dmetar provides tools for various stages of the systematic review process, e.g., visualizing the risk of bias, standard inverse variance meta-analysis, network meta-analysis, three-level meta-analysis, and exploration of the between-study heterogeneity.

Inputs: R code.

Outputs: data results.

• DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA) [35]

Description: MetaDTA is an online interactive application for conducting the meta-analysis of diagnostic test accuracy studies (DTA), requiring no specialist software for the user to install, but leveraging established analysis routines (specifically the lme4 package in R).

Features: the application allows users to upload their own data, customize SROC plots, obtain statistics such as sensitivity and specificity, and conduct sensitivity analyses. All plots and tables are downloadable.

The tool is interactive and uses an intuitive “point and click” interface and presents results in visually intuitive and appealing ways. It is hoped that this tool will assist those in conducting a DTA meta-analysis who are not statistical experts, and, in turn, increase the relevance of published meta-analyses, and in the long term contribute to improved healthcare decision making as a result.

Inputs: csv files.

Outputs: data results; plots.

• Keras [36]

Description: Keras is a library that provides highly powerful and abstract building blocks to build deep learning networks. It is a deep learning API written in Python, running on top of the machine learning platform TensorFlow. It was developed with a focus on enabling fast experimentation.

Features: Keras supports both CPU and GPU computation and is a great tool for quickly prototyping ideas. It reduces the developer’s cognitive load to free him up to focus on the parts of the problem that really matter. It also adopts the principle of the progressive disclosure of complexity. Finally, it provides industry-strength performance and scalability.

Inputs: Python code.

Outputs: data results.

• Meta-Essentials [37]

Description: this is a free excel tool for meta-analysis that facilitates the integration and synthesis of effect sizes from different studies.

Features: Meta-Essentials automatically calculates effect sizes from a wide range of statistics and can be used for a wide range of meta-analysis applications, including subgroup analysis, moderator analysis, and publication bias analysis.

Inputs: xls files.

Outputs: xls files.

• metafor [38]

Description: The metafor package is a free and open-source add-on for conducting meta-analyses with the statistical software environment R.

Features: the package consists of a collection of functions that allow the user to calculate various effect sizes or outcome measures, fit equal-, fixed-, random-, and mixed-effects models to such data, carry out moderator and meta-regression analyses, and create various types of meta-analytical plots.

Inputs: R code.

Outputs: data results; plots.

• MetaXL [41]

Description: MetaXL is an add-in for meta-analysis in Microsoft Excel for Windows. It supports all major meta-analysis methods, plus, uniquely, the inverse variance heterogeneity and quality effects models. Starting with v4.0, it also implements a powerful, yet easy-to-use way to perform network meta-analyses. Output is in table and graphical formats.

Features: MetaXL employs almost the same meta-analysis methods that can be accessed in general statistical packages and in dedicated meta-analysis software.

Inputs: Excel files.

Outputs: Excel files.

4.3. Model/Method/Approach (Includes 10 Studies)

• A Logic of the Meta-Analysis approach [28]

Description: in this position paper, the authors propose, the first as far as is known, an approach for automated reasoning in meta-analyses.

Features: thus, they considered the first steps towards a logic for performing automated meta-analysis based on a finite class of confidence intervals and subset relationships as background knowledge.

Inputs: A machine learning problem.

Outputs: The solution of the problem.

• Causal Learning Perspective [5]

Description: this work demonstrates the efficacy of using causal models to process the outputs of natural language processing (NLP)-based data extraction and achieve the goal of meta-analysis. In this article the authors initially extract information from scientific publications written in natural language. Sequently, from a novel causal learning perspective, they then propose to frame automated meta-analysis—based on the input of the first step—as a multiple causal inference problem where the summary effect is obtained through intervention.

Features: the authors of this article worked toward automating meta-analysis with a focus on controlling for risks of bias. Thus, they proposed the Multiple Causal inference for automated Meta-Analysis (MCMA). MCMA employs existing NLP systems for the extraction of risks of bias and therapeutic association, which are then used to estimate the summary therapeutic association across several Randomized Clinical Trials (RCTs). More analytically, from this perspective, the authors suggest to frame automated meta-analysis—based on the input of the first step—as a multiple causal inference problem where the summary effect is obtained through intervention. Built upon existent efforts for automating the primary steps of the meta-analysis, the proposed approach achieves the goal of automated meta-analysis and largely reduces the human effort involved.

Inputs: free text and data.

Outputs: data; plots.

• DIAeT [11]

Description: DIAeT (Dynamic Interactive Argumentation Trees) is a method of synthesizing the evidence available in clinical trials in an ad-hoc and on-demand manner that automatically organizes such evidence into a hierarchical argument that recommends a treatment as superior to another based on a series of key dimensions corresponding to the clinical points of interest.

Features: the DIAeT method is an argumentation-based method that contributes to supporting the synthesis of clinical trial evidence. A limitation of the method is that it relies on a manually populated knowledge base. This problem can be addressed by applying natural language processing methods to extract relevant information from publications. The method has been implemented as a web tool.

Inputs: SPARQL queries.

Outputs: results on the web.

• metamisc [40]

Description: the metamisc package includes the meta-analysis of diagnostic and prognostic modeling studies. In addition, it summarizes estimations of prognostic factors, diagnostic test accuracy, and prediction model performance. Finally, it validates, updates, and combines published prediction models. It also develops new prediction models with data from multiple studies.

Features: This R package deals with the incomplete availability of study-specific results (performance estimates and their precision), and produces summary estimates of the c-statistic and the observed: the expected ratio and the calibration slope. Furthermore, it tackles the implementation of frequentist and Bayesian meta-analysis methods and proposes novel empirically based prior distributions to improve the estimation of between-study heterogeneity in small samples.

Inputs: R code.

Outputs: data results.

• Comprehensive gene expression meta-analysis [53]

Description: this approach plans a comprehensive gene expression meta-analysis that labels novel immune signatures in patients with rheumatoid arthritis. This pattern suggests meta-analysis to recognize novel gene signatures that take care of providing mechanistic visions into disease initiation, progression, and the development of better therapeutic attacks.

Features: the aim of the meta-analysis method was firstly to extract the intersected genes, then to exclude genes with inconsistent expression, and finally to test them for significance. The weighted Z-method was used to combine the individual q-values of each gene [64] and was implemented using an R package (https://github.com/bhklab/survcomp, accessed on 28 December 2022), [65].

The meta-analysis algorithm was implemented using R.

Inputs: R code.

Outputs: data.

• Text-mining the neurosynth corpus (NeuroSynth #2) [55]

Description: in this work the authors demonstrate that an unsupervised study of the NeuroSynth text corpus using Deep Boltzmann Machines (DBMs) can be effectively employed to learn the distribution of the text corpus. The results of this study show some of the clusters obtained when k-means clustering is applied to word embeddings obtained from the DBM model. The clusters display clear semantic context.

Features: a two-layer DBM was employed consisting of a visible layer of multinomial visible units followed by two binary hidden layers. During pre-training and model selection, DBMs were trained. Briefly, annealed importance sampling was employed to estimate the partition function for each DBM. Thus, the proposed DBM model can be used to obtain both word as well as document embeddings in a high-dimensional vector space.

Inputs: data.

Outputs: data; plots.

• Social brain (NeuroSynth #3) [56]

Description: how the human brain processes social information is an increasingly researched topic in psychology and neuroscience, advancing our understanding of basic human cognition and psychopathologies. In this study, the authors investigated whether these brain regions are evoked by the mere presence of social information using an automated meta-analysis and confirmatory data from an independent study. Results of 1000 published fMRI studies containing the keyword of “social” were subject to an automated meta-analysis. The social/non-social contrast in the independent study showed a strong resemblance to the NeuroSynth map. The Region Of Interest (ROI) analyses revealed that a social effect was credible in most of the NeuroSynth regions in the independent dataset.

Features: the first part of the analyses of this study aimed to identify the brain regions that have shown significant activation in published fMRI studies with a prominent social element in the literature. Using the keyword “social” yielded 1000 published fMRI studies to include in an automated meta-analysis on neurosynth.org. The authors used the reverse inference map of the results of the automated meta-analysis, which represent z-scores corresponding to the likelihood that the term “social” is used in a study given the presence of the reported activation. The significant brain regions showing up in the reverse inference map represent those that are more likely to be reported in “social” studies than in “non-social” studies.

Inputs: data.

Outputs: data; plots.

• MetaCyto [57]

Description: the authors of this article developed MetaCyto for the automated meta-analysis of flow cytometry and mass spectrometry (CyTOF) data.

Features: by combining clustering methods with a scanning method, MetaCyto can identify commonly labeled subsets of cells, thereby enabling meta-analysis. Thus, the application of MetaCyto to a set of cytometric studies allowed for the identification of cell populations that show differences in abundance between demographic groups.

Inputs: R package.

Outputs: data; plots.

• Research Method Classification [61]

Description: this research work presents a prototype that applies deep transfer learning to predict the research methods in scientific publications, which facilitates an automatic discovery of crucial research information from large numbers of publications. The current state-of-the-art for classification of research methods uses Support Vector Models (SVMs).

This article provides the following research contributions: (a) developing an artifact that uses deep transfer learning and outperforms the state-of-the-art of research method classification, (b) using full papers and classifying them into predefined research methods, and (c) demonstrating the performance based on an extensive Information Systems corpus.

Features: the proposed approach outperforms state-of-the-art research method classification that deploys the Support Vector Model (SVM). The proposed deep transfer learning models can lead to a better recognition of research methods than shallower word embedding approaches such as word2vec or GloVe. The results illustrate the potential of establishing semi-automated methods for meta-analysis.

Inputs: free text and data.

Outputs: data.

• AUTOMETA [62]

Description: the proposed system for automating meta-analysis employs existing natural language processing methods for identifying Participants, Intervention, Control, and Outcome (PICO) elements. This system can perform advanced meta-analyses by parsing numeric outcomes to identify the number of patients having certain outcomes. In this study, the authors used the BERT-based approach which is a general-purpose language model trained on a large dataset and uses an attention mechanism that learns contextual relations between words in a text.

Features: the proposed system consists of four major components: crawling PubMed articles, NLP module, creating structured data, and aggregation and visualization. First, a user queries the PubMed database and related articles are returned. Abstracts are then extracted from the articles and passed to the NLP module for preprocessing and extraction of PICO elements. The extracted data are then converted into a structured form. It also parses numeric texts to identify the number of patients having certain outcomes. Identification of the number of patients having certain outcomes is important for statistical analysis to determine the effectiveness of an intervention.

Inputs: free text and data.

Outputs: data.

4.4. Web Application and Integrated Systems (Includes 5 Studies)

• Automated meta-analysis of biomedical texts [10]

Description: in this research article the authors present the results of the automated analysis of the data extracted from abstracts of scientific articles available in PubMed. These results demonstrate the associations between types of tumors and the most used methods for their cell-based immunotherapy.

Features: the proposed method automates the meta-analysis by standardizing the process in a series of steps. In summary, the following are mentioned: (a) crawling abstracts from Pubmed via the Scrapy based web-crawler, (b) rich linguistic features extraction by using the ISANLP framework which is a Python library to obtain the morphology, syntax parsing, and semantic role labeling features [66], (c) combining tumor and cell dictionaries and morphology-based rules to extract entity candidates from the abstracts, d) using syntactic relations and constructing all their possible combinations and applying models (e.g., UMLS Metathesaurus, MetaMap [67] Fasttext model [68]) to map the terms, (e) using syntactic relations and semantic roles to reveal the links between entities and their roles in the sentence, (f) applying a pre-trained sequence-labeling machine learning model to filter uninformative entity candidates, and g) computing co-occurrence statistics and mining associative rules for the extracted entities [69,70] to obtain stable combinations of tumors, therapy, and cell types. We used the Eclat algorithm [71] because of its scalability.

Inputs: biomedical texts; abstracts of scientific articles available in PubMed; Python code.

Outputs: data; plots.

• MetaInsight [39]

Description: MetaInsight is a new tool that is freely available and that conducts network meta-analysis (NMA) via the web.

Features: MetaInsight is a web-based tool allowing users with only standard internet browser software to be able to conduct NMAs using an intuitive “point and click” interface and present the results using visual plots.

Inputs: .csv files.

Outputs: data results; plots.

• Nested-Knowledge [72]

Description: Nested Knowledge offers a comprehensive software platform for systematic literature review and meta-analysis.

Features: the software is composed of two parts which work in tandem. Search, screen, tag, and extract data with AutoLit, and visualize, analyze, publish, and share insights with Synthesis.

Inputs: RIS files.

Outputs: data results; plots; RIS or nBIB files.

• netmeta [42]

Description: an R package for frequentist meta-analysis, this has a comprehensive set of functions providing a lot of methods for network meta-analysis.

Features: this package supports a comprehensive set of functions providing frequentist methods for network meta-analysis such as: the frequentist network meta-analysis; the net heat plot and design-based decomposition of Cochran’s Q; the measurements of characterizing the flow of evidence between two treatments; the ranking of treatments based on the frequentist analogue of SUCRA; the partial order of treatment rankings and the Hasse diagram; and the contribution matrix, etc.

Inputs: R code.

Outputs: data results.

• Whyis [52]

Description: Whyis is the first framework for creating custom provenance-driven knowledge graphs. Whyis knowledge graphs are based on nanopublications, which simplify and standardize the production of structured, provenance-supported knowledge in knowledge graphs.

To create probabilistic knowledge graphs, Whyis [52] implements a method of automated meta-analysis. The authors refined the methods used in [73] by using Stouffer’s Z-Method [64].

Features: Whyis is written in Python using the Flask framework. The RDF database used by default is Fuseki. Whyis uses the SPARQL Query, Update, and Graph Store HTTP Protocol. Storage is provided using the FileDepot Python library to provide the file-based persistence of nanopublications and uploaded files. Whyis also relies on Celery which is a task queuing system that can be scaled by adding more task workers on remote machines. Thus, knowledge graph developers create their knowledge graphs by generating a Python module that contains the configuration, templates, and code adapted to their purposes.

Inputs: Python code and script modules.

Outputs: Views; data; plots.

5. Discussion

5.1. Purpose of This Study

The aim of this article is to discover the most modern and complete tools used to automate the conduct of meta-analyses of clinical trials. In this way, it will contribute, on the one hand, to the identification and promotion of the most suitable candidates, and on the other hand, to the development of research in this field.

5.2. Benefits Arising from Automated Meta-Analysis

The evaluation of automated meta-analysis results is presented in some studies [5,9,10,11,29,37]. Their approaches show positive and promising results in the feasibility, acceptance, reliability, and time consumption. More analytically, the most important benefits are the ability to process large data sets in shorter times without altering expert confidence in the methodological and scientific rigor [6].

5.3. Comparison of Systems and Tools Currently Available

Built upon existent efforts for automating the basic steps of meta-analysis, the proposed approaches achieve the goal of automated meta-analysis and largely reduce the human effort involved [5].

However, although important steps have been taken to date, currently there is no application that can fully replace the human effort in conducting a systematic review to draw conclusions from clinical trials. Thus, while the other tasks of a systematic review have significantly developed, the automation of meta-analyses is still far from being able to significantly support and facilitate the work of researchers, freeing them from manual, difficult, and time-consuming work.

At the same time, it is worth noting that most of the tools are either open source or some are freely available (Table 2). Therefore, the strengthening of research in this field should be important in the immediate future.

The benefits of automating meta-analysis are expected to be particularly important in all areas of evidence-based medicine and especially in cutting edge areas of medical research such as gene therapy and cancer treatment.

5.4. Limitations of This Study

In addition, this overview has some methodological limitations. Initially the author had difficulty in identifying suitable articles. This limitation was partially addressed using snowballing methods. Secondly, the author included articles written only in English.

6. Conclusions and Future Directions

ML is the fastest growing field in computer science, and Health Informatics is amongst the greatest application challenges, providing significant benefits in improved medical prognosis, diagnosis, and pharmaceutical development [74].

Meta-analysis is a systematic approach for understanding a wonder by resolving the results of many previously published exploratory studies. It is used mainly to extract knowledge and decisions about the summary effect of situations, interventions, and treatments in medicine. Unfortunately, meta-analysis involves excellent human exertion, rendering a process that is extremely inefficient and vulnerable to human bias. To overcome these issues, many researchers are studying and proposing architectures, methods, and tools to automate meta-analysis [5]. The researchers’ main goal is to provide a system for automating the meta-analysis process as much as possible to reduce the time taken in conducting a meta-analysis [62].

Moreover, the development and application of ML in the meta-analysis of clinical trials is a promising approach to implement more effective daily clinical practices.

However, extensive future studies are needed to validate the performance of ML tools in their application domain.