Evaluating Machine Learning Algorithms in COVID-19 Research: A Framework Based on Algorithm Co-Occurrence and Symmetric Network Analysis

Abstract

Machine learning (ML) algorithms are reshaping academic research. However, there is a lack of systematic impact analysis in specific domains. We propose a framework for evaluating the knowledge landscape of domain-specific ML research. It consists of three key components: LDA (Latent Dirichlet Allocation) for topic identification, co-occurrence network construction, and influential algorithm scoring using centrality metrics. In a case study on COVID-19 research, we analyze 30,664 ML-related papers. We identify 13 research topics. We construct a symmetric undirected network to quantify algorithm influence. This analysis employs six centrality metrics: mention frequency, weighted degree, degree centrality, eigenvector centrality, closeness centrality, and betweenness centrality. Results were obtained following linear normalisation. The framework highlights the top ten most influential algorithms for each topic. It reveals the evolving impact of algorithms in COVID-19 research. The methodology is adaptable to other domains. It provides a systematic approach to understanding ML domain-specific impact.

1. Introduction

In contemporary, data-driven scientific domains, the ever-expanding volume of data underscores the imperative of employing diverse methodologies, with a particular emphasis on the pivotal role of ML algorithms in advancing scientific discovery [1]. Due to their clearly defined computational frameworks and advanced learning capabilities, ML algorithms present novel methodologies for the analysis of extensive datasets across diverse research domains [2,3], thereby reshaping the development trajectory of multiple academic disciplines [4]. As distinct knowledge entities in academic literature, ML algorithms also play a vital role in extracting valuable insights or knowledge from vast volumes of raw data [5], often necessitating the integration and comparison of various algorithms. Indeed, numerous research domains, such as biomedical research, employ various algorithms [6]. Consequently, the appropriate selection and rational application of ML algorithms serve as pivotal catalysts for the success of data-driven research [7].

However, domain experts frequently lack specialized algorithmic expertise, complicating the selection of appropriate ML algorithms. This challenge is exacerbated by the rapidly expanding corpus of ML-related literature, which obscures domain-specific trends. Researchers urgently require tools to answer fundamental questions: What problems or topics are ML algorithms applied to in a particular research domain? What are the most influential algorithms within a domain? A comprehensive understanding of these issues necessitates a comprehensive evaluation of domain-specific ML algorithms research. However, the exponential growth of academic papers related to ML algorithms presents obstacles for researchers to obtain an overview of a research domain [8]. For instance, ML algorithms have found widespread application in the research domain of healthcare. As of 1 January 2024, the PubMed Central (PMC) database hosts more than 70,000 articles on ML and healthcare. Extracting a comprehensive overview of research on relevant ML algorithms from such an extensive body of academic papers in this field presents a significant challenge.

ML algorithms are method entities, as well as typical knowledge entities. There are two common approaches for evaluating knowledge entities. The first approach involves conducting a comprehensive survey manually by researchers. However, navigating vast literature collections to perform a comprehensive evaluation is both time-consuming and resource-intensive. Another approach involves utilizing bibliometric indicators derived from bibliometric methods. This approach utilizes various frequency-based indicators to evaluate the influence of knowledge entities [9]. In terms of the bibliometric indicators, quantitative characteristics are traditional indicators for evaluating the influence of algorithm entities [10]. Frequency of mention is the most widely used indicator, reflecting the level of attention received by researchers [11]. In contrast to frequency indicators, researchers have proposed network-related indicators by network centrality to measure large-scale algorithm networks [12]. The ML algorithm network reflects how important algorithm entities are to the overall algorithm community in a research domain. However, the construction of a large co-occurrence network ignores the semantic characteristics of the entities. Some studies have demonstrated that influential algorithms vary across different research topics within a domain [13]. Indeed, there are always several research topics in a research domain. Research topics within a research domain may utilize and prioritize distinct algorithms tailored to their unique objectives and challenges. To this end, text mining offers a viable approach for identifying research topics [14]. Text mining leverages automated processes to extract valuable insights from vast collections of unstructured data [15], offering distinct advantages in identifying complex, undetermined topics [16].

In summary, it is essential to evaluate ML algorithms that are employed within a specific research domain. However, traditional evaluation methods encounter significant challenges due to the vast number of academic papers related to ML algorithm applications. Consequently, this paper proposes a methodological framework for the evaluation of ML algorithm entities, which offers a comprehensive overview of domain-specific ML algorithm research. In detail, we employ text mining, particularly, the LDA, to identify the research topic of academic publications for a domain-specific ML algorithm research, then we construct the ML algorithm network by the co-occurrence relationship among algorithms within in a research topic, and explore the dynamic of ML algorithm network, at last we identify the ML algorithm influence in each topic using frequency of mention and several network centrality measures. The proposed methodological framework can help researchers, especially novices, in gaining a comprehensive understanding of the landscape of ML algorithms within a specific research domain.

To validate the efficacy of the evaluation framework, we conducted a case study. The selection of an appropriate case study domain necessitates rigorous consideration of multiple criteria: (1) the domain must possess a corpus of substantial magnitude to effectively demonstrate the framework’s computational efficiency and scalability in processing extensive scholarly literature; (2) the domain should exhibit a diverse taxonomy of research subdomains to verify the framework’s capacity to identify and differentiate algorithmic influence across heterogeneous thematic contexts; (3) the domain must present contemporary relevance to the scientific community, offering substantive insights that address pressing research challenges and contribute to the advancement of knowledge in the field. Considering that ML algorithms have emerged as essential tools for extracting valuable insights during the COVID-19 pandemic [17], we selected ML algorithms from this period for our case study. First, we compiled a comprehensive collection of widely used ML algorithms from Weka and Scikit-Learn, encompassing 95 full names and 95 abbreviations, to form the ML algorithm dictionary, and we searched ML algorithm-related COVID-19 research in the PMC database by combining this dictionary and COVID-19-related terms. Subsequently, we identified 13 key research topics from a dataset of 30,664 relevant papers using the LDA technique. Next, by integrating a rule-based approach with an ML dictionary, we extracted algorithms from the full text of the papers and constructed co-occurrence networks of algorithm entities for each research topic, with network weights determined by the frequency of co-occurrence. To evaluate the impact of the algorithms, we integrated the network centrality indicator with the frequency of algorithmic mentions to identify signature algorithms within each research topic, and incorporated topological features to further evaluate the impact of these algorithmic entities. This approach not only provides a comprehensive overview of algorithms within the field but also uncovers key research topics and identifies influential algorithms within each topic.

The main contributions in this article are as follows:

(1) Our framework contributed to the knowledge entity evaluation study. The evaluation framework integrated text mining techniques, enabling us to investigate research topics related to knowledge entities within a specific research domain. This framework provides an overview of the domain and facilitates the exploration of the application landscape of these entities. This study introduces an innovative approach to assessing algorithmic influence by combining popularity metrics (e.g., mention frequency) with network centrality measures (such as degree centrality and betweenness centrality) at the thematic level. Unlike conventional methods, our approach accounts for both the algorithm’s visibility (mention frequency) and its role within the network in specific research domains. The key innovation lies in integrating these two types of metrics at the thematic level. This integration allows for a more accurate representation of an algorithm’s influence within specific domains. It overcomes the limitations of relying solely on frequency-based statistics or overall network structure. For example, an algorithm may have significant influence within one theme but remain marginal in others—a subtle distinction missed by traditional methods.

(2) We construct algorithm co-occurrence networks and identify influential algorithms within each research topic, thereby presenting a more comprehensive understanding for researchers, particularly novices, within a specific research domain. Despite the symmetric co-occurrence relationship between algorithms, their influence (as measured by the centrality measure) shows an asymmetric distribution over the network. Consequently, our framework broadens the scope of traditional literature review studies.

(3) Our evaluation framework is validated through a case study on ML algorithm-related COVID-19 papers. Although ML algorithms have played a pivotal role in addressing the challenges presented by the COVID-19 pandemic, the expanding body of literature has resulted in information overload. Researchers frequently encounter confusion regarding the selection of the most appropriate ML algorithms for various COVID-19 applications. Our work provides a comprehensive analysis of how ML algorithms can be applied and mapped within the COVID-19 research domain, thereby clarifying the relationship between algorithms and pandemic-related challenges. This paper is arranged as follows: In Section 2, we introduce the literature review. In Section 3, we present the ML algorithm evaluation framework, and the results are analysed in Section 4. Finally, in Section 5 and Section 6, we present the discussion and conclusion, respectively.

2. Literature Review

This section provides a comprehensive review of the knowledge entities evaluation, examines the application of co-occurrence networks in entity evaluation, and analyzes the current status of research on algorithmic evaluation methods for entities.

2.1. Evaluating Knowledge Entities

The extraction and evaluation of knowledge entities can significantly enhance existing knowledge services, thereby facilitating more efficient and precise access to scientific knowledge for researchers [18]. In previous research, the evaluation of the influence of knowledge entities predominantly depends on bibliometric indicators, typically emphasizing the frequency of knowledge entity mentions, citations, and usage within scholarly literature [19]. Among these indicators, citation frequency proves effective in reflecting the extent of peer recognition of an entity [20], particularly within the domain of library and information science (LIS), and serves as a measure of the popularity of software tools or scientific mapping technologies [21]. In contrast, usage frequency offers crucial insights into the practical application of knowledge entities, particularly for methodological entities such as software tools [22]. Mention frequency, as an indicator of an entity’s presence in academic research, can be integrated with time-series analysis to dynamically monitor variations in its influence [23]. Although these quantitative indicators offer valuable insights, they are also constrained by certain limitations. Citation frequency may not yield a comprehensive and direct portrayal of an entity’s impact, whereas mention frequency and usage frequency fail to differentiate between positive and negative reception [24]. More importantly, current methodologies predominantly rely on quantitative indicators, overlooking the contextual characteristics of knowledge entities within academic literature. For instance, aspects such as the relationship between knowledge entities and other entities, or the specific research topics to which they are associated, are often overlooked in current evaluations. Thus, to gain a more comprehensive understanding of the full impact of knowledge entities, it is essential to broaden the evaluation perspective and integrate analysis at the semantic and relational levels, addressing the limitations inherent in these quantitative assessment methods. Following this line, the present study integrates text mining with network analysis: rather than relying solely on single frequency metrics (e.g., citation or mention counts), we combine LDA-based topic modelling with co-occurrence network analysis to extract research themes at the semantic level and construct relational networks of algorithmic entities, thereby identifying key algorithms within different research topics and their relative influence.

2.2. Evaluating Knowledge Entities Based on Co-Occurrence Network

Dynamic algorithm co-occurrence networks are undirected symmetric structures constructed based on algorithm entity co-occurrence relationships in the literature. By analyzing temporal collaborative patterns between algorithms, these networks reflect research topic evolutionary trajectories. The network identifies co-occurrence frequencies and patterns of algorithm entities in publications, forming algorithm relationship topologies. Its dynamic properties capture temporal changes in algorithm usage patterns, providing a visual representation of domain knowledge structure evolution over time.

Within co-occurrence networks, the structural patterns of interconnections among co-occurring nodes serve as indicators of the relative significance of the involved entities. Network analysis is employed to quantitatively assess these interrelationships and to derive insightful and substantive information [25]. Betweenness centrality, closeness centrality, eigenvector centrality and degree centrality are the most popular evaluation indicators in co-occurrence network analysis. Network indicators offer significant support for the evaluation of knowledge entities, particularly within the biomedical domain. For instance, researchers have constructed heterogeneous co-occurrence networks to investigate potential interactions among drugs, genes, diseases, and therapeutic interventions [26]. Furthermore, the method that combined entity evaluation with network analysis has been employed to identify associations between drugs used in autism treatment [27]. Within the domain of COVID-19 research, scholars have employed indicators such as prevalence indices (PI), collaboration indices (CI), and network topology features to identify critical biological entities, including the ACE-2 gene and the C-reactive protein (CRP) gene, through network structural analysis [28]. A key strength of co-occurrence network analysis lies in its ability to capture the relationships among entities. In contrast, traditional frequency-based evaluation models typically consider each entity in isolation. Conversely, co-occurrence networks unveil the relation of entities, which may reflect their roles in the dissemination of knowledge. Leveraging this potential, researchers have constructed co-occurrence networks to investigate the relationships among algorithmic entities. Through the application of centrality indicators derived from large-scale network analyses of algorithms, researchers have assessed the influence of algorithmic entities within the field of natural language processing (NLP) [29]. Although large-scale co-occurrence networks offer a comprehensive overview of the mutual influence between algorithm entities, this method obscures how the method entities are applied in detail. Within a given domain, a knowledge entity, especially the method entities, usually appears in multiple research topics. Thus, it is necessary to further consider the relation between entities and the influence of entities themselves in different research topics.

2.3. Evaluating the Influence of ML Algorithm Entities

Conventional approaches to assessing the impact of algorithmic entities within the field of LIS predominantly rely on survey techniques and bibliometric indicators. Survey techniques play a critical role in certain domains, notably in medical image recognition. For instance, Support Vector Machines (SVMs) and Convolutional Neural Networks (CNNs) are among the most widely used algorithms in the domain of COVID-19 diagnosis [30]. And a research paper has demonstrated a preference for Random Forest (RF) models outperforming deep learning methodologies in early detection and prognostic analyses [31]. Despite the notable accuracy and clarity of survey methods, manual analysis remains labor-intensive and increasingly inadequate in keeping pace with scientific advancements due to the sheer volume of academic papers. Consequently, scholars are investigating alternative automated assessment methodologies. In recent years, methods for the automated evaluation of algorithmic impact have been developed through the integration of bibliometric indicators and text mining techniques. For instance, by analyzing research papers within the field of NLP, scholars have identified which algorithms exert a greater influence in the domain by extracting and evaluating algorithmic entities [32]. In contrast to conventional survey methods, text mining techniques are capable of efficiently extracting influential algorithms from extensive literature and exhibit greater adaptability to the rapid pace of scientific advancements.

Nevertheless, the current algorithm evaluation methodologies still exhibit certain limitations. Conventional bibliometric indicators primarily reflect the popularity of algorithms based on their frequency of occurrence. However, they fail to uncover the micro-level relationships among algorithms. Furthermore, while co-occurrence network analysis offers a broad overview of algorithmic relationships, it remains insufficient for exploring the influence of algorithms within a specific domain [33]. To address these limitations, this paper proposes an automated approach that integrates text mining and network analysis techniques. We employ the LDA model, which is adept at identifying latent topics within the literature through topic modeling [34], followed by the construction of algorithmic co-occurrence networks. This approach enables a more precise identification of the development dynamics of popular algorithms within a specific domain [35], particularly their influence across different research topics. This novel method of algorithmic entity evaluation not only enhances the analytical efficiency but also offers researchers a more detailed understanding of the development of ML algorithms within a specific domain.

3. The ML Algorithm Evaluation Framework

This section introduces an ML algorithm evaluation framework based on text mining and co-occurrence network analysis, designed to assess the impact of algorithmic entities. The COVID-19 research domain is included as a case study. As shown in Figure 1, the detailed work is divided into four main parts: (1) Data collection and processing, (2) Research topic analysis, (3) Algorithm entity co-occurrence network construction (4) Algorithm entity evaluation.

3.1. Data Collection and Processing

We selected PMC as our database [36], due to its status as one of the most prominent open-access repositories for full-text biomedical literature. Our search strategy consists of two parts: ML algorithms and COVID-19 diseases. Building on the previous algorithm classification framework, this study categorizes ML algorithms into ten groups: ensemble, dimensionality reduction (DR), deep learning (DL), artificial neural networks (ANN), association rule (AR), clustering, regression, classification, probability graph model (PGM), and others [37]. Weka and Scikit-Learn are frequently used open-source ML algorithm tools that implement common ML algorithms. According to the algorithm classification framework, we first constructed a collection of ML algorithms containing the full and short names of the ML algorithms from literature acquired using Weka and Scikit-Learn, in which there were 95 full names and 95 abbreviations, respectively. The dictionary of ML algorithms is presented in

Table A1. The dictionary of ML algorithms.

Abbreviation	ML Algorithm	Abbreviation	ML Algorithm
ADB	Adaptive Boosting	LARS	Least Angle Regression
AA	Apriori Algorithm	LDA	Linear Discriminant Analysis
BP	Back Propagation	LRR	Linear Regression
BIRCH	Balanced Iterative Reducing and Clustering using Hierarchies	LR	Logistic Regression
BBN	Bayesian Belief Network	LSTM	Long short term memory
BN	Bayesian network	LOESS	Locally Estimated Scatterplot Smoothing
BR	Bayesian Regression	MDS	Multi-Dimensional Scaling
BA	Bootstrap aggregating	MLP	Multilayer Perceptron
BNB	Bernoulli Naive Bayes	MRF	Markov Random Field
CCA	Canonical Correlation Analysis	MDS	Multidimensional Scaling
CRF	Conditional Random Field	MNB	Multinomial Naïve Bayes
CNN	Convolutional Neural Network	MEN	Multi-task Elastic-Net
CDT	Conditional Decision Trees	MLASSO	Multi-task Lasso
CONB	Complement Naïve Bayes	MARS	Multivariate Adaptive Regression Splines
CART	Classification and Regression Tree	NB	naive Bayes
CANB	Categorical Naïve Bayes	OLS	Ordinary Least Squares
CHAID	Chi-squared Automatic Interaction Detection	OMP	Orthogonal Matching Pursuit
DT	Decision Tree	PLS	Partial least squares
DBN	Deep Belief Network	PA	Passive Aggressive Algorithms
DNN	deep Neural Network	PE	Perceptron
EA	Eclat Algorithm	PNN	Perceptron Neural Network
DBM	Deep Boltzmann Machine	PR	Polynomial regression
DBSCAN	Density-Base Spatial Clustering of Application with Noise	PCR	Principal Component Regression
ENR	Elastic Net Regression	PCA	Principal Component Analysis
EM	Expectation Maximization	QDA	Quadratic Discriminant Analysis
FA	Factor Analysis	RBFN	Radial Basis Function Network
FL	Federated Learning	RF	Random Forest
FC	Fuzzy clustering	RNN	Recurrent Neural Network
GPR	Gaussian Process regression	RBN	Restricted Boltzmann Machine
GBDT	Gradient Boosting Decision Tree	RIR	Ridge regression
GM	Gaussian Mixtures	RR	Robustness Regression
GNB	Gaussian Naïve Bayes	SM	Sammon Mapping
GPC	Gaussian Processes Classification	SC	Spectral Clustering
GLR	Generalized Linear Regression	SOM	Self-Organizing Map
GAN	Generative Adversarial Network	SA	Stacked Auto-encoders
GA	Genetic Algorithm	SG	Stacked Generalization
GBRT	Gradient Boosted Regression Trees	STA	Stacking
GBM	Gradient Boosting Machines	SR	Stepwise Regression
HMM	Hidden Markov Model	SGD	Stochastic Gradient Descent
HC	Hierarchical clustering	SVM	Support Vector Machine
HN	Hopfield Network	SVR	Support Vector Regression
ID3	Iterative Dichotomiser	AC	Agglomerative Clustering
KRR	Kernel ridge regression	AODE	Averaged One-Dependence Estimators
KM	K-Means	TL	Transfer Learning
KME	K-Medians	XB	XGBoost
KNN	K-Nearest neighbor	CB	CatBoost
LVQ	Learning Vector Quantization	LGBM	LightGBM
Lasso	Least Absolute Shrinkage and Selection Operator

of Appendix A. And the terms related to COVID-19 that were identified through a literature survey are listed in

Table A2. Teams in retrieval strategy.

Word	Related Retrieval Keywords
COVID-19	“COVID-19” OR “SARS-CoV-2” OR “2019-nCoV” OR “Novel Coronavirus Pneumonia” OR “Novel Coronavirus Infected Pneumonia” OR “2019 novel coronavirus” OR “coronavirus 2019” OR “coronavirus disease 2019” OR “2019-novel CoV” OR “2019 ncov” OR “covid 2019” OR “corona virus 2019” OR “ncov-2019” OR “ncov2019” OR “nCoV 2019” OR “Severe acute respiratory syndrome coronavirus 2

of Appendix A. The below query term is used:

“(COVID-19 related terms [Title/Abstract]) AND (algorithm-related terms [Title/Abstract])”.

The PMC database was searched with the language restricted to English. Review articles were excluded due to their failure to evaluate the effectiveness of the mentioned ML algorithms. The initial screening retrieved 30,952 accessible full-text records. After removing duplicates and records with empty abstracts, metadata for 30,664 studies were retained. Utilizing the BioC API for PMC [38], a final dataset of 30,664 full-text articles was included in the study. The above screening process is illustrated in Figure 2.

3.2. Research Topic Analysis

The LDA model is widely regarded as a prominent method due to its effective balance between interpretability and model complexity [39]. In this study, we employ the LDA model and the Gensim toolkit to identify influential research topics within the domain of COVID-19 studies. Abstracts were selected as the text corpus due to their concise nature when handling large-scale datasets [40]. To enhance the explainability of the topics, we employed a multi-step interpretation process. Initially, word clouds were generated to visualize the 100 most probable terms associated with each topic. Subsequently, a manual summarization involved randomly selecting 10% of articles for discussion by two graduate students, utilizing the word cloud results, article content, and existing research. Finally, experts reviewed, revised, and improved the topic summaries. Through this process, we successfully identified the main research topics within the field of COVID-19.

3.3. Algorithm Entity Co-Occurrence Network Construction

Entity extraction is a prerequisite for constructing an entity co-occurrence network. In this study, we employed a rule-based method to extract algorithm entities, based on the pre-constructed collection of ML algorithms. Due to variations in the full names of ML algorithms depending on authors’ writing styles, we employed fuzzy matching for extracting these entities. The principle of fuzzy matching is that if multiple words of an ML algorithmic name appear in sequence within the text, it indicates that the algorithm is mentioned in the document, disregarding case sensitivity. We employed the full-text to identify algorithm entities, as it provides more mentions of algorithms that are not cited in academic articles [41]. Additionally, through observation, we excluded the “Introduction” section because it does not analyze algorithms in depth. The METHODS, RESULTS, and DISCUSSION sections, which all mention effective algorithm entities, were retained.

We constructed algorithm entity co-occurrence networks for different research topics. Within the collection of literature for a specific research topic: if the full name or the abbreviation of an ML algorithm appears once or more in the text, its frequency is incremented by one; if two ML algorithms (e.g., A and B) are mentioned in the same article, it indicates that A and B appear together; that is, an edge is formed in the co-occurrence network to connect nodes A and B. For each new co-occurrence of nodes A and B, the weight of the edges in the co-occurrence network is increased by one. In an ML algorithm co-occurring network $\mathrm{G}(\mathrm{N},\mathrm{E})$, N is the set of nodes in the network, where $\mathrm{N}=\{{\mathrm{n}}_1,{\mathrm{n}}_2\dots \dots {\mathrm{n}}_{\mathrm{m}}\}$, and $\mathrm{m}$ is the total number of nodes. E is a set of edges in the network. ${\mathrm{e}}_{\mathrm{i}\mathrm{j}}\left(\mathrm{i}\ne \mathrm{j},0<\mathrm{i}<\mathrm{m},0<\mathrm{j}<\mathrm{m}\right)$ refers to the weight of the edges between node ${\mathrm{n}}_{\mathrm{i}}$ and node ${\mathrm{n}}_{\mathrm{j}}$, and ${\mathsf{\omega }}_{\mathrm{j}}$ denotes the weight of ${\mathrm{n}}_{\mathrm{i}}$. The network is undirected and symmetric: if there is a link from node i to node j, the reverse link from j to i is also present. By construction, this implies a symmetric adjacency matrix, A_ij = A_ji for all i,j. We verified this property by comparing the adjacency matrix A with its transpose A^T and found them to be identical (i.e., ‖A−A^T‖F = 0), confirming that the network is indeed symmetric and undirected.

Entity extraction model validation: To ensure the reliability of our entity extraction approach, we evaluated the model’s performance on the test set. The entity extraction model achieved a precision of 92.3%, a recall of 90.8%, and an F1-score of 91.5%, demonstrating high reliability in identifying algorithm entities within the COVID-19 research domain. This validation confirms the effectiveness of our rule-based extraction method combined with fuzzy matching for capturing ML algorithm mentions from academic literature.

3.4. Algorithm Entity Evaluation

Considering the mutual influence among algorithm entities, we introduced topological features to evaluate the influence of algorithm entities. We referred to relevant studies in social network analysis. If ML algorithm A is closely connected to B, and ML algorithm C is closely connected to D, and if B has greater influence than D, then it can be inferred that the influence of ML algorithm A is greater than that of C. In social network research, the centrality of the nodes represents their influence [42]. Four indicators are commonly used to measure centrality: degree, betweenness, closeness, and eigenvector [43]. A higher degree centrality of an ML algorithm measures greater importance and influence within the network.

Betweenness centrality quantifies the importance of a node by evaluating the number of shortest paths that traverse it. Accordingly, the higher the betweenness centrality of an ML algorithm, the greater its influence.

Closeness centrality evaluates the importance of a node based on its distance to other nodes. The higher the closeness centrality of an ML algorithm, the greater its influence.

Eigenvector centrality measures the importance of a node based on the number and degree of its neighboring nodes. Accordingly, the higher the eigenvector centrality of an ML algorithm, the greater its influence.

Furthermore, the ML algorithm co-occurrence network is a weighted network, with the weighted degree used as the evaluation index [44]. Consequently, six indicators-namely mention frequency, degree centrality, betweenness centrality, closeness centrality, eigenvector centrality, and weighted degree-were selected. The average value of these indicators was then computed after linear normalization, serving as the foundation for evaluating the influence of the ML algorithms. A higher value of the normalized average corresponds to a greater influence.

Table 1. Evaluation index of ML algorithm and calculation method.

Index	Definition	Calculation method
Mention frequency	This index refers to the number of articles mentioning ML algorithms. The higher the mention count, the greater the influence of nodes.	The frequency of a node.
Weighted degree	This index refers to the sum of line weights of nodes in the network. The greater the weighted degree, the greater the influence of nodes.	${\mathrm{w}}_{\mathrm{i}}=\sum _{{\mathrm{n}}_{\mathrm{j}}\in {\mathrm{N}}_{\mathrm{i}}}{\mathrm{e}}_{\mathrm{i}\mathrm{j}}$$,{\mathrm{w}}_{\mathrm{i}}$$\mathrm{is}\mathrm{the}\mathrm{weighted}\mathrm{degree}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$$,{\mathrm{N}}_{\mathrm{i}}$$\mathrm{is}\mathrm{a}\mathrm{set}\mathrm{of}\mathrm{neighbor}\mathrm{nodes}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$.
Degree centrality	This index refers to the degree of nodes divided by the number of nodes in the network. The higher the degree centrality, the more important the node is in the network.	${\mathrm{d}\mathrm{c}\mathrm{e}\mathrm{n}}_{\mathrm{i}}={\displaystyle \frac{{\mathrm{d}}_{\mathrm{i}}}{\mathrm{m}-1}}$$,{\mathrm{d}\mathrm{c}\mathrm{e}\mathrm{n}}_{\mathrm{i}}$$\mathrm{is}\mathrm{degree}\mathrm{centrality}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$$,{\mathrm{d}}_{\mathrm{i}}$$\mathrm{is}\mathrm{degree}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$$,\mathrm{that}\mathrm{is},\mathrm{the}\mathrm{number}\mathrm{of}\mathrm{edges}\mathrm{connected}\mathrm{to}{\mathrm{n}}_{\mathrm{i}}$.
Eigenvector centrality	This index takes into account the interaction between nodes. The greater the influence of a node’s neighbors, the greater the influence of the node.	$\mathrm{The}\mathrm{eigenvalue}\mathsf{\lambda }$$\mathrm{of}\mathrm{A}\mathrm{x}=\mathsf{\lambda }\mathrm{x}$$\mathrm{with}\mathrm{the}\mathrm{maximum}\mathrm{absolute}\mathrm{value}\mathrm{and}\mathrm{its}\mathrm{corresponding}\mathrm{eigenvector}\mathrm{x}={\left[{\mathrm{x}}_1,{\mathrm{x}}_2\dots \dots {\mathrm{x}}_{\mathrm{m}}\right]}^{\mathrm{T}}$$\mathrm{are}\mathrm{calculated}.{\mathrm{d}\mathrm{v}\mathrm{e}\mathrm{c}}_{\mathrm{i}}={\mathrm{x}}_{\mathrm{i}}$$.\mathrm{A}\mathrm{is}\mathrm{the}\mathrm{adjacency}\mathrm{matrix}\mathrm{of}\mathrm{the}\mathrm{co}-\mathrm{occurrence}\mathrm{network},{\mathrm{d}\mathrm{c}\mathrm{e}\mathrm{n}}_{\mathrm{i}}$$\mathrm{is}\mathrm{the}\mathrm{eigenvector}\mathrm{centrality}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$.
Closeness centrality	This index determines whether a node is close to the center of the network. High closeness centrality means that the closer the node is to other nodes, the more important the node is.	${\mathrm{d}\mathrm{c}\mathrm{l}\mathrm{o}\mathrm{s}}_{\mathrm{i}}={\displaystyle \frac{\mathrm{m}-1}{\sum _{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{d}\mathrm{i}\mathrm{s}}_{\mathrm{i}\mathrm{j}}}}$$,{\mathrm{d}\mathrm{c}\mathrm{l}\mathrm{o}\mathrm{s}}_{\mathrm{i}}$$\mathrm{is}\mathrm{the}\mathrm{closeness}\mathrm{centrality}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$${\mathrm{d}\mathrm{i}\mathrm{s}}_{\mathrm{i}\mathrm{j}}$$\mathrm{is}\mathrm{the}\mathrm{shortest}\mathrm{path}\mathrm{length}\mathrm{between}{\mathrm{n}}_{\mathrm{i}}$$\mathrm{and}{\mathrm{n}}_{\mathrm{j}}$, which is the number of edges.
Betweenness centrality	This index is used to determine whether a node occupies an important path in the network from the perspective of network flow. The higher the betweenness centrality is, the shortest paths pass through the node, and the greater the influence of the node.	${\mathrm{d}\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}}_{\mathrm{i}}={\displaystyle \sum _{\mathrm{s}\ne \mathrm{t}\ne \mathrm{i}}}{\displaystyle \frac{{\mathrm{h}}_{\mathrm{s}\mathrm{t}}^{\mathrm{i}}}{{\mathrm{p}}_{\mathrm{s}\mathrm{t}}}},{\mathrm{d}\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}}_{\mathrm{i}}$ $\mathrm{is}\mathrm{the}\mathrm{betweenness}\mathrm{centrality}\mathrm{of}{\mathrm{n}}_{\mathrm{i}}$$,{\mathrm{p}}_{\mathrm{s}\mathrm{t}}$$\mathrm{is}\mathrm{the}\mathrm{number}\mathrm{of}\mathrm{shortest}\mathrm{paths}\mathrm{between}\mathrm{nodes}\mathrm{s}\mathrm{and}\mathrm{t}.{\mathrm{h}}_{\mathrm{s}\mathrm{t}}^{\mathrm{i}}$$\mathrm{is}\mathrm{the}\mathrm{number}\mathrm{of}\mathrm{shortest}\mathrm{paths}\mathrm{between}{\mathrm{n}}_{\mathrm{s}}$$\mathrm{and}\mathrm{t}\mathrm{that}\mathrm{pass}\mathrm{through}{\mathrm{n}}_{\mathrm{i}}$.

lists the description and calculation methods for each evaluation indicator.

4. Result

4.1. Research Topics Analysis

The analysis of research topics aims to describe the core research content within a specific domain, thereby providing researchers with a comprehensive overview of the research landscape. Subsequently, we created a trend graph illustrating the evolution of the number of research topics over time. The results indicate that ML algorithms play a significant role in addressing the social impact of COVID-19, advancing medical technologies, and improving individual health outcomes.

4.1.1. LDA Topic Modeling

Regarding coherence and perplexity, the optimal parameters for LDA modeling were set to topic = 13, alpha = 0.2, and beta = 0.01. As shown in Figure 3. To visualize the content of each topic, we used a word cloud map to show the 100 words with the highest probability of appearing in each topic. Figure 4 presents a word cloud map of the thirteen topics, where the larger the font size of a word, the greater the probability of the word appearing in the topic. As a result of each research topic, we discovered 8 distinct research topics for the application of ML algorithms in COVID-19, which included mortality risk and outcome analysis, test and detection of SARS-CoV-2, social impact, clinical diagnosis and symptoms of COVID-19, mental health, diagnosis of medical images, laboratory research on viruses, and vaccination, as well as 5 other COVID-19 research topics.

Table 2. Research topics in COVID-19.

Research Topic	Topic Number	Count	Overview
Social impact	Topic 1	3289	This topic focuses on analyzing factors to people’s attitudes, the spread of COVID-19, infection, and death. Students and workers participated in surveys on the perceptions and behavioral change regarding COVID-19 and control measures to the highest degree.
Vaccination	Topic 2	1936	This topic regards vaccination, antibody immunization, viral infection, and mutant strain studies. The advantages of ML models in classification and prediction supported vaccine development and vaccination. In addition, studies on vaccination have mainly focused on the public’s trust in the vaccine and factors influencing vaccine hesitancy.
Clinical diagnosis and symptoms of COVID-19	Topic 3	3256	This topic is centered on exploring factors of COVID-19 infection based on case studies and clinical characterization, which involves detecting and diagnosing COVID-19 infection through PCR testing, case analysis, and clinical evaluation of respiratory symptoms.
Diagnosis of medical images	Topic 4	2702	This topic focuses mainly on the features of medical images, the accuracy of diagnostic methods, and diagnosis using the ML network algorithms.
Mental health	Topic 5	3098	This topic focuses on the psychological impacts of COVID-19 and their spread in the general population, including anxiety, depression, and stress. It also focuses on monitoring psychological changes in the public, immediate intervention, and improving the decision-making abilities of all relevant departments.
Laboratory research on viruses	Topic 6	1904	This topic includes research pertaining to infection at the cellular level, drug therapy, immune response, and blood-related research. Predictive studies are carried out based on the genome, transcriptome, and proteome. Predictive markers of disease are identified using ML.
Test and detection of SARS-CoV-2	Topic 8	4019	This topic is about the detection and infection of SARS-CoV-2 antigen, including coverage of antigen detection methods in clinical samples, detection of viral variants, and rapid diagnostic methods, with an emphasis on the level of virology.
Mortality risk and outcome analysis	Topic 9	4400	This topic mainly contains mortality risk prediction, disease severity analysis, and related factor identification. The purpose of this topic is to identify high-risk patients at an early stage and to provide a reference for clinical decision-making and selection of treatment options to enhance treatment outcomes and optimize healthcare resource management.
Other minor COVID-19 research tasks	Topic 7	541	This topic focuses on children’s health issues during COVID-19.
	Topic 10	1524	This topic focuses on women’s health, e.g., studying the association of COVID-19 with pregnancy.
	Topic 11	1352	This topic primarily concerns the clinical management of COVID-19 patients and is particularly oriented toward studies of lung diseases.
	Topic 12	1300	This topic regards the characterization of social life and public behaviors during the COVID-19 pandemic.
	Topic 13	1297	This topic centers on the impact of the COVID-19 pandemic on healthcare issues.

presents details of the topics we discovered, including the research topic, the topic number, the count of included studies, and the overview for each research topic.

4.1.2. The Evolution of COVID-19 Research Topics

To illustrate the evolution of research attention across topics over time, Figure 5a shows the changes in the number of articles for each topic. Figure 5b shows the percentage of articles on each topic over time. From the time dimension, the number of articles on each topic exhibited a surge in early 2020, peaked in 2021, and subsequently plateaued. In the early stages of COVID-19, when the novel coronavirus was yet unknown, there were only three topics on ML algorithms and COVID-19 in January 2020, which were Topic 3 (clinical diagnosis and symptoms of COVID-19), Topic 8 (test and detection of SARS-CoV-2), and Topic 9 (mortality risk and outcome analysis). As COVID-19 continued to spread, ML was increasingly applied to other research topics. In this stage, Topic 4 (diagnosis of medical images) was the fastest-growing topic. During the later stages of COVID-19, researchers directed their attention to the impact of the virus on individuals and society as the pressure of detection eased. The number of studies included in Topic 1 (social impact) and Topic 5 (mental health) was second and third, only to Topic 9 (mortality risk and outcome analysis).

4.2. Landmark ML Algorithms Analysis

4.2.1. Constructing ML Algorithm Co-Occurrence Network

Based on the eight key COVID-19 topics and other minor ones that indicated the main COVID-19 research directions or topics for applying ML algorithms, we then constructed a co-occurrence network of ML algorithms for each topic, as shown in Figure 6. The node size represents the number of articles mentioning ML algorithms, and the line thickness represents the weight of the line. Figure 7 indicates that the network density of Topic 4 (diagnosis of medical images) ranked first, where ML algorithms were widely applied, followed by Topic 6 (laboratory research on viruses) and Topic 9 (mortality risk and outcome analysis). The network density of the remaining topics was low.

Figure 6 visualizes the co-occurrence networks of ML algorithms across the thirteen research topics. Nodes represent algorithms and edges indicate that two algorithms are used together within at least one study; thicker and darker edges correspond to higher co-occurrence frequencies. Dense networks reflect a richer combination of algorithms within a topic, whereas sparse networks indicate that only a few algorithms dominate the methodological landscape.

Several patterns emerge. The networks for Topic 4 (diagnosis of medical images), Topic 6 (laboratory research on viruses) and Topic 9 (mortality risk and outcome analysis) are markedly denser than those for other topics, implying greater algorithmic diversity and more frequent joint use. In Topic 4, a tightly connected cluster centred on CNN, DNN, RF, SVM, LR and DT highlights the widespread integration of deep learning and traditional machine learning in imaging pipelines. In Topic 6, PCR, LR, RF, FA and PCA occupy central positions and are strongly interconnected, underscoring the importance of regression and dimensionality-reduction techniques for high-dimensional laboratory and omics data. In Topic 9, LR, LRR, RF and MEN serve as hubs, consistent with the predominance of regression-based models in risk prediction and prognosis tasks based on tabular clinical data.

By contrast, networks in several other topics (e.g., Topics 7, 8 and 11) are relatively sparse and often exhibit star-like structures centred on a small number of algorithms, suggesting a more concentrated methodological toolkit. Across all topics, regression algorithms (such as LR and LRR) and dimensionality-reduction methods (such as PCR, PCA and FA) repeatedly appear as highly connected nodes, confirming their broad applicability and central role in COVID-19–related ML research.

4.2.2. Evaluating the Influence of ML Algorithms

The linear normalized average of the six indices we proposed was used as the basis for evaluating the influence of ML algorithms in each topic. As shown in Figure 8, Dimensionality reduction (DR) was found to be the most popular type of ML algorithms across all research topics. The algorithm that has received the most attention from researchers in each research topic is shown in Figure 6. Logistic regression (LR) was the landmark algorithm in seven groups. Principal component regression (PCR) was the landmark algorithm in test and detection of SARS-CoV-2, clinical diagnosis and symptoms of COVID-19, laboratory research on viruses, and diagnosis and treatment of lung diseases. CNN was the landmark algorithm in diagnosis of medical images, and so is Linear regression (LRR) in social life and public behavior.

Topic 4 had the highest network density and Topic 9 included the most studies. For brevity, we only explain the results of Topic 4 and Topic 9 and show the top ten ML algorithms for each index and their normalized averages.

The results of the ML algorithms evaluation in Topic 4 (diagnosis of medical images) are listed in

Table 3. Evaluation results of ML algorithms in Topic 4.

No.	Mention Count		Weighted Degree		Degree Centrality		Betweenness Centrality		Closeness Centrality		Eigenvector Centrality		Normalized Average
1	CNN	803	CNN	2084	RF	0.779	RF	0.120	RF	0.819	RF	0.221	RF	0.890
2	DNN	478	SVM	1960	LR	0.714	DT	0.087	LR	0.778	SVM	0.214	CNN	0.888
3	SVM	384	RF	1820	SVM	0.701	CNN	0.077	SVM	0.77	LR	0.213	SVM	0.784
4	RF	375	DNN	1628	CNN	0.688	LR	0.069	CMM	0.762	DT	0.209	DNN	0.733
5	PCR	369	LR	1406	DT	0.688	SVM	0.063	DT	0.755	CNN	0.207	LR	0.723
6	TL	361	DT	1226	DNN	0.636	DNN	0.060	DNN	0.733	DNN	0.198	DT	0.711
7	LR	245	KNN	902	PCR	0.610	LSTM	0.044	PCR	0.72	PCR	0.197	PCR	0.596
8	LSTM	238	LSTM	868	LSTM	0.597	SVR	0.039	LSTM	0.706	LSTM	0.193	LSTM	0.579
9	DT	211	PCR	824	SVR	0.584	PCR	0.032	SVR	0.7	SVR	0.190	SVR	0.518
10	KNN	140	TL	814	LR	0.519	FC	0.027	KNN	0.669	MLP	0.183	TL	0.508

in Topic 4, which shows the top ten ML algorithms ranked by each evaluation index and normalized average. Previous studies evaluated the influence of ML algorithms based on their frequency of use. The top 10 algorithms with the greatest influence according to the method proposed in this study, which is based on the normalized average, were inconsistent with the results of previous studies. Our results showed the ML algorithm with the largest normalized average was random forest (RF), followed closely by CNN. It also showed that RF was more critical in the structure of the ML algorithms’ co-occurrence network despite the high frequency of CNN. RF performed consistently across diverse datasets and topics with a wide range of applicability, and was often adopted by researchers as one of the experimental models. Both RF and CNN played an important role in diagnosis of medical images. In addition, Support Vector Machine (SVM), deep Neural Network (DNN), logistic regression (LR) and Decision Tree (DT) have also received researchers’ attention under this topic, which can contribute to medical image processing.

In diagnosis of medical images, the frequent adoption of Random Forests (RF) and their relative advantages across numerous research scenarios primarily stems from their strong alignment with the practical data and engineering requirements of imaging studies. The workflow from image → radiomics/texture → tabular features typically generates challenges such as high dimensionality, limited sample size, non-linearity, and high-order interactions. RF effectively suppresses variance through bagging and feature subsampling, while inherently accommodating small samples and sparse high-dimensional data. It also demonstrates strong robustness against class imbalance and cross-centre/cross-device heterogeneity, with performance further enhanced through class weighting or threshold adjustments. Moreover, tree-based models provide global feature importance and local explanations via TreeSHAP, while enabling internal validation through out-of-bag (OOB) error estimates. This aligns well with clinical requirements for interpretability, auditability, and regulatory compliance. At the engineering level, RF offers low training and deployment costs, straightforward parameter tuning, minimal GPU dependency, and seamless integration with clinical tabular variables in later stages. In practice, hybrid paradigms such as “deep feature extraction + RF” or “radiomics + RF” preserve the representational power of deep learning while balancing robustness and interpretability, thereby increasing RF’s adoption rate in the literature. However, it should be clarified that deep models such as CNNs and ViTs remain dominant in large-scale, end-to-end, pixel-level tasks with abundant annotations. The “superiority” referenced here primarily reflects RF’s applicability and robustness under constraints of small sample sizes, high dimensionality, and multi-centre heterogeneity, rather than absolute performance dominance across all scenarios.

The ML algorithm with the largest normalized average was logistic regression (LR), which is particularly effective in prediction and classification tasks. In this research area, where the conditions triggered by COVID-19 are complex, LR applied to binomial, polynomial, and ordered classification models. For example, univariate and multivariate ordinal LR models have been successfully employed to identify independent predictors of illness severity [45]. The dominance of LR in this domain can be attributed to its high interpretability, which is critical in clinical decision-making, especially when predicting outcomes like mortality risk. Furthermore, LR’s ability to handle both simple and complex relationships—as seen in its application to evaluate 30-day mortality risk in hemodialysis patients [46]—reinforces its central role in mortality risk analysis, despite the emergence of other algorithms such as XGBoost (XB). In comparison, Principal Component Regression (PCR) and Multi-task Elastic-net (MEN) were also prominent, with PCR excelling in handling high-dimensional data, and MEN being well-suited for multifactorial analyses, such as considering age and comorbidities. Overall, LR and PCR received the most attention from researchers, making them the most influential algorithms in this research topic.

We observe that LR maintains long-term dominance in clinically relevant domains (

Table 4. Evaluation results of ML algorithms in Topic 9.

No.	Mention Count		Weighted Degree		Degree Centrality		Betweenness Centrality		Closeness Centrality		Eigenvector Centrality		Normalized Average
1	LR	2981	LR	1902	LR	0.807	LR	0.384	LR	0.826	LR	0.373	LR	1
2	PCR	1211	PCR	962	PCR	0.632	PCR	0.3	PCR	0.731	PCR	0.309	PCR	0.681
3	MEN	291	MEN	686	MEN	0.474	MEN	0.054	MEN	0.640	MEN	0.296	MEN	0.436
4	LRR	218	LRR	318	LRR	0.386	CART	0.043	LRR	0.6	RF	0.249	LRR	0.335
5	Lasso	96	Lasso	194	RF	0.386	LRR	0.039	RF	0.594	LRR	0.237	RF	0.314
6	RF	80	RF	170	Lasso	0.316	BN	0.037	Lasso	0.57	Lasso	0.228	Lasso	0.277
7	FA	76	SR	164	FA	0.263	BP	0.035	FA	0.564	FA	0.190	FA	0.246
8	SR	73	FA	162	DT	0.263	TL	0.035	CART	0.559	DT	0.183	CART	0.233
9	RR	64	RR	154	CART	0.246	RF	0.031	SVM	0.543	SVM	0.179	DT	0.219
10	PE	43	PE	114	SVM	0.246	KM	0.022	RR	0.538	CART	0.178	SVM	0.216

). This aligns closely with the field’s stringent requirements for interpretability and probability calibration: LR coefficients directly correspond to odds ratios, facilitating risk communication and guideline development; their probability outputs readily undergo temperature/equidistance calibration to meet specific sensitivity/specificity thresholds. Moreover, clinical tabular data frequently exhibits small sample sizes, class imbalance, and missing values, where LR demonstrates practical feasibility in variable selection and robustness. Regulatory processes also favour transparent, auditable models. Conversely, within medical imaging subdomains, deep networks gain an advantage through large-scale pre-training and spatial invariance, reflecting algorithmic preferences driven by differences in data morphology.

5. Discussion

In this study, we propose a methodology to obtain the landscape of ML algorithms within a domain, encompassing the identification of research topics and the exploration of prominent ML algorithms associated with these research topics derived from an extensive corpus of academic literature.

At the research topic analysis level, the abstract-based LDA topic model demonstrated strong performance. Text mining techniques enabled us to identify thirteen research topics for ML applications in COVID-19. First, we confirmed some findings by referring to previous research. Previous surveys [47] indicated that Topic 4 (diagnosis of medical images) consistently garnered the highest number of studies, a topic in which ML algorithms were employed to analyze image data and other features for disease diagnosis and prediction, aligning with our findings. Additionally, research topics concerning policy and societal impacts, clinical trials, mental health, risk diagnosis, treatment, and prognosis, as noted in previous studies, were also represented in the topics we identified. In addition to the consensus established in previous studies, our approach identified previously unfocused research topics, namely the application of ML algorithms in women’s and children’s health studies and the investigation of changes in public behavioral characteristics. These two topics emphasized groups and characteristics that have attracted considerable attention from researchers during the COVID-19 pandemic. Building on this foundation, the evolutionary analysis indicated that ML models of diagnosis, treatment, and prognosis have consistently represented the foremost research direction within the COVID-19 field. In the later stages of COVID-19, ML models based on vast amounts of medical data supported researchers in discovering new findings. The above analysis demonstrates that our approach effectively and comprehensively identifies research topics relevant to ML applications from large academic literature. The identification of research topics contributes to the development of a structured knowledge framework, outlining the specific applications of ML algorithms for researchers.

At the level of ML algorithm influence evaluation, we constructed algorithm co-occurrence networks within key topics and evaluated algorithms using both mention frequency and network centrality indicators. These six indicators were linearly normalized and then equally weighted to obtain a transparent and comparable composite influence score; alternative data-driven weighting schemes (e.g., PCA or entropy) were not adopted because their sample-specific weights are less interpretable. Based on this composite score, our findings reveal that the influence of the same ML algorithm can vary substantially across different research topics. In terms of application scope, regression and dimensionality reduction algorithms were extensively utilized, with logistic regression (LR) emerging as the most influential algorithm across the eight core research topics, followed by principal component regression (PCR). The regression algorithms aimed to achieve highly sensitive prediction of disease or disease effects [48]. Dimensionality reduction algorithms enabled researchers to condense high-dimensional, complex datasets into lower-dimensional spaces, thereby facilitating subsequent analyses such as genomic, transcriptomic, and proteomic data for disease monitoring or non-linear data for medical image analysis. Analyzing the distribution of ML algorithms across different research topics, the density of co-occurrence networks was found to be highest in Topic 4 (diagnosis of medical images), where RF, CNN, SVM, DNN, LR, and DT all exhibited an influence greater than 0.7. Given that deep learning models offer greater utility than traditional statistical methods in addressing complex medical problems characterized by vast amounts of information. In general, the influence evaluation formula we proposed effectively identifies popular algorithms within each research topic. In contrast to methods relying solely on frequency indicators, this approach places greater emphasis on algorithms that assume a central role within the research topics. The construction of co-occurrence networks within each research topic offers researchers a more holistic understanding of the underlying dynamics.

Beyond the technical evaluation of algorithmic influence, the practical and ethical issues of machine learning (ML) applications in COVID-19 research also deserve careful discussion. First, in clinical environments, model interpretability directly affects applicability. Although deep learning models such as CNNs and DNNs have achieved remarkable accuracy in medical image analysis, their “black-box” nature may hinder physicians from understanding the basis of predictions, thereby reducing the reliability of clinical decision-making. In contrast, algorithms such as Logistic Regression (LR) and Decision Trees (DT), though sometimes less accurate, offer greater transparency and interpretability, making them more acceptable in practice.

Second, fairness is another crucial concern. Imbalanced or biased training data may lead to systematic disadvantages for certain populations (e.g., elderly patients, pregnant women, or minority groups), thus exacerbating health inequities. Researchers should therefore not only evaluate algorithms based on performance metrics but also incorporate fairness-oriented indicators into the evaluation process.

Finally, the risks of inappropriate model selection should not be underestimated. During the pandemic, misclassifying high-risk patients as low-risk could result in delayed treatment and severe consequences, while overestimating risks may cause unnecessary strain on medical resources. To address this, evaluation frameworks should be expanded to consider interpretability, fairness, and risk-awareness alongside technical indicators.

6. Conclusions

ML algorithms are widely used in science discovery. As there are vast academic papers, the evaluation of algorithms has become increasingly critical for researchers conducting data-driven investigations. This paper proposes an automated methodology that integrates text mining and co-occurrence network analysis to identify influential ML algorithms. Text mining techniques are utilized to identify key research topics from vast academic literature. The co-occurrence networks for each topic reflect which algorithms are central. Our method’s successful deployment during the case study of COVID-19 demonstrated that this method offers a direct reference for researchers to make decisions regarding choosing ML algorithms.

This study has its limitations. Although the framework proposed herein demonstrates sound applicability in COVID-19 research, several limitations warrant further clarification. Firstly, the issue of algorithmic nomenclature ambiguity cannot be overlooked. For instance, ‘LR’ may denote either Logistic Regression or Linear Regression in different contexts, with such abbreviations potentially introducing bias in algorithmic identification and categories. Secondly, while fuzzy matching methods enhance recall for algorithmic entities, they may also introduce false positives or false negatives, thereby compromising the accuracy of network construction and impact assessment. Thirdly, the present study’s algorithm identification relies primarily on keyword-based extraction strategies, which may overlook contextual semantics. This approach risks obscuring the precise application or research focus of certain algorithms in specific scenarios.

Future research may enhance algorithm identification precision by incorporating more advanced natural language processing (NLP) techniques, such as named entity recognition and context-sensitive language models. Concurrently, integrating manual verification or semi-automated annotation could mitigate risks arising from algorithm name ambiguity. Only by addressing these issues can the proposed framework achieve robust application across larger-scale and more complex cross-domain datasets. In the spirit of open science, all code and data used in this study are available online at https://github.com/beyoungbelong/ML-algorithms-evaluation/tree/main.