Meta-Research in Biomedical Investigation: Gaps and Opportunities Based on Meta-Research Publications and Global Indicators in Health, Science, and Human Development

Abstract

Meta-research in biomedical science is crucial for ensuring rigour, relevance, and transparency in an era marked by the exponential growth of scientific publications. This study examines global and historical trends in meta-research activities within biomedicine and investigates their relationship with health, science, and human development indicators. A systematic analysis of 9633 publications from Scopus, Web of Science, and PubMed was conducted, focusing on publication volume, citation impact, and geographic distribution. Regression analyses reveal a significant positive association between meta-research activity and the Human Development Index (HDI), suggesting that meta-research contributes to societal advancement by enhancing evidence-based decision-making in health. However, no association was found between meta-research output and research and development (R&D) expenditure, reflecting the minimal resource requirements of secondary data-driven studies compared to primary or experimental research. Meta-research activity correlates positively with clinical trial completion, indicating its role in refining study designs and addressing evidence gaps. These findings highlight the importance of expanding meta-research in underrepresented regions to promote equity in scientific advancement and improve the reliability of biomedical knowledge. This result underscores the need for targeted support for meta-research, particularly in low- and middle-income countries with limited scientific infrastructure and resources for new knowledge generation.

1. Introduction

Despite significant variation across fields, the rate of scientific publications has steadily increased over the past century, making it progressively more challenging to track the evolution of broad areas such as biomedical literature (Larsen & von Ins, 2010; Bornmann & Mutz, 2015). The PubMed database currently houses over 21 million English-language biomedical and life sciences articles and has archived more than 35 million scientific papers from the last 50 years (González-Márquez et al., 2024). This fact positions biomedical research as the most prolific area of research globally. Non-linear dimensionality reduction methods such as t-distributed Stochastic Neighbour Embedding (t-SNE), coupled with transformer-based large language models, have been employed to analyse this vast body of data to gain a broad perspective on the evolution of biomedical literature (González-Márquez et al., 2024). However, while these methods are useful for identifying interesting patterns, they do not comprehensively address aspects related to scientific rigour, methodological quality, relevance, or impact. Concern for the validity of scientific evidence in medical and health sciences has been a topic of discussion for several years, which has led to the proposal of novel but little-explored mixed research methods that evaluate various dimensions and determinants of the scientific method applied to health research (Ioannidis, 2005).

In this context, meta-research has emerged as a promising field to accelerate scientific progress by researching research itself. This discipline focuses on five key areas: how to conduct, communicate, verify, evaluate, and reward scientific research (Ioannidis et al., 2015). Meta-research, alongside methodologies like bibliometrics (a branch of scientometrics) (Hicks et al., 2015) and evidence-based research (EBR), which aims to promote a more systematic, transparent, and efficient way of generating high-quality and relevant results (Robinson et al., 2021), are essential to ensuring scientific rigour in an era of massive, often low-quality, and non-reproducible research.

Despite the importance of understanding this concept, it has been demonstrated that few research groups or individual researchers implement them (Ioannidis, 2005, 2008, 2010, 2014). This oversight represents a significant gap, given the vast array of areas and disciplines that comprise medical and health sciences, as well as the high level of heterogeneity and responsibility required to apply the highest quality evidence for evidence-based decision-making in healthcare. However, although meta-research evaluates research itself, its behaviour in biomedical sciences has been scarcely examined since its inception and evolution. Therefore, this study aims to (1) quantify and characterise the global and historical meta-research activity within biomedical research and (2) identify the impact of meta-research publications on global indicators in health, science and human development.

2. The Role and Importance of Meta-Research in Biomedical Sciences

Science has become a global industry, with over 1.5 million scientific articles published only on biomedicine and life sciences yearly (González-Márquez et al., 2024); however, the certainty and applicability of major discoveries are still a concern (Ioannidis, 2014). It has been said that “for most study designs and settings, it is more likely for a research claim to be false than true”, largely due to factors such as study and effect sizes, the number of hypotheses tested, flexibility in study design and outcome definitions, conflicts of interest, and competition in popular research fields (Ioannidis, 2005). But, even if a research discovery is true, the observed effect size would probably be inflated due to underpowered studies and statistical significance thresholds, flexible analysis and selective reporting, or interpretation bias (Ioannidis, 2008).

Over time, even studies considered to be at the top of the evidence pyramid have demonstrated inconsistencies. A clear example is the use of corticosteroids for acute bacterial meningitis (Ioannidis, 2010). In 1994, it was reported that there was “no question about benefits, but beware of harms” (Ioannidis et al., 1994). By 1997, the recommendation shifted to “definite benefit only for some bacteria, then limit to 2 days to avoid harm” (McIntyre, 1997). Then, in 2003, there was a “definite benefit only for children, no increase in harm”, followed by a correction later that year indicating that “benefit is also seen in adults” (Ioannidis, 2010). In 2007, findings suggested: “benefit in high-income countries, but not in low-income countries” (Beek et al., 2008). By 2009: “clear benefit, give it to all” (Vardakas et al., 2009). However, in 2010: “no benefit at all”. The latest systematic review in the Cochrane Library in 2015 stated: “Corticosteroids significantly reduced hearing loss and neurological sequelae, but did not reduce overall mortality. Data support the use of corticosteroids in patients with bacterial meningitis in high-income countries. We found no beneficial effect in low-income countries” (Brouwer et al., 2015).

In 2013, Fanelli et al. (2017) collected 3042 meta-analyses from the Web of Science database, covering the 22 disciplinary categories used by Thompson Reuters’ Essential Science Indicators database. They assessed various biases and the underlying sociological and psychological factors. Their findings indicate that small-study effects, grey literature bias, and citation bias are the most prevalent sources of bias. Additionally, they observed a negative association between authors’ career length and effect size, with early-career researchers being more likely to overestimate effects (Fanelli et al., 2017). Meta-research has enabled a critical appraisal of scientific findings, shedding light on the true nature of data and fostering more informed, evidence-based decision-making.

Despite its significance, the correlation between global biomedical meta-research activity and global indicators in health and science has not been previously studied. Such an analysis can reveal structural patterns in scientific production, highlighting inequalities in the distribution of knowledge and access to research resources. Consequently, this study provides a conceptual framework to guide the development of more inclusive policies and practices. In doing so, it contributes to bridging the gap in understanding the existing methodological and conceptual limitations related to the study of meta-research in health.

3. Global Trends in Meta-Research-Related Publications in Biomedicine

Using a specified search strategy, we searched the Scopus, Web of Science and PubMed databases for all meta-research-related publications in biomedicine (MP) and its synonyms (Supplementary Material). After a manual review of inclusion and exclusion criteria (specified in the supplementary material), the resulting dataset included 9633 articles with information on the journal, country, publication year, total citations, publishing group, and document type. We then filtered for articles with a reported first author’s country, resulting in 6505 MP spanning from 1946 to 2024 across a total of 88 countries. The total number of MP was plotted on a world map (Figure 1) and as a bar plot stratified by years (Figure 2a). The top three countries with the highest volume of MP—the USA, UK, and Canada—accounted for 56.37% (n = 3667) of the total MP, with the USA alone contributing 37.61% (n = 2447). This geographic distribution highlights the concentration of meta-research efforts primarily in North America and Western Europe.

We next analysed citation counts by country, identifying the 15 most-cited countries (Figure 2b). The USA and UK comprised 54.18% (n = 66,318) of total citations across all years. Notably, while the Netherlands ranked seventh in publication volume (n = 171), it was third in citation count (n = 13,253; 10.83%). China, the fourth-largest MP producer (n = 321), ranked sixth in citation counts (n = 5104; 4.17%). Despite high publication counts, countries like South Korea, India, and Brazil did not appear among the top 15 most-cited countries. Conversely, Denmark, Sweden, and Mexico were among the top 15 by citation count but not by publication volume.

To further explore MP characteristics, we analysed document types across all countries in the dataset, identifying the following categories: Articles (n = 3624), Reviews (n = 1198), Editorials (n = 808), Letters (n = 435), Notes (n = 378), and Short Surveys (n = 62). We examined the distribution of these categories among the top five countries by MP volume (Figure 2c). The USA and UK had the largest proportions across all document types. However, when normalising for the total document types within each country, China exhibited a higher proportion of Articles (68.53%) and Reviews (24.92%) compared to other countries (Articles: 52–57%; Reviews: 15–19%). China also had the lowest proportion of Editorials (1.55%), with other countries ranging between 8–15%.

We further assessed MP trends by year across journals, including entries lacking country data, resulting in 9631 articles across 3119 journals (unfiltered by individual journals). The top five journals, each classified as Q1 by the Scimago Journal and Country Rank in 2023, accounted for 11.51% of MP publications. European Food Safety Authority (EFSA) Journal had the highest MP count (n = 644; 6.68%), with all top five journals based in the UK except for the Journal of Clinical Epidemiology, based in the USA (Figure 2d). Filtering the dataset by publishing group (PG) yielded 5623 articles across 559 PGs. Elsevier accounted for 12.8% (n = 720) of the MP, while the top five PGs collectively published 36.74% (n = 2066) of the MP (Figure 2e). Additional information regarding these descriptions can be found in the supplementary material (Tables S1–S3).

4. Comparison Between Meta-Research Activity and Indicators in Global Health, Science and Human Development

Regression models were conducted in R software (version 4.4.0) using MP and specific global indicators of health, science, human development, and innovation to examine the relationship between them. The R code used for the analyses is available at [https://doi.org/10.5281/zenodo.14592259]. The data required to reproduce the analyses can be provided upon reasonable request. First, a comparison was made between meta-research activity and the Human Development Index (HDI). We analysed data covering the period from 1990 to 2022. HDI data was sourced from the United Nations Development Programme (https://tinyurl.com/ycxh9mnn) (accessed on 9 November 2024), representing a composite measure of longevity, education, and standard of living. Only indicators from 85 of the 88 countries with MP were included in the regression model, using the annual total HDI for each country. The annual total of meta-research publications was calculated based on the sources described previously, and the same time period was used for subsequent analyses.

Given the growth patterns in the dependent variable (HDI) and the independent variable (number of publications), a logarithmic transformation was applied to the article count to reduce skewness and enhance model interpretability. We then fitted a linear regression model with the log-transformed publications count as the predictor and HDI as the outcome variable. This approach was chosen over more complex time-series analyses or first-difference models, as our primary goal was to capture the general association between the variables rather than focusing on annual fluctuations.

The logarithmic regression model revealed a positive and significant coefficient for meta-research publications count (β = 6.5454, p < 0.001), indicating a positive association between increased meta-research activity and HDI (Figure 3a). The model’s adjusted r² was 0.7188, suggesting that approximately 71.88% of the variability in HDI can be explained by meta-research activity. Residual analysis showed that normality assumptions were not fully met (p = 0.0039 in the Shapiro-Wilk test), though no significant heteroscedasticity was detected (p = 0.091 in the Breusch-Pagan test). These results indicate that the model is reasonably robust; however, results should be interpreted cautiously due to the non-normality of the residuals.

A first-difference regression analysis between annual changes in publications count and HDI yielded no statistically significant results (p = 0.216) and thus was not included in the main analysis. This finding suggests that the association observed in the logarithmic model likely reflects an overall relationship across the study period rather than specific annual changes. As shown in

Table 1. Summary of regression analyses examining the relationship between log-transformed meta-research publications count and three outcome indicators: HDI, R&D expenditure (% of GDP), and completed clinical trials.

Analysis Type	Outcome	Variable	Estimate	SE	t-Value	p-Value	R2
Simple Regression	HDI	Intercept	29.8701	3.4650	8.621	<0.001	-
		log(Publications)	6.5454	0.7194	9.098	<0.001	0.8422
Simple Regression	R&D Expenditure	Intercept	3.8426	0.5409	7.104	<0.001	-
		log(Publications)	0.0531	0.1092	0.486	0.631	0.0093
Simple Regression	Clinical Trials	Intercept	0.5378	0.8600	0.625	0.54	-
		log(Publications)	1.6056	0.1586	10.120	<0.001	0.8505
Lagged regression (1 yr)	Clinical Trials	Intercept	1.6878	0.8454	1.997	0.062	-
		Lagged log(Publications)	1.4194	0.1567	9.058	<0.001	0.8183

HDI: Human Development Index; R&D: Research and Development; SE: Standard Error.

, the regression results highlight a strong positive association between the number of meta-research publications and HDI.

To examine the relationship between meta-research activity and R&D expenditure (% of gross domestic product [GDP]) (https://tinyurl.com/ykdeecfy) (accessed on 9 November 2024), we analysed annual data from 84 of the 88 countries reporting MP spanning from 1996 to 2022. We employed correlation, linear regression, cointegration, Granger causality, and lagged regression analyses. Results showed no significant association between the number of meta-research publications and R&D expenditure, with a low Pearson correlation (r = 0.097) and non-significant regression coefficient (β = 0.0531, p = 0.631). Cointegration analysis found no long-term relationship, while the Granger causality test suggested a predictive link (p = 0.024). However, the lagged regression indicated no significant effect of meta-research activity on R&D expenditure in the following year (p = 0.845). In summary, no strong association was found between meta-research activity and R&D expenditure, though there is minor evidence of a predictive relationship. These findings suggest limited interaction between these indicators.

Data on completed clinical trials was obtained from ClinicalTrials.gov, encompassing 280,944 completed trials from multiple countries. Meta-research publications counts were obtained as previously described. This analysis assessed the association between annual meta-research activity and the annual total of completed clinical trials from 2005 to 2024. We first computed the Pearson correlation between the log-transformed counts of meta-research articles and completed clinical trials, followed by a linear regression analysis. Both variables were log-transformed to address skewness. Figure 3b illustrates the positive linear association, with a high correlation and significance (p < 0.001), suggesting that increased meta-research activity is associated with higher clinical trial activity rates.

Residual analysis indicated slight non-normality in the residuals (p < 0.05) but confirmed homoscedasticity (p = 0.376), suggesting that the model assumptions hold reasonably well. A Johansen cointegration test revealed the presence of a long-term equilibrium relationship between meta-research articles and completed trials, indicating that these variables move together over time. We conducted a lagged regression analysis to explore potential delayed effects using the prior year’s meta-research activity as a predictor of the current year’s clinical trials. This model showed a significant association, suggesting a one-year lagged effect of meta-research activity on trial completions (Figure 3c). Table 1 summarises the regression results, including the main coefficients, standard errors, and significance levels for both the linear and lagged models. Together, these analyses suggest a robust relationship between meta-research activity and clinical trial activity over the study period.

These findings suggest that meta-scientific activity conducted over time, aimed at promoting rigour, relevance, and appropriateness in generating new knowledge—particularly in health—shows a positive and significant relationship with human development and the execution of clinical trials. This association may be explained by identifying knowledge gaps in health and the timely proposition of solutions through the scientific method. Notably, there is no association between meta-scientific activity and research and development expenditure. This behaviour is expected, as meta-scientific work typically relies on secondary data, which requires fewer resources than primary or experimental data research, even when conducted on a large scale. These results could serve as evidence supporting the development of research lines and workgroups in low- and middle-income countries with low indicators of human and scientific development.

5. Gaps, Needs and Opportunities in Biomedical Meta-Research

The primary goal of science is to expand our knowledge base, ideally leading to practical applications that underscore its value. What began as the pursuit of a few dedicated dilettantes has evolved into a global profession, engaging scientists from diverse fields and nations. Today, science practice requires continuous optimisation to accelerate progress—an endeavour central to meta-research. One key area of meta-research involves examining how research is rewarded, with a particular focus on resource distribution—a critical yet understudied issue. Ironically, although research is fundamentally evidence-based (Faggion, 2021; Lund et al., 2020a, 2020b), allocating research funding, often involving billions of dollars, usually lacks an evidence-driven framework. This oversight leaves us uncertain about the most effective ways to allocate resources, which is a significant gap for several solutions that have been proposed (Ioannidis, 2011).

Meta-research approaches have not been uniformly implemented across all fields of knowledge, leaving notable gaps in areas such as medical ethics. For instance, Faggion (2021) analysed MP from 16 high-impact medical ethics journals and found fewer than 2% (n = 45) articles addressed meta-research topics, with key meta-research areas like reproducibility and incentives particularly underrepresented. One such MP (Wu et al., 2019) identified critical ethical reporting gaps, showing that only half of the prospective clinical studies reviewed had documented informed consent from participants. In methodological rigour, a study published in Developing World Bioethics (Rocha & Grisolia, 2018) found that 84% of toxicology studies from public laboratories reported positive associations between pesticides and cellular changes, while 79% of studies showing no such associations originated from private laboratories. These findings underscore the need for more extensive meta-research across scientific disciplines to enhance transparency and scientific integrity.

The results of this study highlight a significant correlation between meta-research activity in biomedicine and global indicators such as the HDI and the volume of clinical trials. These findings could underscore the critical role that meta-research plays in strengthening scientific rigour, improving methodological standards, and facilitating evidence-based decision-making in health sciences. The positive association with HDI (β = 6.5454, p < 0.001) may reflect how countries with higher meta-research publications have better human development. This outcome could support the hypothesis that nations prioritizing scientific progress have better progress in the areas of education, health, and quality of life.

The strong relationship between meta-research activity and clinical trials (β = 1.6056, p < 0.001) further reinforces the utility of meta-research in promoting clinical research and accelerating the implementation of clinical innovations. The observed one-year lagged effect (β = 1.4194, p < 0.001) suggests that meta-research not only informs but also stimulates subsequent clinical trials, enabling a feedback loop that fosters sustained scientific advancement. This interaction highlights the practical applications of meta-research as a tool to identify knowledge gaps and improve study designs, ultimately driving research efficiency and relevance.

Given these gaps, education on the fundamentals of conducting research on research itself from the early stages of a scientific career is not only an opportunity but a necessity. This foundation is essential to curtail the growing influx of low-quality scientific publications (Fabiano et al., 2024). Aligned with meta-research principles, the appraisal of new research should rely on comprehensive criteria, free from biases associated with traditional indicators like the journal impact factor. Science should be judged by its content, a principle that is especially crucial for shaping the values of future generations of scientists (Hicks et al., 2015; Zhang et al., 2017; Boury et al., 2023).

This study revealed intriguing associations regarding the interest in developing knowledge and hypotheses focused on rigour and relevance in research and its potential impact on scientific, human, and health development indicators. These findings suggest that strengthening standards in research, science, and health innovation could positively influence human and scientific development in the medium- to long-term (Djokoto, 2022; Qin et al., 2023). Therefore, designing new mission-oriented research policies and setting research priorities in health should consider incorporating a section on meta-science applied to biomedical research. This approach aims to improve the quality of future evidence, particularly in contexts with limited resources for research and development funding (Lozada-Martinez et al., 2022, 2023, 2024a, 2024b).

6. Conclusions

This study comprehensively analyses meta-research trends in biomedical science, underscoring the discipline’s role in fostering evidence quality and integrity globally. Key findings demonstrate a robust association between meta-research activity and human development, as evidenced by a significant relationship with the Human Development Index. This result suggests that meta-research enhances scientific rigour and contributes to broader societal outcomes by promoting a culture of evidence-based practice and informed health policies. Interestingly, the lack of association with research and development expenditure highlights a distinctive aspect of meta-research: its reliance on secondary data, which demands fewer resources than primary or experimental studies. This attribute makes meta-research a viable strategy for resource-limited settings, enabling impactful scientific contributions without the prohibitive costs typically associated with high-resource research projects.

The positive correlation between meta-research output and clinical trial completions suggests that meta-research activities may stimulate improved study designs and heightened scrutiny of methodological quality in clinical studies. This relationship is vital in an era where reproducibility and transparency are increasingly scrutinised in scientific practice. The novel insight lies in identifying meta-research as a strategic pathway for low- and middle-income countries to elevate their scientific and human development indicators without extensive financial investment. Encouraging meta-research growth in these regions can bridge scientific disparities and foster a more equitable global research landscape. Consequently, policy interventions to support meta-research infrastructures in underfunded regions could play a transformative role in advancing local and global health research outcomes. Some limitations of the study include that the country of the first author could not be identified in approximately one-third of the originally obtained articles, only a few indicators were included in the regression models, and the possible interactions or influences between them were not assessed. Future research should focus on studying the impact of MP using a wider range of indicators that cover major areas of health research and related fields.