1. Introduction
In Australia, like most other countries, the government supports research by providing funds each year to research projects after an extensive competitive grant round and associated peer review process. The Australian Research Council (ARC) supports basic and applied research in numerous fields to enable engagement with national and international challenges. Increasing focus is placed on the impact of funded research, with return on government investment for all funded projects central to the accountability of public funds. While the measurement of impact continues to evolve, scholarly publications play a fundamental role in the grant cycle. The publication history of grant applicants is an important criterion in the assessment process, as well as being a measure of a project’s impact. Thus, the journal publication outputs of research projects create a self-referencing system.
Government-funded research is commonly held in the highest esteem by academic and research communities. Researchers allocate significant time and effort to preparing grant applications (
Harper et al., 2020). In the Australian context, the ARC has fixed total research funding each year, making obtaining ARC research funding a highly competitive process. In 2021, the percentage of successful applicants for the national priority Discover program was 20%, compared to 29.9% (averaged over Rounds 1 and 2) for the industry collaboration Linkage program (
Australian Research Council, 2021c). Although the funding application is a very well-defined process with clear guidelines, there is no simple method to comparably measure the output and impact of funded projects. The impact principles from the ARC provide a “Research Impact Pathway” table that describes the progression from the inputs needed to conduct research, through the activities carried out and outputs generated, to the outcomes and broader societal benefits that result (
Australian Research Council, 2012). Publications, patents, policy, and media briefings have been identified as high-level examples of output that result in outcomes in the form of commercial products, start-ups, policy implementations, and citations to provide socio-economic benefits and risk reductions in decision making. In terms of outcomes, impact considers scholarly publication as one of the primary output factors and citations as one of the main impact measures of funded projects (
Australian Research Council, 2021b).
From this perspective, the research presented in this paper only focuses on investigating how research funding is associated with research impact, as measured through the quantity and impact of scholarly publications, for ARC funded Discovery (national priority) and Linkage (industry collaboration) projects. The ARC Discovery scheme primarily supports investigator-driven fundamental research, whereas the Linkage scheme is designed to encourage collaboration between universities and industry, government, and community organisations. To achieve this, we consider the funding acknowledgement in research papers, with these references showing the association of publications to the ARC project(s) supporting that research. The objective of this study is not to estimate the causal effect of research funding on scholarly performance but rather to provide a descriptive benchmarking of publication productivity and citation impact across ARC-funded projects at different funding levels. We report on the impact and quantity of research outputs for projects in receipt of differing levels of funding. Lastly, return on investment (ROI) in terms of productivity and scholarly impact is calculated for all funded projects to evaluate the relationship between funding amount and ROI. We begin with a review of the literature on work that considers the relationship between research funding and research impact, and we discuss the different metrics used for quantifying research impact. From this review, we draw upon the metrics discussed to undertake a quantitative analysis of research output and the scholarly impact of ARC funding schemes in alignment with the research questions posed. The research methodology adopted is introduced in the following section, providing details of the data sources used and the data pre-processing procedure. The analysis, results, and discussion of results are included in a consolidated
Section 4, with a consolidated conclusion provided in
Section 5.
Despite a growing body of international research examining the relationship between research funding and scholarly output, several important gaps remain. First, existing studies often focus on aggregate publication or citation outcomes without explicitly assessing funding efficiency or return on investment at the project level. Second, relatively little empirical evidence is available for the Australian research funding context, particularly for ARC-funded projects analyzed across both Discovery and Linkage schemes. Third, prior analyses rarely compare research efficiency systematically across disciplinary groupings such as STEM and HASS using normalized impact indicators. As a result, it remains unclear how funding levels translate into scholarly productivity and impact efficiency within Australia’s primary public research funding system.
2. Review of the Literature
The analysis of the correlation between research funding and subsequent scholarly output has garnered considerable attention, forming the foundation for policy-making concerning the allocation of research grants. Prior studies, with the background details shown in
Table 1, have incorporated both qualitative and quantitative methodologies to probe the influence of various factors on this relationship.
However, the past decade of research has yielded inconclusive results due to a huge variety in scope and methods of studies, as shown in
Table 2. Hence, complicating the task of articulating a coherent rationale for research fund distribution. In the following section, we review recent investigations in this domain and contrast the factors considered to represent research output.
The success of funded research projects has been considered along multiple dimensions of impact including the degree to which they achieve their stated aims or outcomes and the quantity and impact of research outputs produced. The goal achievement of funded projects has been studied for several large-scale projects (
Spanos & Vonortas, 2012), with the goals identified as scientific, technical, and commercial objectives. The results of this research showed that the scale of projects, in terms of team members, has a positive effect to a certain extent on goal achievement, and after that, the positive effect starts diminishing; hence, an inverse U-shaped effect is observed. Similarly, scale in terms of the budget of the project has an inverse U-shaped effect on goal achievement.
The relationship between project size and output was further corroborated by
Bloch et al. (
2016), who analysed the performance of 57 Centres of Excellence (CoE) funded by the Danish National Research Foundation, and the results indicate that the performance of these CoEs increased linearly in terms of number of publications with the duration of the grant, with a peak around grant year 9. The effect of grant size on performance was calculated by dividing the grants into four quartiles based on the average annual grant size. CoEs belonging to different quartiles showed varying performance in terms of Mean Normalized Citation Score (MNCS) throughout the grant period. Overall performance of CoEs was observed to be very high both in terms of MNCS and the share of top 10% most cited publications in the respective fields.
The relationship between citations and the research funding acknowledged in any paper is explored by
Gök et al. (
2016). The purpose of the study was to explore how the acknowledgement of different funding in a paper affects first citation, total citations and also the possibility of the paper becoming highly cited. The authors found that the mentioning of funding for any project in the paper positively relates to the number of total citations for that paper and also to the top percentile citation impact. In a related study,
Jowkar et al. (
2011) analysed the citation impact of funded and unfunded publications by Iranian authors. The results indicated that funded publications have a higher citation impact, implying that funded projects generate higher-impact publications.
The impact of funding size on research outcomes was further studied by the teams from
Danthi et al. (
2015) and
Lauer et al. (
2017). in the context of funding provided by the National Institute of Health, USA.
Danthi et al. (
2015) compared the impact of projects funded through regular grants and those receiving special grants. The regular grant projects resulted in more publications with higher normalised citation impact as compared to the special grant projects. However, there was no difference when the effects based per million dollars spent were considered and the mechanism of funding had no effect on normalised citation impact per million dollars spent. The authors also observed that the factor that contributed most to the prediction of the normalised citation impact per million dollars spent is the total funded amount.
Lauer et al. (
2017) then studied the effect of the Grant Support Index (GSI) on the research output of the National Institute of Health, USA, funded projects. The results showed that research output increased initially with the increase of GSI, but the positive trend started diminishing to the extent that with the increase of GSI, the research output slightly decreased after a certain GSI values.
The potential factors influencing the research output of funded projects were investigated by
Győrffy et al. (
2020). Their analysis indicates that international collaborations and funding amounts have positively impacted the publication output of the funded projects. However, the largest positive influence relates to the scientometric characteristics of the applying principal investigator including H-index, independent citations, and the average number of articles in highly ranked journals.
The relationship between funding amount and research outcomes is made more complex by imperfect measures of impact.
Aagaard et al. (
2020) reviewed existing literature on the relation between funding size and research performance for funded projects. The review showed that there has been research favouring concentration of funding as well as dispersion when it comes to funded research projects, hence, no clear position can be established on the topic of funding dispersion or concentration.
Jung et al. (
2017) also considered these conflicting results when the impact of funding is observed on research output. The authors argued that the results may be due to the way research output is being calculated and mentioned some additional factors like journal impact factor, journal ranking and discipline-specific measures that must be considered while calculating research performance.
One of very few research articles considering the Australian research funding environment, through the ARC research grants schemes, was conducted by
Bromham et al. (
2016). The authors studied the impact of the multidisciplinary research focus of proposed projects on its acceptance for funding. The project proposals submitted to the ARC’s Discovery Program indicated multidisciplinary projects have a much lower probability of receiving funds as compared to projects with a more disciplined focus. This may have a subsequent effect on impact, given the propensity for bibliometric impact measures to favour discipline-focused outputs (
Rafols et al., 2012), introducing a self-referencing effect. Since 2020, no empirical studies have specifically examined the relationship between academic output and Australian research funding.
A requirement of a suitable method and/or tool to assess research impact across different research areas is an open issue, investigated in multiple studies. The two distinct research areas of Humanities, Arts and Social Sciences (HASS) and Science, Technology, Engineering and Mathematics (STEM) require multiple indicators to be considered when it comes to research evaluation in both domains using one bibliometric tool (
Melchiorsen, 2019). The role of different disciplines is studied in multidisciplinary research projects across STEM and non-STEM areas funded by ARC Discovery scheme. Among the STEM disciplines, Engineering, Biological Sciences and Technology appeared most common disciplines. On the other hand, non-STEM interdisciplinary research had Studies in Human Societies, Language, Communication and Culture, and History and Archaeology as the most commonly appearing disciplines. By introducing a new combined discipline of STEMM that contains disciplines of Astrophysics, Computer Science, Engineering, Environmental Studies, Mathematics, Medicine, and Nanotechnology, the authors studied the association of funding amount with the research impact in terms of citations of the publications of those funded projects (
Yan et al., 2018). The number of authors and institutions positively influenced the citation impact in addition to the funding amount.
The body of literature on government-funded research paints a complex picture of the relationship between funding and subsequent academic output. While there is no shortage of effort by researchers to secure these funds, given their prestige and significance in academia, understanding the effects of such funding on research output presents a challenge. In the Australian context, where ARC funding is highly competitive, the difficulty is further compounded. Measuring research impact remains a complex task, with approaches generally highlighting scholarly publication and citation counts as key academic indicators. Taking into account the diverse landscape of research on government-funded projects, our study moves forward to examine the relationship between research funding and the resultant academic output. Specifically, in the competitive setting of ARC funding in Australia, we seek to understand how varying funding levels influence research impact and output.
More recent studies published after 2020 have increasingly emphasised the complexity of evaluating research funding effectiveness, particularly in the context of evolving bibliometric practices and the disruptions associated with the COVID-19 pandemic. This body of work highlights the importance of field-normalised indicators, the limitations of funding acknowledgement data, and the challenges of attributing causality in non-random funding environments.
From this previous work, it is evident that a variety of approaches have been posited for considering relationships between scholarly output and funding amount. The primary purpose of our research is to employ a descriptive benchmarking approach to investigate the relationship between scholarly output and funding amount in the Australian context, using data for ARC-funded projects. By delving specifically into funded projects, we aim to discover nuanced insights into how funding influences scholarly output.
Scholarly output can be measured from two perspectives: the quantity of scholarly output (number of papers produced) and impact of scholarly output (citation counts and journal ranks). To provide focus for our analysis of the data, we seek the answers to the following specific questions:
RQ1: What is the relationship between the quantity of scholarly output and funding amount?
RQ2: What is the relationship between impact of scholarly output and funding amount?
For this study, the impact of scholarly output can be measured via the sum of category-normalized citation impact (CNCI), and the number of highly cited papers generated by the funded project. CNCI is calculated as the number of citations received divided by the expected citations for papers of a similar type, year, and research area. This index is used to eliminate the discipline-specific differences and serves the same purpose as GSI defined by the previous study (
Lauer et al., 2017). Highly cited papers are defined as publications in the top 1% by citations for their research area and publication year. The values of the CNCI, and the highly cited status are provided by InCites. By normalising citations by both publication year and research field, CNCI mitigates temporal bias associated with differential citation windows across publication cohorts. While journal-level indicators (e.g., impact factor quartiles) capture outlet prestige, this study relies on CNCI as an outcome-based measure of research impact, reflecting how publications perform relative to global citation norms within their respective fields and years.
The data sources, data collection and cleansing process, data consolidation, and analysis procedure adopted in this research are explained in the following
Section 3. It is also worth noting that the dataset used in this study predates the COVID-19 pandemic, which significantly disrupted research productivity and publication patterns globally. The global COVID-19 pandemic disrupted academic activity across the world, with noted impacts such as an increase in pandemic-focused publications (
Harper et al., 2020), a decrease in non-medical research output and the impact of those outputs (
Riccaboni & Verginer, 2022), disproportionate impacts on underrepresented minority research (
Carr et al., 2021), and changes in grant applications (
Esquivel et al., 2023). As such, we focus on the period prior to the most acute pandemic-related disruptions (i.e., projects completed before March 2021) to minimise the influence of pandemic-related distortions in research activity and citation dynamics.
4. Results and Discussion
The analysis and results relating to the two research sub-questions presented in
Section 1 are provided and discussed in sequence order in this section. However, prior to the analysis of the research questions, the dataset of 13,880 records has been further investigated for the purpose of better understanding the relationship between funding and scholarly output.
In the ARC dataset, the Program data field identifies the funding program of the project. As previously noted, the ARC has two main funding programs: Discovery and Linkage (
Brooks & Byrne, 2006). The Discovery program focuses on fundamental research to create research areas, opportunities, and impact. These programs support competitive and collaborative research while providing training and career opportunities in research priority areas. The Linkage program focuses on the development of partnerships to transfer knowledge and ideas to industry and businesses for commercial benefit. This program supports industry research collaboration to promote the use of research outcomes. Due to the differences in the focus of these programs, the research output of the projects may be influenced. The projects funded by the Discovery programs aim to publish more high-impact articles on fundamental research. For example, it is assumed that projects supported by the Linkage program focus on applications of the research for industry partners and on generating patents. We therefore expect to see some differences in the publication patterns across these two schemes. Specifically, basic research has been associated with lower levels of funding (
Bentley et al., 2015), which has not been considered through the generated scholarly productivity of these two schemes.
The important aspects of comparison between the Discovery and Linkage programs are funding amount; quantity and impact of publication; and subsequently, a calculation of funding investment return based on this data. T-tests have been conducted between these two programs to consider statistical significance using a 95% confidence interval. For the comparisons including quantity of publication and impact of publication, the test results show p < 0.0001, which means significant differences between Discovery and Linkage programs. For the comparisons including funding investment return including the number of publications per dollar and impact per dollar, the t-test results show p < 0.05, which means there are some differences between Discovery and Linkage programs. Based on these findings, it is necessary to separate the analysis by programs.
Several methodological limitations should be acknowledged. Although citation impact is assessed using normalized indicators to improve comparability, variations in citation practices across disciplines and publication years may still influence observed impact patterns. In particular, differences between STEM and HASS fields and the cumulative nature of citations over time cannot be fully eliminated through normalization. In addition, this study relies on publications that explicitly acknowledge ARC funding, which may lead to an underrepresentation of outputs where funding acknowledgements are incomplete.
The final consolidated dataset of 13,880 records included grants with Active and Closed status. To appropriately investigate funding investment return, only Closed projects have been included for this analysis, assuming grants with an active status may have unrealised outputs. For these projects, the number of all projects (with or without publications) and the number of projects with identified publications from 2002 to 2020 are plotted on a bar chart (
Figure 2). In
Figure 2, the numbers above the pink bars represent the total number of projects, while those above the green bars show the number of projects with publications. The percentage of projects with publications versus the number of all projects dramatically increased from 2002 to 2014. This indicates that an increasing number of projects submitted funding sponsorship details to the publisher over time as they publish. The plot also shows that the number of all projects decreased from 2013 to 2019, which matches the time frame of the ARC projects defined in this dataset from 0 to 8 years for the project duration, aligning with a previous study noting the 9-year peak (
Bloch et al., 2016). From
Figure 2, more than 60% projects between 2009 to 2016 are associated with published articles, while less than 50% of projects from 2002 to 2008 and from 2018 to 2019 have successfully published scholarly articles, with the latter likely attributed to publishing lag.
For meaningful analysis, grant years with more than 60% funded projects having publications were selected for this study to counter issues with publication lag, resulting in the time frame being restricted from 2009 to 2016 (an 8-year period).
Research outputs often take several years to materialise following the receipt of funding. The process typically involves project execution, manuscript preparation, peer review, and eventual publication, followed by the accumulation of citations. Previous bibliometric studies therefore commonly apply extended observation windows to capture the longer-term academic impact of funded research.
Consequently, using this time restriction, a total of 10,565 ARC projects are included in this study.
These ARC projects are categorized based on the funding amounts into six distinct groups: below 1 million, 1–2 million, 2–3 million, 3–4 million, and over 5 million. Furthermore, these projects are further segmented into subcategories within the ARC funding program, as well as research areas defined by Primary FoR4D types. The count of projects within each of these breakdown categories is detailed in
Table 3.
Due to the considerable quantity of projects funded below the 1 million mark, these projects are further subdivided into funding amounts specifically at the 100K level (
Table 4).
To understand the insights of the ROI pertaining to the funding, the analysis in this study is conducted on the basis of 100K (0.1 million).
Prior to answering the research questions, the associations among all the data fields including the funding amount, the quantity of scholarly output, and the impact of scholarly output are explored. A correlation heatmap is employed to explain the complex relationships between funding amount and all other attributes, presented in
Figure 3.
Funding Amount: There is a moderate positive correlation (0.327) with the Number of Publications and a weak positive correlation (0.197) with CNCI Score. Weak positive correlations are also observed with Collaborating Universities (0.254) and Collaborating with Top 10 (0.152).
Primary FoR4D: There are weak negative correlations with most variables, indicating very little linear relationships.
Collaborating Universities: Strong positive correlations are seen with Number of Publications (0.667) and CNCI Score (0.567), indicating that as the number of collaborating universities increases, the number of publications and CNCI score also tend to increase.
Collaborating with Top 10: Similar to Collaborating Universities, there are positive correlations with Number of Publications (0.420) and CNCI Score (0.295), but slightly weaker.
Number of Publications: Strong positive correlations are seen with Collaborating Universities (0.667) and Collaborating with Top 10 (0.420), indicating that more collaborations lead to more publications.
CNCI Score: Strong positive correlations are seen with Collaborating Universities (0.567) and Number of Publications (0.775), suggesting that higher CNCI scores are associated with more collaborations and publications.
Number of Authors and Highly Cited Score: Both have weak correlations with other variables in the table, indicating minimal linear relationships.
From
Figure 3, this analysis suggests that collaborations with universities, especially top-ranked ones, and higher numbers of publications are associated with increased funding amounts and CNCI scores. However, causality cannot be inferred from correlation alone, and further analysis is needed to understand the underlying relationships.
This section presents a descriptive benchmarking analysis of publication productivity and citation impact across ARC-funded projects at different funding levels. To avoid misinterpretation, it should be noted that related metrics reported in this section serve different benchmarking purposes and are not intended as alternative formulations of the same result.
To inspect the big picture, scatter plots are generated to demonstrate the relationship between the quantity of scholarly output and the funding amount. The three scatter plots in
Figure 4 show all funding levels, funding less than and equal to 5 million, and funding less than and equal to 1 million. The scatter plots are generated based on two categories, Discovery and Linkage. Throughout all funding levels, most of the projects received funding less than five millions dollars. More specifically, funding less than or equal to 1 million dollar is the most common funding levels for both Linkage projects and Discovery projects. These projects generated the highest number of publications. However, Discovery projects have a higher number of publications than Linkage projects.
To better understand the relationship between the quantity of scholarly output and funding amount, the analysis was conducted from the following three sections:
- –
Total Number of Publications vs. Funding Amount
- –
Funding Return on Productivity (Publication per Million Dollar) vs. Funding Amount
- –
Funding Return on Productivity (Million Dollars per Publication) vs. Funding Amount
4.1. Total Number of Publications vs. Funding Amount
To measure the quantity of scholarly output, the total number of publications generated by a funded project is used for the analysis. The mathematical equation is presented as follows.
On average, five publications were generated for each closed project, with projects receiving an average of
$437,720 AUD funding from 2009 to 2016. Using
$100,000 increments, the number of publications from closed projects from 2009 to 2016 inclusive are plotted on a bar chart in
Figure 5 to show the distribution of associated publications. From this graph, clear divisions emerge, allowing the funding amount to be grouped into three clusters. The first cluster ranges from 0.0 to 1.9 million dollars, the second cluster from 2.0 to 3.2 million dollars, and the remaining cluster groups funding over 3.2 million dollars. From
Figure 5, it is evident that the second cluster funding levels from 2.0 to 3.2 million dollars generated the majority of publications, while the third cluster funding (levels above 3.2 million dollars) generated the least number of publications. This pattern matches with the inverse U-shaped effect from an earlier study (
Spanos & Vonortas, 2012).
From
Figure 5, there is no strong relationship between the number of publications and the funding amount as the number of publications varies with the growing funding amount. However, among the projects up to 1 million dollars in funding, the higher the funding goes, the more publications the project generates; however, there are exceptions to this rule.
To understand the difference between outputs for funded projects for the Discovery program and Linkage program, the number of publications based on different levels of funding amount is categorized and distributed on bar plots. From
Figure 6, it is evident that funded projects over 3.2 million dollars are associated almost exclusively with Linkage projects, although the higher funded Linkage projects do not produce more publications than lower funded Linkage projects. Both the Discovery and Linkage projects with funding between 1 and 3.2 million dollars have dramatic changes in the number of publications in each level of funding amount, demonstrating that projects with similar funding amounts produce varying numbers of publications. For funded projects with less than 1 million, both Discovery projects and Linkage projects have an increasing number of publications.
From
Figure 6, it is also evident that, for projects funded with less than 1 million dollars, Discovery projects result in more publications than Linkage projects, simply based on the total number of publications. The measure using the number of publications simply shows the total number of publications for different funding levels, but it does not show the productivity of each funding level. To compare the productivity of all fund levels, the funding return on productivity is analysed. The number of publications produced per million dollars is adopted to measure funding return productivity. The number of publications per million dollars is calculated using the number of publications produced per project and dividing this by the funding amount in million dollars for each project. On average, 12 publications were generated by each ARC-funded million dollars from 2009 to 2016.
To understand the disparities between research outputs for funded projects based on the research area by the FoR4D program, the number of publications categorized by either STEM or HASS is presented on bar plots according to different funding levels.
The analysis of
Figure 7 reveals the number of HASS publications and STEM publications. Notably, as funding amounts increase, the difference in publication counts between these two project types tends to widen progressively. However, this trend is not applicable for the rest of the funding levels. A few HASS projects within the funding range of 1.5 to 3.2 million demonstrate visible prolific publication outputs, exceeding the productivity of STEM projects within a similar funding range. This could be caused by the number of funded HASS projects being higher than the number of STEM projects at the same funding level.
4.2. Funding Return on Productivity (Publication per Million Dollar) vs. Funding Amount
To visualize the calculated results across all funding levels, funding return by publication per million dollars based on funding amount per 100K are plotted for the two funding programs (
Figure 8).
The most noticeable return is for funding amounts less than 100K (0.0–0.1 million) from Discovery projects, which is more than 5 fold higher than the other funding levels (within 1 million). As such, the results suggest that Discovery projects with funding less than 100K are the most successful projects in terms of producing outcomes through peer-reviewed journal publications. Additionally, projects with less than 200K funding from both Discovery and Linkage programs generated (on average) more than 25 publications per million dollars, resulting in higher productivity than any other funding level. In general, funding returns for projects of less than 2 million dollars are higher than projects with more than 2 million dollars. It can be concluded that the lower the funding level, the higher the productivity in terms of publications the project generates. This result contradicts the earlier research (
Danthi et al., 2015) on general funding and special funding grants provided by the National Institute of Health in 2009, which may be attributed to discipline-specific issues and differences in selection criteria for the grants.
To visualize the calculated results for STEM/HASS projects across all funding levels, funding return by publication per million dollars based on funding amount per 100K are plotted for the two funding programs (
Figure 9).
As demonstrated in prior research (
Kulczycki et al., 2018), STEM projects generally produce a greater number of publications compared to HASS projects. Our study aligns with this pattern on the whole. However, notable exceptions are observed. Specifically, HASS projects receiving funding under 100K dollars, as well as a few HASS projects with substantial funding, stand out from the trend. Within the subset of projects funded under 100,000 dollars, the return on investment, measured by the number of publications per million dollars, is significantly higher for HASS projects than for STEM projects.
4.3. Funding Return on Productivity (Million Dollars per Publication) vs. Funding Amount
In the previous analysis, the funding return per million dollar is hardly observable for the projects exceeding 5 millions. To overcome this particular challenge, a second funding return algorithm—million dollars required to produce one publication—is used to demonstrate the relationship between funding return and funding amount. Million dollars per publication is calculated based on the funding amount in units of million dollars divided by the number of publications by each project. The mathematical equation is presented as follows.
The results of funding return (million dollars per publication) and funding amount categorized by funding categories are plotted in
Figure 10.
In opposition to the data shown in
Figure 8, projects with high funding and low publication yields become obvious in
Figure 10. The projects with 50 million dollars in funding require the highest funding amount to produce one publication, indicating that those highly funded projects produce the lowest yields. To consider funding return by million dollars required to produce one publication further, only funding amounts less than and equal to 5 million dollars are plotted (
Figure 11).
For funding levels less than 1 million dollars, there is steady growth in funding return, meaning the more funding a project gets, the higher the amount of funding that is required to produce one publication. However, in general, there is no relationship between funding amount and funding return in terms of number of publications. Two types of funding return analysis, including Publication per Million Dollar and Million Dollar per Publication, confirmed these results.
To understand the disparities in funding required per publication by research area, the results of funding return (million dollars per publication) and funding amount categorized by STEM/HASS are plotted in
Figure 12.
Similarly to the analysis in the funding category analysis, in opposition to the data shown in
Figure 9, projects with high funding and low publication yields become obvious in
Figure 12. It can be concluded that highly funded projects produce the lowest yields, especially in a few STEM projects. To consider funding return by million dollars required to produce one publication further, only funding amounts less and equal to 5 million dollars are plotted (
Figure 13).
Figure 13 allows us to visually observe the funding return per publication for the STEM/HASS projects with less than or equal to 5 million funding. For the low funded projects with less than 1 million dollars, there is steady growth in required funding for each publication for both STEM and HASS. Generally speaking, through this visual chart, STEM projects have higher productivity than the HASS projects in terms of required funding to produce one publication. However, there is no strong relationship between funding amount and funding return in terms of number of publications.
To inspect the big picture, scatter plots are generated to demonstrate the relationship between the impact of scholarly output and the funding amount. The three scatter plots in
Figure 14 show all funding levels, funding less than and equal to 5 million, and funding less than and equal to 1 million. The scatter plots are generated based on two categories, Discovery and Linkage. Throughout the entire funding levels, most of the impact are generated from the projects received funding around 3 millions dollars or less. More specifically, most of the impact is generated from the projects less than or equal to 1 million dollar. Although Discovery and Linkage projects within 1 million dollar funding generated high impact, Discovery projects in this funding level generated higher impact than Linkage projects.
To better understand the relationship between the impact of scholarly output and funding amount, the analysis was conducted from the following two sections:
4.4. CNCI vs. Funding Amount
The first measure of impact of scholarly output is CNCI. On average, each project with 437,720 AUD funding generated 88 CNCI from 2009 to 2016.
Based on the different levels of funding (using 100K increments), the CNCI values of publications from closed projects from 2009 to 2016 inclusive are shown in
Figure 15 to demonstrate the distribution of publication impact. Based on the results of the impact on this graph, the funding amount can be grouped into three clusters. The first cluster ranges from 0.0 to 1.9 million dollars, the second cluster from 2.0 to 3.2 million dollars, and anything above 3.2 million dollars is assigned to the third cluster. From
Figure 15, it is evident that the second cluster funding levels from 2.0 to 3.2 million dollars generated the highest impact, and the third cluster funding (levels above 3.2 million dollars) generated the lowest impact. These results align with those for publication productivity (
Figure 5) and also match the inverse U-shaped effect noted in an earlier study (
Spanos & Vonortas, 2012).
From
Figure 15, there is no obvious pattern between CNCI of publications and funding amount. However, among the projects with less than 1 million funding, the higher funding, the higher the impact those publications generated.
To understand the difference between ARC Discovery and Linkage programs, the CNCI values based on different levels of funding amount were categorized by program type (
Figure 16).
Figure 16 demonstrates that funding over 3.2 million dollars from Linkage projects produces substantially less impact than that for projects with lower funding. This result aligns with the quantity of scholarly output in
Figure 5. Both findings can conclude that higher funded projects do not always generate a higher number of publications or higher impact.
Both Discovery and Linkage projects with funding between 1 and 3.2 million dollars have dramatic changes in CNCI in each level of funding amount. This indicates that projects with similar funding amounts produce various impacts, irrespective of research area. For funded projects with less than 1 million, both Discovery and Linkage projects show an increasing number of publications. It can be concluded that there is no strong relationship between CNCI and funding amount. However, for projects with less than 1 million dollars funding, the higher the funding amount, the more CNCI generated.
To understand the difference between STEM and HASS projects, the CNCI values based on different levels of funding amount were categorized by research area (
Figure 17).
Based on the visualized chart (
Figure 17), it can be observed from the big picture that the majority of impacts are oriented from STEM projects. Nevertheless, despite being in the minority, a few HASS projects have the potential to yield significant impacts in the academic realm.
4.5. Highly Cited Status vs. Funding Amount
The second measure of impact of scholarly output is the number of highly cited papers. To observe the distribution of highly cited papers, the highly cited papers from closed projects from 2009 to 2016 inclusive along different levels of funding amount (using 100K increments) are plotted (
Figure 18).
From this graph, it can be concluded that there is no relationship between highly cited status and funding amount, with no trends evident in this data.
The analysis results from two measures using CNCI and Highly Cited Status indicate that there is no relationship between scholarly impact and funding amount. To further analyse the relationship between scholarly impact and funding amount, the funding return on impact is included in this study. The results of CNCI produced per million dollars are adopted to measure funding return on impact. Funding Return on impact per million dollar is calculated using the sum of CNCI of all publications and dividing by the funding amount in million dollars for each project. On average, 202 CNCI scores were generated from each million dollar from 2009 to 2016.
To observe the distribution of highly cited papers for STEM/HASS, the highly cited papers from closed projects from 2009 to 2016 inclusive are plotted (
Figure 19). From this graph, it can be detected that there is no relationship between highly cited status and funding amount for STEM/HASS, with no trends evident in this data. Therefore, it can be concluded that highly cited papers are not correlated with the amount of funding received.
4.6. Funding Return on Impact (CNCI per Million Dollar) vs. Funding Amount
To observe the return on impact across all funding levels, funding return on impact per million dollars based on funding amount by 100K is considered, categorized by funding program, in
Figure 20. The most noticeable return is from funding amount less than 100K (0.0–0.1 million), with more than double that for any other funding levels, and for other funding levels within 1 million, return on productivity is more than 5 fold higher (
Figure 20).
Thus, it can be concluded that funding less than 100K is the most successful in terms of producing publications and generating impact than any other funding levels. This result contradicts the research conclusion from
Győrffy et al. (
2020), who indicated that large funding levels leads to higher impact. Despite this, funded projects between 100K and 1 million demonstrated high variability such that it can be concluded that there is no strong relationship between funding returns on impact and funding amount. For funded projects between 1 and 3.2 million, the number of publications produced per million dollars is low compared to funded projects less than 1 million, and for funding levels over 3.2 million, it is hard to visually detect any pattern from
Figure 20 due to the low number of publications.
In summary, the two measures, namely CNCI and High Cited Status, considered in terms of analysis of funding return on impact (CNCI per Million Dollar) show that there is no strong relationship between the scholarly impact and funding amount.
To observe the return on impact categorized by STEM/HASS, funding returns on impact per million dollars categorized by research area are plotted in
Figure 21.
Through this analysis, the most noticeable return in
Figure 21 is also from funding amount less than 100K (0.0–0.1 million), with more than double that of any other funding levels, for both research areas in STEM and HASS. For both STEM and HASS projects, there is a trend where the impact per mission dollar decreases as the funding increases, limited within the range of 1.0 million dollar funding. Furthermore, it is noteworthy that a few highly funded HASS projects have yielded impacts higher than the average.
5. Conclusions
This study provides a comprehensive project-level bibliometric assessment of scholarly output and return on investment (ROI) associated with Australian Research Council (ARC) funding between 2009 and 2016. Across both Discovery and Linkage programs, the results show no strong or systematic relationship between funding amount and either publication volume or citation impact. Instead, research productivity and impact vary substantially among projects with similar funding levels. Notably, projects receiving less than AUD 100,000 consistently demonstrate the highest returns on investment in terms of both publication productivity and normalized citation impact. While STEM projects exhibit higher average productivity and impact overall, a subset of HASS projects achieves exceptionally high efficiency relative to funding received.
This study focuses exclusively on ARC-funded projects that can be linked to journal publications through funding acknowledgements. Projects whose outputs are not captured in bibliographic databases or whose publications do not consistently report funding information are not included in the analysis. This limitation should be considered when generalising the findings beyond journal-based scholarly outputs.
Through our analysis and comparisons, some interesting phenomena are identified that may or may not be limited within ARC-funded projects. Over the time frame from 2009 to 2016, there were on average five publications produced from each closed project within the Discovery and Linkage projects. In most cases, funded Discovery projects resulted in more articles than Linkage projects. Categorizing all projects into three clusters by the funding amount, 0.0–1.9 million, 2.0–3.2 million, and 3.2+ million, it was observed that, in general, projects awarded 2.0–3.2 million dollars in grant funding generated the highest number of publications. An inverse U-shaped pattern in the number of publications was noted, matching results from a previous study (
Spanos & Vonortas, 2012). However, no strong relationship between publication productivity and funding amount for projects with smaller funding amounts awarded was observed.
We identified no evident connection between publication productivity and funding level; however, the analysis presented in this paper does produce some other interesting results. For funding amounts less than 1 million dollars, as project funding increases, the more publications the project produces. Analysing return on investment for the projects at this funding level, there is a consistent and gradually increasing pattern in funding returns. This also implies that as the project receives greater funding, the corresponding amount of funding associated with generating a single publication increases. This trend applies to both Discovery projects and Linkage projects and to both STEM and HASS projects and suggests a negative return on investment mechanism is at play. Across all funding levels considered in this study, the maximum return on investment projects were associated with the Discovery research funding program and grants with less than 100K of awarded funding, which produces the highest yields in terms of publication productivity per million dollars. Specifically, this funding scheme and level produces approximately 150 publications per million dollar investment. STEM projects typically produce a greater number of publications than HASS projects. Our study generally fits this pattern. However, notable exceptions have been observed. Specifically, HASS projects that received funding under 100K and a handful of HASS projects that received substantial funding stood out from this trend. In the subset of projects funded under 100K, HASS projects had significantly higher ROIs (measured as publications per million dollars) than STEM projects. Another interesting result occurs with Linkage projects and project funding exceeding 5 million dollars, which produce substantially fewer publications compared to projects with lower funding. This may be attributed to the industry-linked nature of the funding scheme, and the propensity for funded projects to generate outputs other than scholarly work, such as patents, industry and/or government reports, and applied innovations. This result does however signal a need for more nuanced approaches to measuring research funding return on investment and impact, noting though that to be effective, this also needs to extend to how the productivity of academics and researchers is measured and accounted for from a career enhancement perspective. It is important to interpret the differences between STEM and HASS disciplines with caution. Bibliometric data derived from Web of Science may underrepresent certain forms of scholarly output common in HASS fields, such as monographs and regional journals. Consequently, citation-based metrics may not fully reflect the broader scholarly and societal contributions of HASS research.
From the perspective of measures, in this study, two types of citation metrics, CNCI and highly cited status, are taken into account during the analysis to measure scholarly impact. From our analysis, there is no obvious connection between the funding amount and the scholarly impact across all funding levels. Much of the impact comes from STEM programs. Nonetheless, while a handful of HASS projects have the potential to have a significant impact in the academic space. Upon closer examination of the particulars, an intriguing phenomena emerged within projects with funding totalling less than one million dollars. In the scope of this investment bracket, the higher funding is, the higher impact the publications generated in terms of citation-metrics-based scholarly impact.
While useful insights have emerged from this study, the methodology employed in this work is not without limitations. Firstly, the publications used to measure the scholarly output are extracted from only WoS, potentially missing some outputs. Additionally, while it is expected that publications produced as a result of an ARC-funded project will reference the funding scheme, omissions or errors in recording ARC funding numbers cannot be ruled out. In these cases, publications are identified as having received funding from ARC; however, these publications do not include the corresponding ARC funding reference number and thus could not be included in this study.
This study focuses on ARC-funded projects between 2009 and 2016 in order to mitigate issues associated with publication lag and citation accumulation. A considerable time window is required to ensure that research outputs resulting from funded projects have sufficient opportunity to be published and cited. However, it is important to acknowledge that the Australian research policy landscape has evolved in recent years, particularly with the introduction of the ARC Engagement and Impact Assessment Pilot in 2018. This initiative expanded the evaluation of research performance beyond traditional scholarly metrics to include broader societal, policy, and industry impacts. As a result, contemporary research outputs increasingly include policy reports, industry collaborations, and other non-academic contributions that may not be captured by citation-based metrics. Therefore, while this study provides insights into the scholarly publication outcomes of ARC-funded research, it does not capture the broader spectrum of research impact that has become increasingly important in recent years.
Furthermore, citation metrics are not the only mechanism for measuring scholarly and research impact, and thus, broader measures of impact such as policy contributions or patents are not captured. Despite these limitations, through our literature review, the dataset we utilized for our study comprising 13,880 funded projects represents one of the most extensive collections of research output and research impact studies within a single funding agency available. Moreover, it is important to note that the analysis in this study is exclusively limited to the publications funded by ARC. The publications not funded by ARC were not taken into consideration in this study.
From the results, it can be concluded that the most successful ARC projects, in terms of the quantity and impact of scholarly output, are Discovery projects with less than 1 million dollars in investment. This indicates that comparatively modestly funded projects yield substantial scholarly output and have a noteworthy scholarly impact and thus may be leveraged for optimal research impact of government funding schemes such as the ARC in Australia. While the work presented here provides an extensive analysis of the correlations between scholarly productivity and research funding amounts, there are opportunities for further exploration. These include but are not limited to, attributes such as the correlation between team size, professorial involvement, and institutional collaborations as a function of funding amount. Lastly, the work provided here provides an approach for a more holistic consideration of impact and return on investment for research grant scheme administrators.
This study focuses exclusively on academic outputs arising from ARC-funded projects. Comparative analyses involving projects supported by alternative funding sources remain a promising direction for future research. Moreover, consistent with the descriptive benchmarking orientation of this study, future work could build on these findings by employing multivariate or quasi-experimental approaches to further examine robustness and potential causal mechanisms underlying the observed patterns.