The Impact of Undergraduate Research Journals on the Scholarly World: Present but Small

Background: Undergraduate research journals are a popular mechanism for inducting students into research, communication, and publication facets of academia. A thematic review of 17 review papers found little evidence for journal impact. Methods: A scoping review identified 91 journals. A systematic search identified the journal website, its International Standard Serial Number (if any), its citation rate on Google Scholar, its start year and end year (if applicable). Results: Seventy-five journals had both a Google Scholar h-index and a discoverable start year. Sixty-eight had been cited one or more times. The median h-index was 2, mode was h = 1, and the average h-index = 4.38. Correlation with start year was not statistically significant, neither was content field of journals. Conclusions: Surprisingly, almost all currently published journals have been cited at least once, showing that undergraduate research journals have some impact on the scholarly world. Further analyses are suggested to examine career impact of publication on students and faculty.


Introduction
Undergraduate research journals (URJs) are formal publications of material generated by undergraduate students. Worldwide, the volume and popularity of URJs appears to be growing at an impressive rate. As evidence for this, the Council on Undergraduate Research (CUR: https://www.cur.org/resources/students/journals/) and the British Conference of Undergraduate Research (BCUR) both have large databases of undergraduate journals in the United States of America and Britain, respectively. In 2008, the CUR website listed approximately 40 undergraduate journals in the United States [1], which increased to 70 journals by 2011 [2] and reached 115 in 2013 [3]. This number has increased substantially since then. This demonstrates the reported "explosion" of interest in USA based URJs in recent years [4]. A similar trend can be seen in the United Kingdom; in 2008, the BCUR listed 10 journals [5] and by 2017, 27 were listed. Undergraduate journals are not limited to the UK or the USA with Macquarie University (https://www.mq.edu.au/lih/altc/ug_research/websites.php# Undergraduatejournals/) (Australia) currently listing five URJs. Obviously, URJs can be published in any language, but the analyses conducted in this paper were restricted to English-language reviews, which led to identification of only two journals from outside the anglophone sphere.
Despite this evident growth in popularity, the academic nature, publication process, and organisational infrastructure of URJs varies has only been intermittently and selectively reviewed.
It is surprising that, to our knowledge, the objective academic impact of URJs has not been formally assessed. A review of this publishing sector in this regard is now long overdue and has now been suggested on several occasions [6,7]. In particular, Stone [5] explicitly stated the need for further research on the academic impact of the articles being published (i.e., citations).
Academic impact is multifactorial. This paper addresses impact by undertaking a conventional raw frequency citation analysis (i.e., not adjusted by number of papers published within a fixed time period) for each identified journal. This approach allows us to determine the scholarly impact of a research journal; if authors cite papers from a journal, then it can be presumed that the journal is having some impact on the scholarly world. This is key to understanding the contribution URJs make and helps to distinguish them from vanity projects and/or simple coursework. Thus, the aim of this paper was to provide a summary of the scope, features, and characteristics of URJs and evaluate their citation rates. To do this, a scoping review and thematic analysis of the previous literature reviews of undergraduate journals was undertaken. While the previous review we use to describe URJs is somewhat dated, its advantage is that the scoping review identified URJs that were sufficiently important to be listed and are sufficiently established so as to have both reputation and potential for citation. This means that a more realistic citation impact is likely compared to evaluating recent journals that may not yet be well-read. This strategy allows for more thorough evaluation of journal impact and longevity.
In Section 2, we describe our scoping review process and present a thematic synthesis of URJs. In Section 3, we report a formal quantified evaluation of the citation rates of the URJs identified in the scoping review. This is where our paper uniquely distinguishes itself from previous articles in this area, as it is the first detailed analysis of citation impact rates for URJs.

Scoping Review
A search for published reviews or commentaries on the field was done in the southern hemisphere summer of 2016-2017. A range of search strings were entered into Google and Google Scholar. These were 'undergraduate journals', 'undergraduate research journal', 'student run journals', 'undergraduate student research journals', 'student research journals articles', and 'student run journals. These search strings produced 17 relevant review articles. Excluded articles were those that did not specifically focus on undergraduate journals and discussed undergraduate education more broadly. A thematic review of these papers was conducted to gain a description of the processes in setting up and maintaining a journal, as well as the perceived value of undergraduate journals.

Scoping Themes
The following summarises the key characteristics and issues in the reviewed articles with regard to four major themes (i.e., 'academic characteristics', 'publication process', 'organisational infrastructure', and 'academic impact').

Academic Characteristics
The aims of URJs generally focus on teaching research practices, showcasing undergraduate work, providing future students with an example of high calibre research, gaining experience of the peer-review process, and establishing career profiles [4][5][6]15]. Scope is either inter-or multi-disciplinary or specialised on a single subject [9,12]. Common subjects include biology or natural sciences, chemistry, physics, psychology, and mathematics [6,9,14]. Humanities subjects are also well represented, including economics, political science, international affairs, philosophy, history, English and language studies [9]. Article types are similar to professional journals with empirical studies, literature reviews, and special features [10]. A few URJs limit contributions to original empirical research [10,11] while others publish a range from any student-authored submission regardless of quality to only quality-assured work [2,6,[13][14][15]].

Publication Processes
Not all URJs are fully student-run, with many having faculty as advisors or editor-in-chief [5,8,9,14]. The challenge in having faculty involvement, understandably, is the relative low priority in career advancement for such service [1]. Consequently, it is important that faculty train students in the arts of reviewing and editing [11]. Because review and edit work is considerable for student volunteers [1,8,11], incentives for involvement (e.g., course credit or embedded as coursework) are offered [4,6,8,11]. Financial rewards to undergraduate student authors have been used in some journals [1,5].
Ensuring longevity is a challenge for URJs because when supportive faculty leave an institution, it is easy for a journal to wither and cease [1,3,5]. Nonetheless, continuation over the long haul is feasible [6] with BIOS-A Quarterly Journal Undergraduate Biology having been published since 1930. Single-subject journals have been suggested as having greater longevity [1], though this has not been tested.
Online formats have now more or less superseded print [6,8] because of the diversity of content possible (i.e., multi-media content), cost efficiency, the capacity to broaden readership [1,2,8,15], and the much reduced costs relative to printing journals [7, 14,17]. However, print versions can be good for marketing and prestige [11]. Frequency of publication ranges from no set schedule [9,12] to annual issues [6,8,9] and multiple issues per year [6,9]. Publication with the assistance of the institutional library has been noted as a positive factor [3,12]. Given that even online journals have technology costs (e.g., web hosting), funding is usually required; sources include the sponsoring institution or student organisations, outside organisations, and even paid advertisements [3,6,8]. Institutional enthusiasm for a journal is considered more essential than financial support as this ensures readership and collaborative faculty and students [11].

Organisational Infrastructure
Multiple methods are used to solicit submissions, including direct email to students, posting notices around campus, posting on social media, sharing of print copies, faculty announcements to class, and word of mouth [1,5,11,14]. This marketing aspect of a journal [5,14] could attract participation from marketing and business students [11], rather than being something only contributors participate in. The decision as to who an author can be varies, with some journals only publishing students from the sponsoring institution, while others allow international contributions [8,9]. However, some journals allow recent graduates to submit their undergraduate work [11] and others, student-faculty co-authorship [5]. Many journals only accept work that had faculty endorsement for quality [5,6]. Peer review is a crucial part of formalising the publication process and adds to the reputation and credentials of undergraduate journals [1,5,11].
Of the 17 review articles we consulted, only 5 offered some form of metric on either journal views or downloads: • These statistics are important because they inform journal staff and publishers about the impact of their content. However, these statistics overlook an important question within academia, that is, whether the content influences readers sufficiently to be cited in their own writing. This question can be partially addressed through analysis of citation rates.

Method
A primary search of citation rates for the 91 unique URJs discovered in our initial scoping review was conducted. The list was reduced to 89 journals because two journals (Origin and Biolog-E) were merged into a third journal called Bioscience Horizons. We restricted the search to these journals on the assumption that they had been in the field long enough to have made some impact. In arts, humanities, and social science professional journals, new papers are often not cited for several years from publication because of the long process of drafting, submitting, reviewing, before publication in those disciplines. We expected that by restricting our search to journals that had already been identified, there would be a higher probability of articles being cited.
These journals were then evaluated for their citation rates using Harzing's Publish or Perish software [18]. This is a computer program that retrieves and analyses academic citations using data taken from Google Scholar computing a number of different impact indices. The Hirsch index (h-index) [19] is derived by ranking articles according to the number of times cited and finding the rank order of citation which has the same number of citations or more. This is a widely used measure of impact in the scholarly world [20]. An h-index of 10 means that the 10 most frequently cited articles each have 10 or more citations, while the 11th most cited article has fewer than 11 citations. The use of this index to evaluate journals has been proposed [21].
Google Scholar was used because it indexes all scholarly publications independent of their status. However, as a database, Google Scholar is prone to some inflation of citations because it includes citations in sources that are not peer reviewed. However, it is arguably a superior method for determining impact compared to those embedded in for-profit systems (e.g., Web of Science, Scopus) that only count articles and citations published in journals included on selective lists [22]. This is especially true for humanities and social sciences academics whose h-indices are 1.5 to 2.7 times higher under Google Scholar than in Web of Science or Scopus [23]. Furthermore, it is highly unlikely many university undergraduate journals would appear in selective database systems.
While not required, obtaining an International Standard Serial Number (ISSN) suggests that the journal intends to take its place in the full panoply of published journals. Where ISSN values were provided by a publisher, these were used to ensure only articles from that journal were located. All journals were searched in the ISSN portal (https://portal.issn.org) to identify if they had an ISSN. Otherwise, the journal title within quotation marks was used. To avoid inaccurate results, each article was inspected to ensure it belonged to the relevant journal. This continued until the h-index = 0.
This analysis is restricted to the journals identified in the 17 reviewed papers. This means that some journals of similar age would not be captured by our search simply because they were not mentioned in the papers we reviewed. For example, GeoView (ISSN: 1448-6482), an online undergraduate review of geography and environmental studies, was not captured and is no longer published. Also, not in the list was Press Start (ISSN: 2055-8198), beginning in 2014, which publishes both undergraduate and postgraduate student content. Hence, we do not claim that this is a comprehensive or encyclopedic review of undergraduate research journal impact. This study is restricted to the journals that were mentioned in the previous scoping papers we reviewed.
Of the original list of 91 journals, three had been merged into one title, giving 89 possible journals. Despite extensive searches, 13 journals could not be found in Google Scholar, giving 76 journals that could be evaluated for citations. The journal start year could not be found for one of these journals, giving a final set of 75 journals (Figure 1). Among these, active journals with ISSN counted 41, another 21 active journals did not have ISSN, and 15 defunct journals had discoverable citation rates and start years. Educ. Sci. 2020, 10, x FOR PEER REVIEW 6 of 17

Results
Of the 75 URJs with both a Google Scholar h-index and a discoverable start year, six had zero citations, meaning 69 journals had been cited at least once. Using all 75 journals, the median h-index was 2, the mode was 1 (seen in 19 journals), and the average h-index = 4.38, with a range of 0 to 25 ( Table 2). All journals, except the Croatian Ekscentar, publish articles in English.

Results
Of the 75 URJs with both a Google Scholar h-index and a discoverable start year, six had zero citations, meaning 69 journals had been cited at least once. Using all 75 journals, the median h-index was 2, the mode was 1 (seen in 19 journals), and the average h-index = 4.38, with a range of 0 to 25 ( Table 2). All journals, except the Croatian Ekscentar, publish articles in English.
The first year of publication was 1930 (BIOS: A Quarterly Journal of Biology) and the most recent starting year was 2015. About two-thirds (n = 45) of the journals had been started since 2000. The Pearson correlation between start year and number of citations was not significant at r = −0.21 (p = 0.07), and with the outlier 1930 journal removed, the association was likewise not statistically significant at r = −0.01 (p = 0.93). Hence, age of journal is not meaningfully associated with how frequently articles are cited.
A total of 13 journals out of 75 (17%) of the journals with Google Scholar and start year values were no longer published. The average length of publication for these URJs was 11 years, with a range from 3 to 32. Among the currently published URJs, the average length of publication was 20 years (Median = 18 years, SD = 12.87 years), with a range from 5 to 90 years.    Among the currently published URJs in our list with a Google Scholar h-index impact factor (n = 62), those in multidisciplinary topics predominated (n = 23, 37%), followed by those in the fields of Science, Technology, Engineering, and Medical Health Sciences (STEM) (n = 20, 32%) and Humanities (i.e., arts, politics, and economics; n = 14, 23%). Psychology had five journals (8%). The Humanities journals had the biggest average h-index (Mean = 4.50, SD = 5.75, Median = 2), followed closely by the STEM journals (Mean = 4.05, SD = 4.62, Median = 2), and the Multidisciplinary journals (Mean = 4.00, SD = 3.54, Median = 3). The Psychology journals had the lowest average (Mean = 2.80, SD = 2.68, Median = 1). Given that the standard deviations in each category were so large (Figure 2), it is not surprising that ANOVA revealed there was no statistically significant difference in impact (F (3,57) = 0.18, p = 0.91).
Pearson correlation between start year and number of citations was not significant at r = −21 (p = 07), and with the outlier 1930 journal removed, the association was likewise not statistically significant at r = −01 (p = 93). Hence, age of journal is not meaningfully associated with how frequently articles are cited.
A total of 13 journals out of 75 (17%) of the journals with Google Scholar and start year values were no longer published. The average length of publication for these URJs was 11 years, with a range from 3 to 32. Among the currently published URJs, the average length of publication was 20 years (Median = 18 years, SD = 12.87 years), with a range from 5 to 90 years.
Among the currently published URJs in our list with a Google Scholar h-index impact factor (n = 62), those in multidisciplinary topics predominated (n = 23, 37%), followed by those in the fields of Science, Technology, Engineering, and Medical Health Sciences (STEM) (n = 20, 32%) and Humanities (i.e., arts, politics, and economics; n = 14, 23%). Psychology had five journals (8%). The Humanities journals had the biggest average h-index (Mean = 4.50, SD = 5.75, Median = 2), followed closely by the STEM journals (Mean = 4.05, SD = 4.62, Median = 2), and the Multidisciplinary journals (Mean = 4.00, SD = 3.54, Median = 3). The Psychology journals had the lowest average (Mean = 2.80, SD = 2.68, Median = 1). Given that the standard deviations in each category were so large (Figure 2), it is not surprising that ANOVA revealed there was no statistically significant difference in impact (F(3,57) = 0.18, p = 91).  Three of the more frequently cited URJs (i.e., Journal of Young Investigators, Undergraduate Economic Review, and The Park Place Economist) are published at Illinois-Wesleyan University (IWU), a private U.S. liberal arts college. The Journal of Young Investigators (JYI) seeks to improve undergraduate science training by providing innovative, high-quality educational experiences in science writing, publication, and the peer-review process. The Undergraduate Economic Review publishes U.S. and international student content, while the Park Place Economist specialises in work from senior students at IWU. Also highly cited are the Columbia University Journal of Politics and Policy, The Plymouth Student Scientist from the University of Plymouth in England, and the no longer published Earth and Environment from University of Leeds. The Plymouth Student Scientist is an electronic journal showcasing undergraduate student research from the University of Plymouth's STEM disciplines and was developed with external funding. It is interesting to consider that, despite the prestige of elite universities sponsoring some of these journals (e.g., Yale, Harvard, or Stanford), such sponsorship does not guarantee impact; the four journals associated with those three institutions had an average h-index = 1.75, maximum = 4.

Discussion
To the best of our knowledge, this study is the first comprehensive analysis of citation rates for URJs. While we recognise that citation rates might not be the best way to assess impact, this review of 75 URJs, 65 of which are currently active, does indicate that almost all of them get cited at least once, meaning that they can claim to contribute to the discipline. Our analysis also shows that neither age nor field of research contribute meaningfully to citation rate. Overall impact is understandably small. Nevertheless, for student authors the impact on careers could be substantial and for sponsoring institutions and organisations the prestige impact could be substantially valuable.
This paper highlights novel information on the citation rates of URJs which were identified in a scoping review of the field conducted some time ago. This allows researchers to see that URJs not only have plausible impact on students, faculty and connected institutions, but also contribute in some degree to the scholarly world. Although URJs have usually a small impact in terms of being cited, this does not mean that they are not valuable and successful in other domains. As Weiner [3] noted, success for an undergraduate research journal does not have to be seen in a citation rate.
An important function of URJs is as a venue for initiating new scholars into the full research cycle, including peer review and publication [17,24,25]. The current study does not allow us to evaluate that impact, a matter for future research. Examining postgraduate enrolment and careers of contributing authors would more robustly identify the impact of contributing to an undergraduate research journal. An examination of the impact of participation as a co-author or advisor by academic staff on their promotion, retention, or recruitment would better evaluate the benefit of such journals.
We had anticipated that URJs would be most prevalent in STEM fields; the current study shows instead that humanities fields and multidisciplinary journals outnumber the STEM area. Of course, a future study could work with the full panoply of contemporary journals and that may well show that journals in STEM topics are most prevalent. Nonetheless, there were no mean citation rate differences among the four identified topic areas, suggesting that the specific field or topic of undergraduate research journals should not concern institutions planning to develop a new journal.
It is worth noting that in some of the U.S.-based journals, honours students and programmes are categorised as undergraduate, meaning that such students can legitimately submit papers. Elsewhere, such students may be considered postgraduate (e.g., New Zealand). If more senior students qualify for an URJ, this may contribute to more frequent citation. Nevertheless, the heterogeneity of how undergraduate status is defined may influence why some journals obtain higher citation rates. In some of these higher-ranked journals, we found that co-publication with faculty was the norm. Hence, the notion of undergraduate research is not universal across these journals. A future study could classify and analyse the impact of pure undergraduate authorship versus blended authorship.
Putting aside the influence of the nation-wide or international Honor Societies, more impact was seen among public institutions, some of which would be considered regional in terms of reputation (e.g., Illinois Wesleyan U. and Penn State Berks-Lehigh Valley College). Hence, it would seem factors other than the status of a sponsoring university contribute to citation.
An issue revealed from our investigation was the lack of longevity among URJs. Sustaining journals is challenging when reliant on the enthusiasm of individual faculty members. There are several long-term successful journal exemplars that showcase the diversity of successful formulas associated with survival. The first and oldest is the journal BIOS published by Beta Beta Beta (TriBeta) an honor society for undergraduates from Oklahoma City University, a private university historically affiliated with the United Methodist Church. The journal is dedicated to improving the understanding and appreciation of biology. As a journal its success is likely attributable to its national TriBETA society association and therefore large catchment and readership. This arrangement means there is a steady pipeline of publication material related to honor society events as well as promoting its key undergraduate source of research papers.
In terms of scholarly impact, it is necessary to note some of the challenges posed by relying on Google Scholar as a data source. Sometimes results would include articles from different journals. This is a known feature of Google Scholar that, in being inclusive, it can capture non-relevant articles [22]. Where possible, entering unique journal ISSN values ensures data accuracy, but not all journals have an ISSN. Thus, our current review is constrained by that factor. A broader search for all undergraduate research journals, albeit time-consuming, would provide a fuller picture.
We did not have time or resources to examine in detail the features of the journals with or without ISSN and with or without impact factors. It would be interesting to identify characteristics of higher impact journals to ascertain their staffing model, scope, sponsor status, funding model, authorship of highly cited papers, and so on. The field needs a deeper understanding of what makes an undergraduate research journal highly cited. Further, this review has focused almost exclusively on URJs in the English-speaking academic world; it is entirely possible that in other languages quite different patterns of citation could be evident.

Conclusions
In conclusion, it appears that citations of URJ articles are not the purview of elite institutions, nor dependent on age of the journal or the field of research. The U.S. honor society mechanism seems specially to lend itself to generating research that is cited, although not all jurisdictions have that framework. Nonetheless, undergraduate research journals clearly do contribute to academic research because they are being cited.