The Impact of Undergraduate Research Journals on the Scholarly World: Present but Small

Abstract

Background: Undergraduate research journals are a popular mechanism for inducting students into research, communication, and publication facets of academia. A thematic review of 17 review papers found little evidence for journal impact. Methods: A scoping review identified 91 journals. A systematic search identified the journal website, its International Standard Serial Number (if any), its citation rate on Google Scholar, its start year and end year (if applicable). Results: Seventy-five journals had both a Google Scholar h-index and a discoverable start year. Sixty-eight had been cited one or more times. The median h-index was 2, mode was h = 1, and the average h-index = 4.38. Correlation with start year was not statistically significant, neither was content field of journals. Conclusions: Surprisingly, almost all currently published journals have been cited at least once, showing that undergraduate research journals have some impact on the scholarly world. Further analyses are suggested to examine career impact of publication on students and faculty.

1. Introduction

Undergraduate research journals (URJs) are formal publications of material generated by undergraduate students. Worldwide, the volume and popularity of URJs appears to be growing at an impressive rate. As evidence for this, the Council on Undergraduate Research (CUR: https://www.cur.org/resources/students/journals/) and the British Conference of Undergraduate Research (BCUR) both have large databases of undergraduate journals in the United States of America and Britain, respectively. In 2008, the CUR website listed approximately 40 undergraduate journals in the United States [1], which increased to 70 journals by 2011 [2] and reached 115 in 2013 [3]. This number has increased substantially since then. This demonstrates the reported “explosion” of interest in USA based URJs in recent years [4]. A similar trend can be seen in the United Kingdom; in 2008, the BCUR listed 10 journals [5] and by 2017, 27 were listed. Undergraduate journals are not limited to the UK or the USA with Macquarie University (https://www.mq.edu.au/lih/altc/ug_research/websites.php#Undergraduatejournals/) (Australia) currently listing five URJs. Obviously, URJs can be published in any language, but the analyses conducted in this paper were restricted to English-language reviews, which led to identification of only two journals from outside the anglophone sphere.

Despite this evident growth in popularity, the academic nature, publication process, and organisational infrastructure of URJs varies has only been intermittently and selectively reviewed. It is surprising that, to our knowledge, the objective academic impact of URJs has not been formally assessed. A review of this publishing sector in this regard is now long overdue and has now been suggested on several occasions [6,7]. In particular, Stone [5] explicitly stated the need for further research on the academic impact of the articles being published (i.e., citations).

Academic impact is multifactorial. This paper addresses impact by undertaking a conventional raw frequency citation analysis (i.e., not adjusted by number of papers published within a fixed time period) for each identified journal. This approach allows us to determine the scholarly impact of a research journal; if authors cite papers from a journal, then it can be presumed that the journal is having some impact on the scholarly world. This is key to understanding the contribution URJs make and helps to distinguish them from vanity projects and/or simple coursework. Thus, the aim of this paper was to provide a summary of the scope, features, and characteristics of URJs and evaluate their citation rates. To do this, a scoping review and thematic analysis of the previous literature reviews of undergraduate journals was undertaken. While the previous review we use to describe URJs is somewhat dated, its advantage is that the scoping review identified URJs that were sufficiently important to be listed and are sufficiently established so as to have both reputation and potential for citation. This means that a more realistic citation impact is likely compared to evaluating recent journals that may not yet be well-read. This strategy allows for more thorough evaluation of journal impact and longevity.

In Section 2, we describe our scoping review process and present a thematic synthesis of URJs. In Section 3, we report a formal quantified evaluation of the citation rates of the URJs identified in the scoping review. This is where our paper uniquely distinguishes itself from previous articles in this area, as it is the first detailed analysis of citation impact rates for URJs.

2. Scoping Review

A search for published reviews or commentaries on the field was done in the southern hemisphere summer of 2016–2017. A range of search strings were entered into Google and Google Scholar. These were ‘undergraduate journals’, ‘undergraduate research journal’, ‘student run journals’, ‘undergraduate student research journals’, ‘student research journals articles’, and ‘student run journals. These search strings produced 17 relevant review articles. Excluded articles were those that did not specifically focus on undergraduate journals and discussed undergraduate education more broadly. A thematic review of these papers was conducted to gain a description of the processes in setting up and maintaining a journal, as well as the perceived value of undergraduate journals.

2.1. Description of Scoping Data

This paper thematically synthesised content from 17 scoping articles (

Table 1. Scoping Review Sources.

Source	# Journals Reviewed	Journals Reviewed	Field
[1]	Multiple	12	Multidisciplinary
[2]	2	2	Geography
[3]	1	Journal of Purdue Undergraduate Research (JPUR)	Library perspective
[4]	1	IMPULSE	Neuroscience
[5]	1	The Journal of Huddersfield Student Research	Multidisciplinary
[6]	Multiple	35	Science
[7]	2	The Park Place Economist; The Undergraduate Economic Review.	Economics
[8]	Multiple	13	Political science
[9]	Multiple	42
[10]	Multiple	8	Psychology
[11]	1	Clocks & Clouds	Political science, international relations, and public policy
[12]	1	Undergraduate Research Journal at UCCS	Library-run undergraduate journal
[13]	1	The Plymouth Student Scientist	Science
[14]	1	PennScience	Science & Engineering
[15]	1	Geoverse	Geography
[16]	1	Motor City Review	Sociology
[17]	1	Un-named	Medical science

) that were published between 1979 and 2016, with all but one published after 2004. In total, these articles reviewed 91 URJs that were either based in Europe or the United States of America. Five of these articles [1,6,8,9,10] are multi-journal literature reviews with each review considering between 8 and 42 URJs. Two of the five had a broad focus [1,9], encapsulating various types of journals in their review. Three literature reviews focused on single field (i.e., political science [8], science [6], and psychology [10]). Eleven articles [2,3,4,5,7,11,12,13,14,15,16] reviewed one or two URJs. The aims of the articles varied. For example, some were instructional papers on how to set up an URJ [11,16], others reviewed the features of existing journals [5,6,8,9,12,14,15], and a few recorded the reported experiences of individuals who had participated in an URJ [2,3,7,13,14].

2.2. Scoping Themes

The following summarises the key characteristics and issues in the reviewed articles with regard to four major themes (i.e., ‘academic characteristics’, ‘publication process’, ‘organisational infrastructure’, and ‘academic impact’).

2.2.1. Academic Characteristics

The aims of URJs generally focus on teaching research practices, showcasing undergraduate work, providing future students with an example of high calibre research, gaining experience of the peer-review process, and establishing career profiles [4,5,6,15]. Scope is either inter- or multi-disciplinary or specialised on a single subject [9,12]. Common subjects include biology or natural sciences, chemistry, physics, psychology, and mathematics [6,9,14]. Humanities subjects are also well represented, including economics, political science, international affairs, philosophy, history, English and language studies [9]. Article types are similar to professional journals with empirical studies, literature reviews, and special features [10]. A few URJs limit contributions to original empirical research [10,11] while others publish a range from any student-authored submission regardless of quality to only quality-assured work [2,6,13,14,15].

2.2.2. Publication Processes

Not all URJs are fully student-run, with many having faculty as advisors or editor-in-chief [5,8,9,14]. The challenge in having faculty involvement, understandably, is the relative low priority in career advancement for such service [1]. Consequently, it is important that faculty train students in the arts of reviewing and editing [11]. Because review and edit work is considerable for student volunteers [1,8,11], incentives for involvement (e.g., course credit or embedded as coursework) are offered [4,6,8,11]. Financial rewards to undergraduate student authors have been used in some journals [1,5].

Ensuring longevity is a challenge for URJs because when supportive faculty leave an institution, it is easy for a journal to wither and cease [1,3,5]. Nonetheless, continuation over the long haul is feasible [6] with BIOS—A Quarterly Journal Undergraduate Biology having been published since 1930. Single-subject journals have been suggested as having greater longevity [1], though this has not been tested.

Online formats have now more or less superseded print [6,8] because of the diversity of content possible (i.e., multi-media content), cost efficiency, the capacity to broaden readership [1,2,8,15], and the much reduced costs relative to printing journals [7,14,17]. However, print versions can be good for marketing and prestige [11]. Frequency of publication ranges from no set schedule [9,12] to annual issues [6,8,9] and multiple issues per year [6,9]. Publication with the assistance of the institutional library has been noted as a positive factor [3,12]. Given that even online journals have technology costs (e.g., web hosting), funding is usually required; sources include the sponsoring institution or student organisations, outside organisations, and even paid advertisements [3,6,8]. Institutional enthusiasm for a journal is considered more essential than financial support as this ensures readership and collaborative faculty and students [11].

2.2.3. Organisational Infrastructure

Multiple methods are used to solicit submissions, including direct email to students, posting notices around campus, posting on social media, sharing of print copies, faculty announcements to class, and word of mouth [1,5,11,14]. This marketing aspect of a journal [5,14] could attract participation from marketing and business students [11], rather than being something only contributors participate in. The decision as to who an author can be varies, with some journals only publishing students from the sponsoring institution, while others allow international contributions [8,9]. However, some journals allow recent graduates to submit their undergraduate work [11] and others, student-faculty co-authorship [5]. Many journals only accept work that had faculty endorsement for quality [5,6]. Peer review is a crucial part of formalising the publication process and adds to the reputation and credentials of undergraduate journals [1,5,11].

2.2.4. Academic Impact of Undergraduate Research Journals

Evidence for the impact of URJs is largely anecdotal. Benefits for undergraduate students who publish their work are claimed to include: curriculum vitae (CV) enhancement [1,2,3,4,5,6,9,10,12,13,14,15,16], greater competitiveness for jobs and graduate school applications [2,3,4,9,10,14], research and publication experience [2,3,4,5,6,8,9,10,11,13,14], gain career-relevant experience [1,2,3,4,6,7,8,14,15], and greater understanding of the publication process [7,13]. The opportunity to publish one’s work also seems to motivate students to up the standard of work in their courses [2,4,5,6,7,8,9,10,11,13,16]. Academic faculty who work on these journals also gain value through working with and mentoring high-achieving students [3,10,13]. These journals allow institutions to showcase the quality of undergraduate work conducted there [5,6,8,10,12,13,16] and to recruit and retain high quality students [2,3,11,14]. Greater cross-campus communication and collaboration is also supported [2,6,12].

Of the 17 review articles we consulted, only 5 offered some form of metric on either journal views or downloads:

• Two issues of Undergraduate Research Journal—University of Colorado at Colorado Springs (URJ-UCCS) were viewed over 350 times [12],
• Journal of Purdue Undergraduate Research (JPUR) had over 24,000 page views in less than a year [3],
• The Plymouth Student Scientist received 2.2 million hits in 12 months [13].
• 500 downloads of The Journal of Huddersfield Student Research in two months [9],
• Over 13,000 downloads at Journal of Purdue Undergraduate Research in less than two years [3], and
• 5363 full text downloads of Undergraduate Economic Review articles in October 2012 [7].

These statistics are important because they inform journal staff and publishers about the impact of their content. However, these statistics overlook an important question within academia, that is, whether the content influences readers sufficiently to be cited in their own writing. This question can be partially addressed through analysis of citation rates.

3. Citation Impact of Undergraduate Research Journals

3.1. Method

A primary search of citation rates for the 91 unique URJs discovered in our initial scoping review was conducted. The list was reduced to 89 journals because two journals (Origin and Biolog-E) were merged into a third journal called Bioscience Horizons. We restricted the search to these journals on the assumption that they had been in the field long enough to have made some impact. In arts, humanities, and social science professional journals, new papers are often not cited for several years from publication because of the long process of drafting, submitting, reviewing, before publication in those disciplines. We expected that by restricting our search to journals that had already been identified, there would be a higher probability of articles being cited.

These journals were then evaluated for their citation rates using Harzing’s Publish or Perish software [18]. This is a computer program that retrieves and analyses academic citations using data taken from Google Scholar computing a number of different impact indices. The Hirsch index (h-index) [19] is derived by ranking articles according to the number of times cited and finding the rank order of citation which has the same number of citations or more. This is a widely used measure of impact in the scholarly world [20]. An h-index of 10 means that the 10 most frequently cited articles each have 10 or more citations, while the 11th most cited article has fewer than 11 citations. The use of this index to evaluate journals has been proposed [21].

Google Scholar was used because it indexes all scholarly publications independent of their status. However, as a database, Google Scholar is prone to some inflation of citations because it includes citations in sources that are not peer reviewed. However, it is arguably a superior method for determining impact compared to those embedded in for-profit systems (e.g., Web of Science, Scopus) that only count articles and citations published in journals included on selective lists [22]. This is especially true for humanities and social sciences academics whose h-indices are 1.5 to 2.7 times higher under Google Scholar than in Web of Science or Scopus [23]. Furthermore, it is highly unlikely many university undergraduate journals would appear in selective database systems.

While not required, obtaining an International Standard Serial Number (ISSN) suggests that the journal intends to take its place in the full panoply of published journals. Where ISSN values were provided by a publisher, these were used to ensure only articles from that journal were located. All journals were searched in the ISSN portal (https://portal.issn.org) to identify if they had an ISSN. Otherwise, the journal title within quotation marks was used. To avoid inaccurate results, each article was inspected to ensure it belonged to the relevant journal. This continued until the h-index = 0.

This analysis is restricted to the journals identified in the 17 reviewed papers. This means that some journals of similar age would not be captured by our search simply because they were not mentioned in the papers we reviewed. For example, GeoView (ISSN: 1448-6482), an online undergraduate review of geography and environmental studies, was not captured and is no longer published. Also, not in the list was Press Start (ISSN: 2055-8198), beginning in 2014, which publishes both undergraduate and postgraduate student content. Hence, we do not claim that this is a comprehensive or encyclopedic review of undergraduate research journal impact. This study is restricted to the journals that were mentioned in the previous scoping papers we reviewed.

Of the original list of 91 journals, three had been merged into one title, giving 89 possible journals. Despite extensive searches, 13 journals could not be found in Google Scholar, giving 76 journals that could be evaluated for citations. The journal start year could not be found for one of these journals, giving a final set of 75 journals (Figure 1). Among these, active journals with ISSN counted 41, another 21 active journals did not have ISSN, and 15 defunct journals had discoverable citation rates and start years.

3.2. Results

Of the 75 URJs with both a Google Scholar h-index and a discoverable start year, six had zero citations, meaning 69 journals had been cited at least once. Using all 75 journals, the median h-index was 2, the mode was 1 (seen in 19 journals), and the average h-index = 4.38, with a range of 0 to 25 (

Table 2. Citation analysis of 89 undergraduate research journals ranked by h-index.

Journal	ISSN	h-Index	Start Year	End Year	Sponsor	Website
1. Bioscience Horizons (incorporating Biolog-E and Origin)	1754–7431	25	2008	2018	Oxford University Press, UK	https://academic.oup.com/biohorizons
2. BIOS: A Quarterly Journal of Biology	0005–3155; 1943-6289	17	1930	NA	TriBeta honor society	http://www.bioone.org/loi/bios
3. Psi Chi Journal of Undergraduate Research	1089–4136	17	1996	NA	International Honor Society in Psychology	https://www.psichi.org/?page=journal_main#.WI_mSIU_XBK
4. Undergraduate Economic Review		17	2005	NA	Illinois Wesleyan University	http://digitalcommons.iwu.edu/uer/
5. Journal of Undergraduate Study and Independent Research		16	nf	nf	Nf	http://www.jusir.org
6. The Park Place Economist		14	1993	NA	Illinois Wesleyan University	https://www.iwu.edu/economics/PPE.html
7. Columbia University Journal of Politics and Policy		13	1989	NA	Columbia University	https://www.helvidius.org/
8. The Plymouth Student Scientist	1754–2383	11	2008	NA	University of Plymouth, UK	https://pearl.plymouth.ac.uk/handle/10026.1/12776
9. Journal of Young Investigators	1539–4026	11	1998	NA	Illinois Wesleyan University	http://www.jyi.org/
10. Undergraduate Research Journal for the Human Sciences		11	2002	2016	Kappa Omicron Nu, National Honor Society for Human Sciences	http://www.kon.org/CFP/cfp_urjhs.html
11. Earth and Environment	1744–2893	10	2005	2013	University of Leeds, UK	http://www.see.leeds.ac.uk/misc/ejournal/
12. Young Scholars in Writing	2152–6516; 2152–6524	8	2003	NA	Penn State Berks-Lehigh Valley College	https://youngscholarsinwriting.org/index.php/ysiw
13. Canadian Undergraduate Journal of Cognitive Science	1499–7487	8	2002	2016	Simon Fraser University, CA	http://www.sfu.ca/cognitive-science-old/journal/
14. Ekscentar	1848–6398; 1331–4939	7	1997	NA	University of Zagreb, Croatia	https://hrcak.srce.hr/ekscentar
15. Lethbridge Undergraduate Research Journal	1718–8482	7	2006	NA	Lethbridge University, CA	https://lurj.org/
16. Journal of Psychological Inquiry	1085–6641	7	1996	NA	Great Plains Behavioral Research Association	https://www.psychinquiry.org/
17. MIT Undergraduate Research Journal		7	1981	NA	MIT	http://murj.mit.edu/
18. University Avenue: Undergraduate Journal of Economics		7	1997	2004	Illinois-Wesleyan University	https://digitalcommons.iwu.edu/uauje/
19. Berkeley Undergraduate Journal	1099–5331	6	1987	NA	University of California, Berkeley	https://buj.berkeley.edu/
20. Berkeley Scientific	1097–0967; 2373–8146	6	1996	NA	University of California, Berkeley	https://bsj.berkeley.edu/
21. UCI Undergraduate Research Journal	0277–4739	6	1998	NA	University of California, Irvine	https://www.urop.uci.edu/journal.html
22. Caltech Undergraduate Research Journal		6	2001	NA	Caltech	https://curj.caltech.edu/
23. Electronic Journal of Undergraduate Mathematics		6	1995	2006	Furman University	http://math.furman.edu/~mwoodard/fuejum/content/toc.html
24. Journal of Purdue Undergraduate Research (JPUR)	2158–4044; 2158–4052	5	2011	NA	Purdue University	http://docs.lib.purdue.edu/jpur/
25. History Matters: An Undergraduate Journal of Historical Research	1934–4651	5	2004	NA	Appalachian State University	https://historymatters.appstate.edu/
26. The Politic: A Yale Undergraduate Journal of Politics	1539–7513	4	2001	NA	Yale University	http://thepolitic.org/past-issues/
27. Undergraduate Research Journal at UCCS	2693–3918	4	2008	NA	University of Colorado, Colorado Springs	https://urj.uccs.edu/index.php/urj
28. Undergraduate Journal of Psychology	2325–0917	4	1987	NA	University of North Carolina Charlotte	https://journals.uncc.edu/ujop/index
29. Fields: Journal of Huddersfield Student Research		4	2015	NA	University of Huddersfield, UK	https://www.fieldsjournal.org.uk/
30. Inquiry Journal		4	2005	NA	University of New Hampshire	https://www.unh.edu/inquiryjournal/
31. Stanford Undergraduate Research Journal		4	2002	NA	Stanford University	http://surj.stanford.edu/
32. Princeton Bioethics Journal		4	1998	2002	Princeton University	https://www.scimagojr.com/journalsearch.php?q=13067&tip=sid&clean=0
33. The Oswald Review: An International Journal of Undergraduate Research and Criticism in the Discipline of English	1520–9679	3	1999	NA	University of South Carolina	https://scholarcommons.sc.edu/tor/
34. Discovery: Undergraduate Research Journal	2328–7039	3	2012	NA	University of Georgia	https://scholarworks.gsu.edu/discovery/about.html
35. Penn Bioethics Journal	2150–5470; 2150–5462	3	2004	NA	University of Pennsylvania	http://bioethicsjournal.com/
36. PennScience: Journal of Undergraduate Research		3	2009	NA	University of Pennsylvania	https://www.pennscience.org/
37. MarSci		3	2002	2018	University of South Carolina	https://sites.google.com/site/marscijournal/about-me
38. American Journal of Undergraduate Research	1536–4585; 2375–8732	2	2002	NA	Oswego State University, NY	http://www.ajuronline.org/
39. IMPULSE: An Undergraduate Journal for Neuroscience	1934–3361	2	2003	NA	Appalachian State University	https://impulse.appstate.edu/
40. Clocks & Clouds	2572–3146	2	2012	NA	American University	https://edspace.american.edu/clocksandclouds/
41. Indiana University South Bend: Undergraduate Research Journal	2379–5611	2	1998	NA	Indiana University South Bend	https://scholarworks.iu.edu/journals/index.php/iusburj
42. The Eagle Feather: A Publication for Undergraduate Scholars (TEF)	2332–4066	2	2004	NA	University of North Texas	http://eaglefeather.honors.unt.edu/
43. Pi Sigma Alpha Undergraduate Journal of Politics	1556–2034	2	2001	NA	Pi Sigma Alpha, the National Political Science Honor Society,	https://www.psajournal.org/
44. University of Florida Journal of Undergraduate Research (UFJUR)	1947–8836	2	1999	NA	University of Florida	journals.fcla.edu/ufjur
45. GEOverse	1758–3411	2	2008	NA	Oxford Brookes University, UK	http://geoverse.brookes.ac.uk/
46. Augsburg Honors Review		2	2008	NA	Augsburg University	https://www.augsburg.edu/honors/honors-review/
47. BlueSci journal		2	2004	NA	University of Cambridge	https://www.bluesci.co.uk/
48. Critique: A Worldwide Journal of Student Politics		2	2001	NA	Illinois State University	https://about.illinoisstate.edu/critique/
49. Georgia Political Review		2	2011	NA	University of Georgia	http://georgiapoliticalreview.com/
50. Journal of Science and Health at the University of Alabama (JOSHUA)		2	2002	NA	University of Alabama	https://joshua.ua.edu/
51. Northwestern Journal of International Affairs		2	1979	NA	Northwestern University	https://scholarlycommons.law.northwestern.edu/njilb/history.html
52. Journal for Undergraduate Research Opportunities (JURO)		2	2001	2018	University of Georgia	http://juro.uga.edu/2010about.html
53. Undergraduate Psychology Journal (Simon Fraser University)	2368–6340; 2368–6359	1	2014	NA	Simon Fraser University, CA	https://www.sfu.ca/psychology/ugrad/ujp.html
54. Biologos	1579–4350	1	2002	NA	Universidad de la Rioja, Spain	https://dialnet.unirioja.es/servlet/revista?codigo=17616
55. Stanford Undergraduate Journal (SURJ)	1751–4436; 1751–4428	1	2002	NA	Stanford University	https://surj.stanford.edu/
56. UCLA Undergraduate Science Journal (UCLA USJ)	0894–6167	1	1987	NA	University of California Los Angeles	http://uclausj.weebly.com/
57. Texas Undergraduate Research Journal (UT Austin URJ)	1538–9421	1	2002	NA	University of Texas, Austin	http://texasurj.com/wp/
58. Furman University Electronic Journal of Undergraduate Mathematics	0022–5339	1	1995	NA	Furman University	https://scholarexchange.furman.edu/fuejum/
59. Pittsburgh Undergraduate Review	0734–3140	1	1981	NA	University of Pittsburgh	http://purtemporarypage.weebly.com/
60. Journal of Undergraduate Chemistry Research	1541–6003	1	2001	NA	Virginia Military Institute	http://www.vmi.edu/academics/departments/chemistry/journal-of-undergraduate-chemistry-research/
61. Harvard Political Review	0090–1032	1	1969	NA	Harvard University	http://harvardpolitics.com/
62. Journal of Undergraduate Sciences	1523–8482	1	1999	NA	Harvard University	https://www.hcs.harvard.edu/~jus/home.html
63. Modern Psychological Studies	1076–0806	1	1992	NA	University of Tennessee Chattanooga	https://scholar.utc.edu/mps/
64. Journal of Undergraduate Reports in Physics (JURP)	0731–3764	1	2012	NA	Society of Physics Students (SPS) and Sigma Pi Sigma, the physics honor society.	https://www.spsnational.org/jurp
65. Catalyst: Rice Undergraduate Science Review		1	2008	NA	Rice University	http://ricecatalyst.org/
66. The Michigan Journal of Political Science		1	1981	NA	Michigan University	https://mjps.polisci.lsa.umich.edu/
67. UCLA Undergraduate Psychology Journal		1	2013	NA	University of California Los Angeles	https://urjp.psych.ucla.edu/
68. The Journal of Political Research		1	2010	2016	Creighton University	https://www.creighton.edu/ccas/politicalscience/undergradstudents/journalofpoliticalresearch/
69. Government and Politics Review		1	2010	2016	University College Cork, Ireland	https://www.ucc.ie/en/government-and-politics/governmentandpoliticsreview/aboutthejournal/
70. Journal of Psychology and the Behavioural Sciences		1	nf	nf	Fairleigh Dickinson University	https://www.fdu.edu/academics/colleges-schools/psychology/the-journal-of-psychology-and-the-behavioral-sciences/
71. Beloit Biologist		1	1982	2014	Beloit College	https://beloitarchives.libraryhost.com/repositories/2/accessions/418
72. The UBC Journal of International Affairs	1913–9322; 1913–9314	0	2010	NA	University of British Columbia, CA	https://issuu.com/ubcjia
73. Meteorite: The Student Journal of Philosophy	1099–8764	0	1998	NA	University of Michigan	https://meteorite.philosophy.lsa.umich.edu/
74. Saltman Quarterly: Undergraduate Biological Research Publication		0	2002	NA	University of California San Diego	https://sqonline.ucsd.edu/
75. The Dialectics: Journal of Leadership, Politics, and Society		0	2006	NA	Pennsylvania State University, Abington	https://sites.psu.edu/dialectics/
76. Journal of Undergraduate Research		0	1998	NA	University of Wisconsin La Crosse	https://www.uwlax.edu/urc/jur-online/
77. Student Journal of Health Sciences	1402–2230; 1402–2249	0	1998	2001	Linkoping University, Sweden	https://ep.liu.se/ej/sjhs/
78. Rose-Hulman Institute of Technology Undergraduate Math Journal		nf	2000	NA	Rose-Hulman Institute of Technology	https://scholar.rose-hulman.edu/rhumj/
79. The Online Journal of Undergraduate Reports in Physics		nf	1982	NA	Society of Physics Students	https://www.spsnational.org/jurp
80. The Xavier Journal of Politics		nf	2010	NA	Xavier University	https://www.xavier.edu/xjop/
81. Geoversity e-journal for Oxford Brookes University undergraduates	1758–8022	nf	nf	2011	Oxford Brookes University, UK	https://civismandcities.wordpress.com/2011/12/18/geoversity-e-journal-for-oxford-brookes-university-undergraduates/
82. Student CMJ		nf	Nf	2001	Croatian Medical Journal	announced 2001 but not found
83. The Morehead Journal Electronic Journal of Applicable Mathematics		nf	2001	2007	Morehead State University	https://scholarworks.moreheadstate.edu/mejam_archives/
84. Biosciences Undergraduate Research at Nottingham (BURN)		nf	2006	2009	University of Nottingham	https://rdmc.nottingham.ac.uk/handle/internal/309
85. Chrysalis: The Murray State University Journal of Undergraduate Research		nf	2005	2010	Murray State University	https://campus.murraystate.edu/services/ursa/Chrysalis.html
86. Critical Theory & Social Justice Journal of Undergraduate Research		nf	2003	2011	Occidental College	https://scholar.oxy.edu/handle/20.500.12711/152/browse?type=issue&value=urc_student
87. Interface: An Interdisciplinary Journal of Student Research		nf	1973	2000	Harvey Mudd College	https://ccdl.claremont.edu/digital/collection/ija
88. Inquiry: Undergraduate Research		nf	2000	2016	University of Arkansas	https://scholarworks.uark.edu/inquiry/
89. Motor City Review: A sociology journal of student research		nf	nf	nf	Wayne County Community College	nf

Note: ISSN = International Standard Serial Number (https://portal.issn.org/); NA = not applicable; CA = Canada; UK = United Kingdom; nf = not found.

). All journals, except the Croatian Ekscentar, publish articles in English.

The first year of publication was 1930 (BIOS: A Quarterly Journal of Biology) and the most recent starting year was 2015. About two-thirds (n = 45) of the journals had been started since 2000. The Pearson correlation between start year and number of citations was not significant at r = −0.21 (p = 0.07), and with the outlier 1930 journal removed, the association was likewise not statistically significant at r = −0.01 (p = 0.93). Hence, age of journal is not meaningfully associated with how frequently articles are cited.

A total of 13 journals out of 75 (17%) of the journals with Google Scholar and start year values were no longer published. The average length of publication for these URJs was 11 years, with a range from 3 to 32. Among the currently published URJs, the average length of publication was 20 years (Median = 18 years, SD = 12.87 years), with a range from 5 to 90 years.

Among the currently published URJs in our list with a Google Scholar h-index impact factor (n = 62), those in multidisciplinary topics predominated (n = 23, 37%), followed by those in the fields of Science, Technology, Engineering, and Medical Health Sciences (STEM) (n = 20, 32%) and Humanities (i.e., arts, politics, and economics; n = 14, 23%). Psychology had five journals (8%). The Humanities journals had the biggest average h-index (Mean = 4.50, SD = 5.75, Median = 2), followed closely by the STEM journals (Mean = 4.05, SD = 4.62, Median = 2), and the Multidisciplinary journals (Mean = 4.00, SD = 3.54, Median = 3). The Psychology journals had the lowest average (Mean = 2.80, SD = 2.68, Median = 1). Given that the standard deviations in each category were so large (Figure 2), it is not surprising that ANOVA revealed there was no statistically significant difference in impact (F_(3,57) = 0.18, p = 0.91).

Of the current URJs, 50 were sponsored by United States universities, 5 by student honor societies, and 11 were based outside the USA, including Canada (n = 3), United Kingdom (UK) (n = 3), Croatia (n = 2) and one each in Spain, Sweden, and Ireland. The most highly cited journal was the UK consortium journal Bioscience Horizons that was for a time published by Oxford University Press; however, publication ceased in 2018. Altogether, 11 journals had h-index ≥ 10. Of these, three were sponsored by student honor societies. The most frequently cited journal (BIOS) is the journal of the Beta Beta Beta Biological Honor Society, being published since 1930, and emphasises publication of undergraduate papers. The Psi Chi Journal is published open access by the International Honor Society in Psychology and welcomes submissions from undergraduate and graduate students, as well as faculty. These two journals also publish submissions from faculty, leaving open the possibility that the citations were to articles written by academic staff rather than students. The third journal (Undergraduate Research Journal for the Human Sciences) is sponsored by Kappa Omicron Nu the National Honor Society for Human Sciences and was published only between 2002 and 2016.

Three of the more frequently cited URJs (i.e., Journal of Young Investigators, Undergraduate Economic Review, and The Park Place Economist) are published at Illinois-Wesleyan University (IWU), a private U.S. liberal arts college. The Journal of Young Investigators (JYI) seeks to improve undergraduate science training by providing innovative, high-quality educational experiences in science writing, publication, and the peer-review process. The Undergraduate Economic Review publishes U.S. and international student content, while the Park Place Economist specialises in work from senior students at IWU. Also highly cited are the Columbia University Journal of Politics and Policy, The Plymouth Student Scientist from the University of Plymouth in England, and the no longer published Earth and Environment from University of Leeds. The Plymouth Student Scientist is an electronic journal showcasing undergraduate student research from the University of Plymouth’s STEM disciplines and was developed with external funding. It is interesting to consider that, despite the prestige of elite universities sponsoring some of these journals (e.g., Yale, Harvard, or Stanford), such sponsorship does not guarantee impact; the four journals associated with those three institutions had an average h-index = 1.75, maximum = 4.

4. Discussion

To the best of our knowledge, this study is the first comprehensive analysis of citation rates for URJs. While we recognise that citation rates might not be the best way to assess impact, this review of 75 URJs, 65 of which are currently active, does indicate that almost all of them get cited at least once, meaning that they can claim to contribute to the discipline. Our analysis also shows that neither age nor field of research contribute meaningfully to citation rate. Overall impact is understandably small. Nevertheless, for student authors the impact on careers could be substantial and for sponsoring institutions and organisations the prestige impact could be substantially valuable.

This paper highlights novel information on the citation rates of URJs which were identified in a scoping review of the field conducted some time ago. This allows researchers to see that URJs not only have plausible impact on students, faculty and connected institutions, but also contribute in some degree to the scholarly world. Although URJs have usually a small impact in terms of being cited, this does not mean that they are not valuable and successful in other domains. As Weiner [3] noted, success for an undergraduate research journal does not have to be seen in a citation rate.

An important function of URJs is as a venue for initiating new scholars into the full research cycle, including peer review and publication [17,24,25]. The current study does not allow us to evaluate that impact, a matter for future research. Examining postgraduate enrolment and careers of contributing authors would more robustly identify the impact of contributing to an undergraduate research journal. An examination of the impact of participation as a co-author or advisor by academic staff on their promotion, retention, or recruitment would better evaluate the benefit of such journals.

We had anticipated that URJs would be most prevalent in STEM fields; the current study shows instead that humanities fields and multidisciplinary journals outnumber the STEM area. Of course, a future study could work with the full panoply of contemporary journals and that may well show that journals in STEM topics are most prevalent. Nonetheless, there were no mean citation rate differences among the four identified topic areas, suggesting that the specific field or topic of undergraduate research journals should not concern institutions planning to develop a new journal.

It is worth noting that in some of the U.S.-based journals, honours students and programmes are categorised as undergraduate, meaning that such students can legitimately submit papers. Elsewhere, such students may be considered postgraduate (e.g., New Zealand). If more senior students qualify for an URJ, this may contribute to more frequent citation. Nevertheless, the heterogeneity of how undergraduate status is defined may influence why some journals obtain higher citation rates. In some of these higher-ranked journals, we found that co-publication with faculty was the norm. Hence, the notion of undergraduate research is not universal across these journals. A future study could classify and analyse the impact of pure undergraduate authorship versus blended authorship.

Putting aside the influence of the nation-wide or international Honor Societies, more impact was seen among public institutions, some of which would be considered regional in terms of reputation (e.g., Illinois Wesleyan U. and Penn State Berks-Lehigh Valley College). Hence, it would seem factors other than the status of a sponsoring university contribute to citation.

An issue revealed from our investigation was the lack of longevity among URJs. Sustaining journals is challenging when reliant on the enthusiasm of individual faculty members. There are several long-term successful journal exemplars that showcase the diversity of successful formulas associated with survival. The first and oldest is the journal BIOS published by Beta Beta Beta (TriBeta) an honor society for undergraduates from Oklahoma City University, a private university historically affiliated with the United Methodist Church. The journal is dedicated to improving the understanding and appreciation of biology. As a journal its success is likely attributable to its national TriBETA society association and therefore large catchment and readership. This arrangement means there is a steady pipeline of publication material related to honor society events as well as promoting its key undergraduate source of research papers.

In terms of scholarly impact, it is necessary to note some of the challenges posed by relying on Google Scholar as a data source. Sometimes results would include articles from different journals. This is a known feature of Google Scholar that, in being inclusive, it can capture non-relevant articles [22]. Where possible, entering unique journal ISSN values ensures data accuracy, but not all journals have an ISSN. Thus, our current review is constrained by that factor. A broader search for all undergraduate research journals, albeit time-consuming, would provide a fuller picture.

We did not have time or resources to examine in detail the features of the journals with or without ISSN and with or without impact factors. It would be interesting to identify characteristics of higher impact journals to ascertain their staffing model, scope, sponsor status, funding model, authorship of highly cited papers, and so on. The field needs a deeper understanding of what makes an undergraduate research journal highly cited. Further, this review has focused almost exclusively on URJs in the English-speaking academic world; it is entirely possible that in other languages quite different patterns of citation could be evident.

5. Conclusions

In conclusion, it appears that citations of URJ articles are not the purview of elite institutions, nor dependent on age of the journal or the field of research. The U.S. honor society mechanism seems specially to lend itself to generating research that is cited, although not all jurisdictions have that framework. Nonetheless, undergraduate research journals clearly do contribute to academic research because they are being cited.