Committees or Markets? An Exploratory Analysis of Best Paper Awards in Economics

Abstract

Despite the general usefulness of citations as a sort of test of the value of one’s work in the marketplace of ideas, journals and publishers tend to use alternative bases of judgment, namely committees, in selecting candidates for the conferral of journals’ best paper awards. Given that recognition—sometimes in the form of compensation and on other occasions in the form of awards—in academe is geared toward incentivizing the production of impactful research and not some less desirable goal or outcome, it is important to understand the sensitivity in the outcomes of best paper award selection processes to the types of processes used. To that end, this study compares the selection of best paper awards for journals affiliated with several of the world’s top economic associations by committees to a counterfactual process that is based on citations to published studies. Our statistical exploration indicates that in most cases and for most awards, the most cited paper was not chosen. This requires further discussion as to the core characteristics that quantitatively represent the highest impact.

1. Introduction

A recent study by Hamermesh (2018) asserts that there at least two bases for the importance of citations to academic research. First, the research carried out by members of the academy has to be assessed in order to make administrative decisions related to tenure, salary, and hiring. Second, and perhaps more importantly, a relatively large number of citations to a scholar’s work reflects a professional career resulting in a noteworthy contribution to society (Hamermesh 2018). Hamermesh’s (2018) focus on citations is based on the weight of opinion from the early academic literature (e.g., Graves et al. 1982; Davis and Papanek 1984; Leibowitz and Palmer 1984; Laband and Piette 1994) that has tended to favor citations-based analysis as a “market test” of research productivity/impact.1 The studies that followed those early generation investigations (e.g., Laband and Tollison 2000; Kalaitzidakis et al. 2003; Mixon and Upadhyaya 2008) maintained support of the use of citations as a measure of impact both within and beyond the research community.2

Despite the general usefulness of citations as a sort of test of the value of one’s work in the marketplace of ideas, journals and publishers tend to use alternative bases of judgment, namely committees, in selecting candidates for the conferral of journals’ best paper awards. Given that recognition—sometimes in the form of compensation and on other occasions in the form of awards—in academe is geared toward incentivizing the production of impactful research and not some less desirable goal or outcome (e.g., see Kerr 1975), it is important to understand the sensitivity in the outcomes of best paper award selection processes to the types of processes used. To that end, this study aims to explore the efficacy of current best paper awards processes by comparing the selection of best paper awards for journals affiliated with several of the world’s top economic associations (e.g., American Economic Association, Econometric Society) by committees to a counterfactual process that is based on citations to published studies. Our statistical exploration indicates that in most cases and for most awards, the most cited paper was not chosen. This requires further discussion as to the core characteristics that quantitatively represent the highest impact.

Before turning to a discussion of our empirical exploration, we first provide a review of the economics literature on related areas of interest. This review is followed by a vignette concerning the Hicks-Tinbergen Award, which is conferred biennially to the authors of the best paper published in the official journal of the European Economic Association. Lastly, following a discussion of the study’s limitations and recommendations for future research, which is preceded by a section focusing upon the data and statistical analysis, we offer some concluding remarks.

2. Related Literature

Exploration of the economics of best paper awards does not, to our knowledge, constitute a current research stream in the economics literature. Thus, our literature review below focuses on related areas of interest, such as the economics of workplace awards, academic publishing, and academic committees. Each of these is discussed in turn within the three sub-sections below.

2.1. Some Economics of Workplace Awards

The conceptual framework for this analysis draws on the relatively new branch of research into the economics of workplace awards and recognition (Frey and Gallus 2017). Research into the economics of workplace awards demonstrates that they serve as direct incentives when employees pursue those activities that are required to earn them. Awards also provide indirect incentives as previous winners often serve as role models to employees (Frey and Neckermann 2009; Frey and Gallus 2014). For recipients (employees), awards often provide a financial reward that may not be taxed as heavily as other forms of compensation (Frey and Neckermann 2009). Field experiments in this area attest to the power of awards and recognition in motivating employees, as relatively minor awards have been shown to increase workplace productivity by more than 10 percent (Kosfeld and Neckermann 2011).3 A portion of this increased productivity may be due to the higher levels of employee cooperation and teamwork that awards can generate (Neckermann and Frey 2013).

The growing stream of economics research reviewed above also extends its lens to the power of awards and recognition in academia (e.g., Mazloumian et al. 2011; Chan et al. 2014a; Mixon et al. 2017; Chan et al. 2018), with particular attention to such accolades in the field of economics. A study by Chan and Torgler (2012), for example, reports that 14.4 percent of all individuals selected for a Fellowship of the Econometric Society (FES) before 1970 would also later become Nobel laureates. As the study concludes, this finding implies that a strong link exists between early receipt of an FES fellowship and becoming a Nobel laureate later in one’s career, perhaps through increased motivation (Chan and Torgler 2012).4 Follow-up work by Chan et al. (2014a) examines the importance of the conferral of economics’ John Bates Clark Medal by comparing John Bates Clark Medal winners’ post-award publication counts with those of a synthetic counterfactual control group of non-John Bates Clark Medal winners with similar human capital and productivity characteristics.5 As Chan et al. (2014a) report, by five years after conferral of the John Bates Clark Medal, the typical John Bates Clark Medal winner has produced 13 percent more weighted publications than a control group.6 In addition to this motivation-related effect, the conferral of such awards and honors has the added benefit of increasing access to greater research grant funding, lower teaching loads, and more talented colleagues, each of which supplements motivation in enhancing research productivity (Chan et al. 2014a).7

2.2. Some Economics of Academic Publishing

Some of the conclusions of the analysis presented in this study relate to economic aspects of the academic publishing process. An early study in this genre by Laband (1985a) posits that the length of journal articles is a function of article quality, the author’s reputation, average length of journal’s articles, and editorial discretion, where length is measured as the number of American Economic Review-equivalent-sized pages published across 5880 articles and notes issued from 1974 through 1976 in 39 leading economics journals. Results from the study indicate that authors from the same institutional affiliation as the editor are permitted more page space per article than are non-affiliated authors. One clear implication of this type of discrimination is, according to Laband (1985a), that departmental ranking schemes based on quantitative publication performance of faculties may be highly inaccurate given that they reflect discretionary editorial policies favoring departments with one or more affiliated journals.

Similar research by Mixon (1998) reports that the ordering (placement) of articles in printed issues of top economics journals is influenced by editorial connections. More specifically, the paper seeks to analyze the determinants of article placement in economics journals within the context of favoritism versus showcasing high-impact scholarly work (Mixon 1998). Employing data on article placement by editors from two quality economics journals (i.e., Journal of Political Economy and Quarterly Journal of Economics), ordered probit results suggest that favoritism may play some role in article placement by journal editors. Given that lead article position often conveys information on article quality, such choices may be inefficient and lead to false quality signals early in the life of journal articles (Mixon 1998).

The type of public choice research discussed above continues to the present day. A recent study by Colussi (2018) presents key stylized facts about author–editor connections in the publication process, employing a unique data set containing detailed information on the academic histories of all scholars who published at least one article, or served as editor in the leading general interest journals in economics, from 2000 through 2006.8 Focusing on four different types of author–editor connections, data presented in the study indicate that about 43 percent of the 1620 articles were written by scholars connected to at least one editor in charge of the journal at the time of the publication, while editors’ former graduate students and faculty colleagues authored 15 percent and 29 percent of the articles, respectively.9 Econometric estimates suggest that the number of articles authored by connected scholars increases when their connection is in charge as editor, with this effect being particularly relevant for faculty colleagues and former graduate students, who experience an increase of about 8 percent and 14 percent at the baseline, respectively (Colussi 2018).

There is, however, a body of work indicating that editorial connections lead to efficient outcomes in the publishing process (e.g., Hilmer and Hilmer 2011; Coehlo et al. 2014). Laband and Piette (1994), for example, compiled detailed information on 1051 articles published in 28 top economics journals (in 1984) in order to examine the extent to which an author’s personal ties to the editor of a journal influences subsequent citations to published articles.10 This was conducted in order to test two competing arguments about editorial favoritism. One theory holds that the publication standard applied by editors to papers submitted by authors with personal ties is lower than the one that must be met by individuals with no connection to the editor (Laband and Piette 1994). On the contrary, a good graduate training provides a good foundation for academic success. In addition, there is a consistent shortage of truly good papers authored by scholars in economics, and journal editors compete to identify and publish the few papers with relative and substantive impact. Editorial connections assist with this process, as the estimated impact of author/editor connection on citations is positive and highly significant (Laband and Piette 1994). Laband and Piette (1994) assert that this finding provides empirical support for the contention that the editorial process is competitive and that editors use their connections to actively search out high-impact papers for publication in their journals. A more recent study by Torgler and Piatti (2013) took a closer look at publishing in the American Economic Review. Their results also indicate a lack of support for the hypothesis that connections help authors to publish papers of lower quality.

In other research, Medoff (2003) examines articles published in six core economics journals in 1990, and finds that articles authored by those with editorial connections (e.g., service on the publishing journal’s editorial board) are statistically and numerically of higher quality than articles by those without such connections, and that this quality difference does not decrease over time. Lastly, Brogaard et al. (2014) employ publication and citations data for more than 50,000 articles from 30 major economics and finance journals in order to investigate whether network proximity to an editor influences research productivity. They find that during an editor’s tenure, his or her current university colleagues publish 100 percent more papers in the editor’s journal, compared to years when he/her is not editor (Brogaard et al. 2014). In contrast to editorial nepotism, the study reports that such “inside” articles have significantly higher ex post citation counts, even when same-journal and self-cites are excluded. As Brogaard et al. (2014) conclude, the results suggest that despite potential conflicts of interest faced by editors, personal associations are used to improve selection decisions.

2.3. Some Economics of Academic Committees

As Faria et al. (2021) point out, the academic labor market is noticeably segmented, as top academic departments often hire faculty with in-house doctoral degrees, while also attempting to attract newly-minted doctorates from similarly situated institutions, all in an effort to improve their reputations through publications in leading journals and larger citation counts. Just recently, for example, Yuret’s (2018) analysis of the educational backgrounds of almost 15,000 full professors from the top universities in the United States reveals that more than 70 percent of the professors obtained their doctorate degrees from only 20 universities. This type of market segmentation is also quite profound in the field of law. Segall and Feldman (2019) report that of the tenured or tenure-track law school faculties at Harvard University and Yale University holding domestic academic credentials, almost all hold degrees from among the top 10 institutions as ranked by U.S. News & World Report. Similar results hold for other top law schools, such as those affiliated with Stanford University, the University of Chicago, the University of Michigan, and the University of Pennsylvania.

The economics discipline is, as noted in the introduction above, also prone to labor market segmentation. A study by Klein (2005) utilizing international departmental rankings from Coupé (2003) reports that more than 80 percent of economics faculty at the top economics departments earned their doctorates at one of the top 35 economics departments. Additionally, the data indicate that these same top 35 economics departments draw more than 75 percent of their combined faculty from themselves (Klein 2005).11 Klein supplements these results by reiterating the earlier finding in Kocher and Sutter (2001), that the top 10 institutions conferred the doctorates of more than 50 percent of the authors in 15 top economics journals, and Coupé’s (2004) observation that the top departments have disproportional influence with book publishers, foundations, and government grant-making organizations.

A later study in this genre by Combes et al. (2008) analyzes the determinants of success at the concours d’agrégation en sciences économiques, which is a centralized hiring procedure through which professors of economics are selected in France. Using data from 1984 to 2003, these researchers focus in part on the role played by candidates’ networks—defined as professional links between candidates and the hiring committee members who make the recruitment decisions—when securing academic employment (Combes et al. 2008). They report that professional connections to hiring committee members is significantly related to the probability of securing employment. In fact, the effect of network connections is sufficiently strong that a substantial improvement in a candidate’s publication record is required to compensate for not being linked to the hiring committee (Combes et al. 2008).

Interestingly, there is some evidence that such hiring practices are not necessarily producing the desired outcome. In a study of the research productivity of new graduates from North American doctoral programs in economics from 1986 to 2000, Conley and Önder (2014) find that the rank of the graduate departments themselves provides a surprisingly poor prediction of future research success. For example, at the top 10 departments as a group, the median graduate has fewer than 0.03 American Economic Review-equivalent publications six years after graduation, a record they characterize as un-tenurable at a top department. Lastly, they also report that doctoral program graduates of equal percentile rank from certain lower-ranked departments have stronger publication records than their counterparts at higher-ranked departments (Conley and Önder 2014).

3. A Vignette: The Hicks-Tinbergen Award

Our exploratory examination of committee selection efficacy of best paper awards begins with a vignette focusing on the Hicks-Tinbergen Award, which was created by the European Economic Association (EEA) in 1991 and is awarded once every two years (i.e., on even-numbered years) to the author(s) of the best paper published in the Association’s official journal during the two preceding years.12 Through the EEA’s presentation of the 2002 Hicks-Tinbergen Award, the EEA’s official journal was the European Economic Review (EER). Beginning with the presentation of the 2004 Award, the group’s official publication was, and remains, the Journal of the European Economic Association (JEEA).13 The selection committee for the Hicks-Tinbergen Award is composed of the editor of the JEEA, the EEA’s past president and the EEA’s vice-president. This committee begins by seeking award nominations from the membership body of the EEA.14 The selection committee then discusses the nominations put forward by EEA members and makes a shortlist. The editors of the JEEA who viewed the short-listed papers are often called upon to evaluate the nominations. In the spring of each year in which the award is given, the committee informs the EEA’s Executive Committee of its decision.15 The winner is announced immediately after the meeting and a statement is posted on the EEA’s website.

All past Hicks-Tinbergen Award winners are listed in Table 1 along with their respective institutional affiliations, while the titles of the award-winning studies are listed in Table A1 in the Appendix A. The 15 prior awards have been shared by 34 economists, for an average of about 2.3 winners (authors) per award. On only two occasions—2008 and 2012—has the Award been conferred upon a single winner. These were Botond Köszegy of the University of California—Berkeley and Guido Tabellini of Bocconi University. Of the remaining 13 awards, seven have gone to teams of two, while the remaining six were received by teams of three or more. Table 1 also provides the number of Google Scholar citations per year for each of the 15 Award-winning articles. Garnering almost 286 citations per year, the 2004 awardees—Frank Smets of the European Central Bank and Raf Wouters of the National Bank of Belgium—produced the most impactful winning study to date. This study is followed by the 2018 study authored by Luigi Giusi of the Einaudi Institute for Economics and Finance, Paola Sapienza of Northwestern University and Luigi Zingales of the University of Chicago, and published in the JEEA, which has garnered about 180 citations per year. The least impactful award-winning study is the 1994 study by Robert Innes of the University of Arizona and Richard Sexton of the University of California—Davis. This study, which is published in the EER, has garnered only about 2.5 citations per year, yet is separated by only about 1.8 citations per year from the next least-cited award-winning paper by Juan Carrillo of Free University Brussels and Thomas Mariotti of the London School of Economics. The latter study was published by the EER in 2002 and has garnered 4.3 cites per year.

Table 1. Hicks-Tinbergen Award, 1994–2020.

Year	Winner(s)	C/Y	Top-Cited	C/Y	Cites Ratio
1992	A.P. Barten L.J. Bettendorf Catholic University of Leuven	10.06	Gene M. Grossman Princeton University Elhanan Helpman Tel Aviv University	74.10	0.1358
1994	Robert Innes University of Arizona Richard J. Sexton University of California—Davis	2.36	Marco Pagano Bocconi University	102.57	0.0230
1996	Jan van Ours Erasmus University Rotterdam Geert RidderFree University Amsterdam	6.62	David T. Coe International Monetary Fund Elhanan Helpman Tel Aviv University	290.85	0.0228
1998	Laura Bottazzi Bocconi University Paolo Pesenti Princeton University Eric van Wincoop Boston University	8.32	Alberto Alesina Harvard University Roberto Perotti Harvard University	172.92	0.0481
2000	Ernst Fehr University of Zurich Georg Kirchsteiger University of Vienna Arno Riedl Institute for Advanced Studies	25.00	Richard Clarida Columbia University Jordi Galí New York University Mark Gertler New York University	150.00	0.1667
2002	Juan Carrillo Free University Brussels Thomas Mariotti London School of Economics	4.30	Jeffrey D. Sachs Andrew M. Warner Harvard University	266.85	0.0161
2004	Frank Smets European Central Bank Raf Wouters National Bank of Belgium	285.89	Jean-Charles Rochet Toulouse 1 University Jean Tirole Toulouse 1 University	321.44	0.8894
2006	Gary Gorton University of Pennsylvania Frank A. Schmid Federal Reserve Bank of St. Louis	19.71	Rodrigo Cifuentes Central Bank of Chili Gianluigi Ferrucci Bank of England Hyun S. Shin London School of Economics	68.63	0.2872
2008	Botond Köszegy University of California—Berkeley	32.80	Daron Acemoglu Massachusetts Institute of Technology Philippe Aghion Harvard University Fabrizio Zilibotti Stockholm University	164.73	0.1991
2010	Denis Fougère Francis Kramarz Julien Puget National Institute of Statistics and Economic Studies	32.75	Alberto Alesina Filipe R. Campante Harvard University Guido Tabellini Bocconi University	96.00	0.3411
2012	Guido Tabellini Bocconi University	242.18	Thomas Dohmen Maastricht University Armin Falk University of Bonn David Huffman Swarthmore College Uwe Sunde University of St. Gallen Jürgen Schupp Free University Berlin Gert G. Wagner Max Planck Institute	329.50	0.7350
2014	Amy Finkelstein Massachusetts Institute of Technology Erzo F.P. Luttmer Dartmouth College Matthew J. Notowidigdo University of Chicago	60.88	Gianmarco I.P. Ottaviano London School of Economics Bocconi University Giovanni Peri University of California—Davis	219.33	0.2776
2016	Ricardo Puglisi University of Pavia James M. Snyder Jr. Harvard University	14.50	Wolfgang Dauth Institute for Employment Research Sebastian Findeisen University of Mannheim Jens Suedekum Heinrich-Heine-University Düsseldorf Düsseldorf Institute for Competition Economics	66.14	0.2192
2018	Luigi Guiso Einaudi Institute for Economics and Finance Paola Sapienza Northwestern University Luigi Zingales University of Chicago	180.40	Luigi Guiso Einaudi Institute for Economics and Finance Paola Sapienza Northwestern University Luigi Zingales University of Chicago	180.40	1.0000
2020	Jose Asturias Georgetown University Qatar Manuel García-Santana Pompeu Fabra University Roberto Ramos Bank of Spain	25.50	David Card University of California—Berkeley Jochen Kluve Humboldt University Andrea Weber Central European University	262.33	0.0972
means		63.42 (92.80)		184.39 (92.98)	0.2972 (0.3195)

In Table 1 are also listed the top-cited articles for each of the evaluation periods covering the 15 Hicks-Tinbergen Awards. First among these is the top-cited study published during the 2012 Award evaluation period, authored by Thomas Dohmen of Maastricht University, Armin Falk of the University of Bonn, David Huffman of Swarthmore College, Uwe Sunde of the University of St. Gallen, Jürgen Schupp of Free University Berlin, and Gert Wagner of the Max Planck Institute. The study, published in the JEEA, has received 329.5 citations per year since publication. This piece is closely followed by the 2004 top-cited study by Jean-Charles Rochet and Jean Tirole, both of Toulouse 1 University. Their study has picked up almost 321.5 citations per year since publication by the JEEA. Lastly, the least impactful top-cited study is the 2016 paper by Wolfgang Dauth of the Institute for Employment Research, Sebastian Findeisen of the University of Mannheim, and Jens Suedekum of both Heinrich-Heine-University Düsseldorf and the Düsseldorf Institute for Competition Economics. At more than 66 citations per year since publication by the JEEA, this study has still been more impactful than 12 of the 15 award-winning selections shown on the left-hand side of Table 1.

Next, Table 1 also includes the “Cites Ratio” for each award period for each of the 15 Hicks-Tinbergen Awards. This ratio represents the proportion of the top-cited study’s citations per year garnered by the award-winning study. As shown in Table 1, this ratio ranges from 0.0161 to one, meaning that the award-winning articles have garnered anywhere from about only 1.6 percent of the citations earned by the top-cited article to as many citations as the top-cited article. The former case occurred in 2002, with the award-winning paper by Carrillo and Mariotti and the top-cited paper by Jeffrey Sachs and Andrew Warner of Harvard University. The former has accrued 4.3 citations per year since publication by the EER, while the latter has garnered about 267 citations per year. The second case referenced above occurred in 2018, when the award-winning paper by Guiso, Sapienza and Zingales also became the top-cited paper during the same selection period.

Lastly, some summary statistics are presented along the bottom of Table 1. Among these are the means and standard deviations for citations per year for each category of papers—the Hicks-Tinbergen Award winners and the top-cited study during each two-year evaluation cycle. The mean number of citations per year across the 15 award-winning papers is 63.42, while that for top-cited papers stands at 184.39. The difference between these two means, or 120.97 citations per year, is treated stochastically. According to a means-difference test, this difference is, given a t-ratio of 3.57, greater than zero at the 0.001 level of statistical significance. In addition, presented at the bottom of Table 1 is the mean of “Cites Ratio,” which is 0.2972. This indicates that at the mean the award-winning papers have garnered just under 30 percent of the citations per year that have been garnered by the top-cited papers.

4. Data and Statistical Analysis

A broader exploratory analysis involves examination of the best paper awards at other economics journals. This study examines the best paper awards at four journals in the American Economic Association journal portfolio. These fall under the American Economic Journal titles, with specific entries subtitled Applied Economics (AEJAE), Economic Policy (AEJEP), Macroeconomics (AEJMa), and Microeconomics (AEJMi). In addition to these, we also examine best paper awards conferred by Quantitative Economics (QE) and Theoretical Economics (TE), two journals in the Econometric Society’s portfolio. Combined with analysis of the EEA’s EER and JEEA, our study includes data from three of the top economic associations in the world. In addition to these, we examine best paper awards from the Journal of the Association of Environmental and Resource Economists (JAERE) and the Journal of Environmental Economics and Management (JEEM), two journals that have been part of the Association of Environmental and Resource Economists’ portfolio.16 Next, our analysis includes best paper awards from the Economic Record (ER), an official journal of the Economic Society of Australia, Environmental and Resource Economics (ERE), which is affiliated with the European Association of Environmental and Resource Economists, and the International Journal of the Economics of Business (IJEB), which maintains an association with the Society of Business Economists.

The best paper awards listed above are administered using processes similar to that for the Hicks-Tinbergen Award.17 For example, the Econometric Society adopted its best paper prizes for QE and TE in 2015 in order to highlight the best paper published in each of the journals in the areas of quantitative economics and economic theory. Prior to 2019, the journals’ editors and co-editors selected a list of nominees, from which the associate editors elected the winning paper. In 2019, the process was enhanced, awarding the prize by an external committee alternating annually between QE and TE. Currently, each of these awards is presented to a paper published in the journal during the two calendar years immediately preceding the year in which the award is made. Similarly, to select the best paper published in the Economic Society of Australia’s ER in a given year, a selection panel reviews the published papers and evaluates them according to their relevance and importance, originality in the use of data and theory, elegance of method and exposition, and strength of policy conclusions. The panel then selects the best paper published that year.

Table 2 lists information on the best paper awards in economics examined in this study—including details on the Hicks-Tinbergen Award conferred originally for the best paper in the EER, and later for the best paper in the JEEA. The table also includes mean citations for the top-cited paper during each evaluation period for the best paper awards under study, as well as the mean citations per year accrued by the winning papers. In each of the 15 cases presented in Table 2, the top-cited paper’s mean exceeds that of the winning paper. In fact, the difference ranges from just over five citations per year to almost 167 citations per year. As with the Hicks-Tinbergen Award vignette presented in the prior section, these differences are treated stochastically by testing them against a null hypothesis of 0. As shown in Table 2, each of the 15 is greater than 0 at the 0.10 level of significance, while 13 (11) of the 15 are significant at the 0.05 (0.01) level of significance.

Table 2. Sample Means and Difference-in-Means Tests.

Journal Title	Association	Time Frame	Top-Cited Mean	Winner Mean	Difference in Means	p-Value
American Economic Journal: Applied Economics	AEA	2011–2020	222.02	88.97	133.05	0.009
American Economic Journal: Economic Policy	AEA	2011–2020	149.47	81.12	68.35	0.010
American Economic Journal: Macroeconomics	AEA	2011–2020	143.01	84.23	58.78	0.007
American Economic Journal: Microeconomics	AEA	2011–2020	77.55	11.08	66.47	0.000
Economic Record	ESA	2009–2019	39.98	1.69	38.29	0.082
EER/JEEA	EEA	1992–2020	184.39	63.42	120.97	0.001
Environmental and Resource Economics	EAERE	2012–2020	32.33	6.06	26.27	0.000
European Economic Review	EEA	1992–2002	176.22	9.44	166.78	0.001
International Journal of the Economics of Business	SBE	2014–2020	10.63	5.40	5.23	0.092
Journal of the Association of Environmental and Resource Economists	AERE	2014–2019	79.92	8.67	71.25	0.006
JAERE/JEEM	AERE	2008–2019	72.50	14.58	57.92	0.000
Journal of Environmental Economics and Management	AERE	2008–2013	63.60	21.68	41.92	0.008
Journal of the European Economic Association	EEA	2004–2020	189.83	99.40	90.43	0.042
Quantitative Economics	ES	2015–2019	47.04	9.26	37.78	0.043
Theoretical Economics	ES	2015–2020	39.44	11.36	28.08	0.006

Notes: EER = European Economic Review; JEEA = Journal of the European Economic Association; JAERE = Journal of the Association of Environmental and Resource Economics; JEEM = Journal of Environmental Economics and Management; AEA = American Economic Association; ESA = Economic Society of Australia; EEA = European Economic Association; EAERE = European Association of Environmental and Resources Economists; SBE = Society of Business Economists; AERE = Association of Environmental and Resource Economists; ES = Econometric Society.

Next, Table 3 presents the concordance ratios for each of the 15 best paper award entries in Table 2. For each best paper award, the concordance ratio represents the proportion of cases where the best paper award was conferred upon the top-cited paper. As indicated in Table 3, the mode of this ratio is 0, with seven of the 15 entries representing award processes wherein the top-cited paper was never selected for the best paper award. The largest concordance ratio of 0.30, belonging to both the AEJAE and AEJEP, indicates that 30 percent of best paper award selection processes involving these two journals resulted in the top-cited paper receiving the best paper award. The concordance ratios for these two journals only marginally exceeds that of 0.29 for the IJEB, while the remaining concordance ratios are relatively small, ranging from 0.05 to 0.11.

Table 3. Concordance Ratios and Significance Tests.

Journal Title	Association	Time Frame	Concordance Ratio	p-Value
American Economic Journal: Applied Economics	AEA	2011–2020	0.30	0.030
American Economic Journal: Economic Policy	AEA	2011–2020	0.30	0.030
American Economic Journal: Macroeconomics	AEA	2011–2020	0.10	0.152
American Economic Journal: Microeconomics	AEA	2011–2020	0.00	0.500
Economic Record	ESA	2009–2019	0.00	0.500
EER/JEEA	EEA	1992–2020	0.05	0.154
Environmental and Resource Economics	EAERE	2012–2020	0.00	0.500
European Economic Review	EEA	1992–2002	0.00	0.500
International Journal of the Economics of Business	SBE	2014–2020	0.29	0.063
Journal of the Association of Environmental and Resource Economists	AERE	2014–2019	0.00	0.500
JAERE/JEEM	AERE	2008–2019	0.00	0.500
Journal of Environmental Economics and Management	AERE	2008–2013	0.00	0.500
Journal of the European Economic Association	EEA	2004–2020	0.11	0.159
Quantitative Economics	ES	2015–2019	0.00	0.500
Theoretical Economics	ES	2015–2020	0.00	0.500

Notes: EER = European Economic Review; JEEA = Journal of the European Economic Association; JAERE = Journal of the Association of Environmental and Resource Economics; JEEM = Journal of Environmental Economics and Management; AEA = American Economic Association; ESA = Economic Society of Australia; EEA = European Economic Association; EAERE = European Association of Environmental and Resources Economists; SBE = Society of Business Economists; AERE = Association of Environmental and Resource Economists; ES = Econometric Society.

As before, these differences are treated stochastically by testing them against a null hypothesis of 0. As shown in Table 2, none of the concordance ratios reaches the 0.01 level of significance, while only those that are greater than 0.29 (0.28) achieve the 0.05 (0.10) level of significance. Of the 15 entries in Table 3, only two (three) achieve the 0.05 (0.10) level of significance. Thus, those results indicate that committees are not picking the most successful article in terms of citations. Lastly, and relatedly, the exploratory analysis discussed above provides a rationale for awards committees’ consideration of alternative processes for recognizing the authors of meritorious research. One avenue for consideration is to delay the conferral of awards in order to allow for a market-based determination of the merits of each published paper in the volume(s) of a journal. Given that any delay further increases the time between the production and conferral of awards, perhaps as little as a five-year delay would be sufficient to provide an indication to awards committees of the relative merits of a set of publications.

5. Limitations and Recommendations for Future Research

The analysis presented above in this study is not without limitations. As has been discussed in the scientometrics literature, citations can, in some cases, be “arbitrary,” which makes it difficult to achieve the level of precision proposed in this study. For example, the often abundant literature on a subject obliges researchers to make a decision, often arbitrarily, on the references cited. Similarly, a researcher may not be able to identify all of the literature in his or her field, especially in the current context with the multiplication in the number of journals, and, therefore, he or she may ignore relevant publications in the list of bibliographical references.

Added to these issues are the problems related to the network effect, and to citations to authors by virtue of their fame. The Matthew effect, for example, implies the existence of a boundary to be reached in order to increase citations (see Schneider 2013; Waltman 2016; Thelwall 2017; Thelwall and Fairclough 2017). These problems relate to the historical debate over the use of citations versus other measurements, such as publication counts (e.g., see Gerrity and McKenzie 1978; Graves et al. 1982; Leibowitz and Palmer 1984; Davis and Papanek 1984; Laband 1985a, 1985b, 1985c; Berger and Scott 1990; Gibbons and Fish 1991; Conroy et al. 1995; Scott and Mitias 1996; Mixon and Upadhyaya 2001, 2016a, 2016b, 2019), as well as to modern declarations to several initiatives (e.g., DORA, Leiden manifesto) cautioning the university community against using citations for the individual evaluation of researchers (e.g., see Bladek 2014; Hicks et al. 2015). These issues suggest the use of citation classes instead of a single bibliometric index. Future work in this area might examine if prize-winning papers are, for example, among the top one percent (five percent) most cited in the topic. If so, one might assume that there is a concordance between objective measurement (i.e., citations) and subjective measurement (i.e., committees).

Next, our analysis does not control for article type or topic. For example, review articles are generally highly cited, whereas they constitute only an analysis of a collection of existing studies. As such, it is unlikely that they are selected for best paper awards. Although this issue may not be prevalent in the field of economics, wherein review articles are both less common than in other fields and there is an outlet, the Journal of Economic Surveys, dedicated to such work, it would be an important consideration if our approach is applied to other fields, which is an obvious avenue for future research. Similar issues may also exist with regard to the type of article (e.g., theoretical vs. empirical), the openness status of the article (i.e., open access vs. subscription), the number of authors and representation of countries, and the interdisciplinary nature of the study. Future research might control for these elements in the analysis.

6. Conclusions

What constitutes a high-quality paper is not an easy question. Committees and scientific organizations have the difficult role of assessment when deciding how to recognize the best papers. One may argue that the value of a publication is not just defined by its citations, but then the question arises as to what other objective selections play a role in the selection process. The selection of a single paper over many others can be seen as a complex task and opinions within the committee may vary a lot, taking into account the large number of potential candidate papers. Recognizing the best papers can be a reflection of how well an academic society or the committee members are able to identify scientific impact or significant achievement and can itself be seen as a catalyst or sign of a successful association or journal. The result of the exploratory analysis discussed in this paper indicates that in most cases and for most awards, the most cited paper was not chosen. This requires a discussion as to the core characteristics that quantitatively represent the highest impact. How to best identify and recognize the best papers is of practical importance in science.