Pre-Registration in Experimental Economics: An Editorial Policy

1. Motivation

First and foremost, pre-registration is not the all-in-one solution for experimental economics. Pre-registration alone cannot overcome all methodological crises, such as the ongoing replicability crisis, reproducibility crisis, publication bias, etc. Even if strict pre-registration requirements were established with immediate effect across all journals publishing experimental economics papers, scientists could and likely would still abandon projects without clear results, run pilots, and carry out many other practices that would contribute to some of the aforementioned phenomena. Ultimately, there is no substitute for good taste, scientific integrity and honesty, and scientific misconduct will not be stopped, perhaps not even reduced, by any pre-registration policy.

Nevertheless, there is great benefit in pre-registration if the aim of a study is clearly set out to be that of falsifying a concrete theory or prediction, or of reproducing and replicating a prior study. That reason that this is the case succinctly relates to two types of empirical research that Karl Popper describes in his study on falsification and scientific method [1,2]. One of the two uses of empirics is as a basis to formulate new theories, i.e., for hypothesis generation. Such research may qualitatively describe data or make use of formal statistics, including the language of confidence and significance, yet clearly such research is exploratory and usually requires “playing around” with data as a theory is formulated in co-evolution with identifying patterns in the data. Perhaps the most important example of such research is Gregor Mendel’s experimentation with peas [3], the pioneer of modern genetics. In experimental economics, one example of a seminal hypothesis-generating experiment that was used for theory proposition is Ernst Fehr and Klaus Schmidt’s experimentation underlying “A theory of fairness, competition, and cooperation” [4]. That paper reproduces the falsification of narrow material self-interest of seminal earlier experiments, such as those from the Ultimatum and Dictator Games by Kahnemann, Knetsch and Thaler [5], and by Forsythe, Horowitz, Savin and Sefton [6]. It then proceeds to propose a theory that fits their own data, and qualitatively matches data from the aforementioned earlier studies.

Empirical observations, such as Mendel’s or Fehr-Schmidt’s, permit novel theory formulations without requiring high levels of power, and pilot studies may be run before scientists settle on the right experimental framework for their objectives. Falsification is not the goal of such studies since, in any case, novel theories are eventually formulated to match the patterns from data. According to Karl Popper, what must come next in the scientific process is that scientists, ideally not those who might have vested interests in the survival of their own theories, should reproduce and replicate original data analyses and then derive “risky” predictions from these theories, run controlled experiments to test them, and ideally falsify them. Any failure to falsify a theory, especially based on risky and novel predictions, corroborates the original theory, and we can think of this as increasing the “worth” of the scope for future falsification attempts.

Pre-registration is ideally suited for a hypothesis-testing kind of experiment (

Table 1. Analysis of hypothesis-testing.

	Analysis: Exploration/Auxiliary Experiment/Illustration/Fitting New Theory	Replication/Reproduction/Falsification/Testing Existing Theory
Pre-registration required?	No	Yes
Provisional acceptance of pre-registered experiments possible?	No	Yes

). Its aim is to falsify existing theories (as opposed to the hypothesis-generating experimentation that is used to formulate or illustrate a new theory). In the history of science, Albert Einstein’s thought experiments hold a special place of this kind of approach, as they are essentially pre-registrations of experiments to be conducted later. In economics, Vernon Smith’s counterexample-type experiments are an example. These experiments show that markets may converge to Walrasian competitive equilibria, which famously falsify the predominant view at the time that complete information and perfect knowledge are prerequisites for market efficiency [7].

With this Popperian distinction in mind between hypothesis-generating and hypothesis-testing experimentation, we can better understand one of the key problems with the scientific practice of experimental economics, which is a lack of appropriate language and norms for presenting results that clearly demarcate between the two approaches. In reality, many experimental papers in economics contain both kinds of analyses without a clear differentiation. We must agree on a language for presenting results that renders hypotheses-generating or a hypotheses-testing analyses unambiguous, the latter ideally being pre-registered, although the same does not apply to the former. Perhaps a certain language should be reserved for pre-registered analyses. Thus, pre-registration would be a tool, not a dogma.

2. Editorial Policy

For experimental papers to be submitted to Section Learning and Evolution in Games, the following editorial policy for experimental papers is implemented as of 1 January 2023:

• By submitting an experimental paper to the section, authors make a “declaration of honor” that all data and analysis files will be made available along with publication of the article and that all other relevant data, including all data from pilots, is appropriately described in the article. If authors wish not to make their data and/or analysis files available, or if authors prefer not to disclose all data/pilots, a separate statement explaining the reasons needs to be submitted and will be appended to the article for review and publication purposes.
• Submissions should label themselves as either hypothesis-generating or as hypothesis-testing. Authors are encouraged to pre-register all hypothesis-testing projects. All pre-registered analysis should be reported in the main sections of the paper. Other analyses should be labelled as exploratory.
• Pre-registered experimental projects can be submitted instead of full papers and will be peer-reviewed with the possibility of provisional acceptance, without the inclusion of results from the pre-registered analysis, both before and after data collection.