Registered Reports

Registered Reports are scientific articles which are peer-reviewed before the research is performed and the data are collected.

The proposals that meet high scientific standards, such as rigor, soundness, significant importance and implications for the science community, are then provisionally accepted for publication before data collection starts.

This format helps to considerably reduce publication bias and to avoid researchers having to invest much time and effort into questionable research studies, while allowing enough flexibility to perform unregistered experiments and report serendipitous findings.

The review process for Registered Reports can be briefly summarized into two main stages.

Stage 1:

• Submission: Authors propose a clear hypothesis (including background, experimental methods, and analyses) that they plan to investigate.
• Peer review: Reviewers evaluate the registered study proposal before data are collected (research importance, rigor, soundness, etc.).

Following an in-principle acceptance (IPA), the authors are asked to perform the experiments, following exactly the peer-reviewed procedures.

Stage 2:

• Submission: Authors come back with a complete study, including the results and analyses.
• Peer review: Reviewers perform further evaluation, this time considering the full study and assessing compliance with the original protocol.

Guidelines for Authors

Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. This format is designed to minimize bias in deductive science, while also allowing complete flexibility to conduct exploratory (unregistered) analyses and report serendipitous findings.

The cornerstone of the Registered Reports format is that a significant part of the manuscript will be assessed prior to data collection, with the highest quality submissions being issued an IPA. Initial submissions will include a description of the key research question and background literature, hypotheses, experimental procedures, analysis pipeline, a statistical power analysis (or Bayesian equivalent), and pilot data (where applicable).

Initial submissions will be triaged by an editorial team for suitability. Those that pass triage will then be sent for in-depth peer review (Stage 1). Following review, the report will then be either rejected or accepted in principle for publication. Following an IPA, the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures. When the study is complete, the authors will submit their finalized manuscript for re-review (Stage 2) and will upload their raw data, digital study materials, and laboratory log to a publicly accessible file-sharing service. Pending quality checks and a sensible interpretation of the findings, the manuscript will be published irrespective of the results.

The review process for Registered Reports

Stage 1: Initial Manuscript Submission and Review

Stage 1 submissions should include the manuscript (details below) and a brief cover letter. Please note that the editorial board will not agree to send manuscripts for an in-depth review until a complete Stage 1 submission has been considered.

The title of the paper should include the words “Stage 1”.

The Stage 1 cover letter should include:

• A brief scientific case for consideration. Authors are encouraged to refer to the likely replication value of the research. Replication studies are welcome in addition to novel studies.
• A statement confirming that all necessary support (e.g., funding, facilities) and approvals (e.g., ethics) are in place for the proposed research. Note that manuscripts will be generally considered only for studies that are able to commence immediately; however, authors with alternative plans are encouraged to contact the Editorial team for advice.
• An anticipated timeline (typically within 12 months) for completing the study if the initial submission is accepted. Authors should contact the Editorial team for advice if longer time is needed.
• A statement confirming that the authors agree to share their raw data, any digital study materials, and analysis code as appropriate.
• A statement confirming that, following Stage 1 IPA, the authors will then proceed to conduct the study, adhering exactly to the peer-reviewed procedures.

Manuscript Preparation Guidelines – Stage 1

Initial Stage 1 submissions should include the following sections:

• Introduction
• A review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Please note that following an IPA, the Introduction section cannot be altered apart from correction of factual errors, typographic errors and altering of tense from future to past (see below).
• Methods
• A full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g., outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.
• A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly in the subsequent experiments or any Stage 2 manuscript can be rejected.
• Proposed analysis pipeline, including all preprocessing steps, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below).
• Studies involving Neyman-Pearson inference must include a statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature or theory. Since publication bias overinflates published estimates of effect size, power analysis must be based on the lowest available or meaningful estimate of the effect size. For frequentist analysis plans, the a priori power must be 0.9 or higher for all proposed hypothesis tests. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for “peeking” employed, and a final stopping rule for data collection outlined.
• Methods involving Bayesian hypothesis testing are encouraged. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified. For example, will you use a uniform up to some specified maximum, or a normal/half-normal to represent a likely effect size, or a JZS/Cauchy with a specified scaling constant? For inference by Bayes factors, authors must be able to guarantee data collection until the Bayes factor is at least 6 times in favor of the experimental hypothesis over the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however, to be eligible for advance acceptance this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field. For further advice on Bayes factors or Bayesian sampling methods, prospective authors are encouraged to read this key article by Schönbrodt and Wagenmakers (https://doi.org/10.3758/s13423-017-1230-y).
• Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the experimental hypotheses.
• A timeline for completion of the study and proposed resubmission date if Stage 1 review is successful must be included. Extensions to this deadline can be discussed with the Editorial Team.
• Any description of prospective methods or analysis plans should be written in future tense.
• Pilot Data
• Optional. Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot experiments will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered experiment(s).
• Secondary Registrations
• The journal welcomes submissions proposing secondary analyses of existing data sets, provided authors can supply sufficient evidence (e.g., self-certification; letter from independent gatekeeper) to confirm that they have had no prior access to the data in question. For advice on the eligibility of specific scenarios, authors are welcome to contact the editorial office.

Stage 1 submissions that are judged by the editorial board to be of sufficient quality and within journal scope will be sent for in-depth peer review. In considering papers at the registration stage, reviewers will be asked to assess:

1. The importance and soundness of the proposed hypotheses;
2. The suitability and feasibility of the experimental and analysis methodology;
3. Whether there are sufficient details given to replicate the proposed experimental procedures and analysis;
4. Whether there are sufficient outcome-neutral tests of the hypotheses, including positive controls and quality checks.

Following Stage 1 peer review, manuscripts will be either rejected outright, offered the opportunity to revise, or accepted. Proposals that exceed the highest standards of importance and scientific rigor will be issued an IPA, indicating that the article will be published pending completion of the approved methods and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results. We encourage authors to post Stage 1 reports on Preprints.org. Authors can then cite their Stage 1 preprint in the final Stage 2 manuscript and link it to Stage 1, indicating the manuscript is a registered report.

Authors are reminded that any deviation from the stated experimental procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. In cases where the pre-registered protocol is altered after the IPA due to unforeseen circumstances (e.g., change of equipment or unanticipated technical error), the authors must consult the Editorial team immediately for advice, and prior to the completion of data collection. Minor changes to the protocol may be permitted per editorial discretion. In such cases, the IPA would be preserved and the deviation reported in the Stage 2 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript (see below).

Stage 2: Full Manuscript Review

Once the study is complete, authors prepare and resubmit their manuscript for full review, with the following additions:

• The title of the paper should include words “Stage 2”.
• Cover letter. The Stage 2 cover letter must confirm:
• That the manuscript includes a link to the accepted Stage 1 preprint if available, or to the public archive containing anonymized study data, digital materials/code and the laboratory log. The cover letter should state the page number in the manuscript that lists the URL.
• That, for primary Registered Reports, no data for any pre-registered study (other than pilot data included at Stage 1) was collected prior to the date of the IPA. For secondary Registered Reports, authors should confirm that no data (other than pilot data included at Stage 1) was subjected to the pre-registered analyses prior to the IPA.
• Submission of anonymized raw data, digital study materials, and laboratory log
• Anonymized raw data and digital study materials must be made freely available in a public repository/archive with a link provided within the Stage 2 manuscript. Authors are free to use any repository that renders data and materials freely and publicly accessible and provides a digital object identifier (DOI) to ensure that the data remain persistent, unique and citable. Potential repositories include (but are not limited to), Figshare, Harvard Dataverse, and Dryad. For a comprehensive list of available data repositories, see http://www.re3data.org.
• Data files should be appropriately time stamped to show that data was collected after the IPA and not before. Other than pre-registered and approved pilot data, no data acquired prior to the date of the IPA is admissible in the Stage 2 submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline. Authors are required to upload any relevant analysis scripts and other digital experimental materials that would assist in replication.
• Any supplementary figures, tables, or other text (such as supplementary methods) can either be included as standard supplementary information that accompanies the paper, or they can be archived together with the data. Please note that the raw data itself should be archived (see above) rather than submitted to the journal as supplementary material.
• A basic laboratory log must also be provided outlining the range of dates during which data collection took place. This log should be uploaded to the same public archive as the data and materials.
• Background, Rationale and Methods
• Apart from minor stylistic revisions, the Introduction cannot be altered from the approved Stage 1 submission, and the stated hypotheses cannot be amended or appended. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g., correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of the IPA should be covered in the Discussion.
• Results and Discussion
• The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
• It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between the IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory Analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.
• Authors reporting null hypothesis significance tests are required to report exact p values and effect sizes for all inferential analyses.

The resubmission will most likely be considered by the same reviewers as in Stage 1, but could also be assessed by new reviewers. In considering papers at Stage 2, reviewers will be asked to decide:

1. Whether the data can test the original proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls);
2. Whether the stated hypotheses tested was the same as the approved Stage 1 submission;
3. Whether the authors adhered precisely to the registered experimental procedures or were able to sufficiently justify any changes;
4. Whether any new analyses (not mentioned at Stage 1) are methodologically sound and relevant;
5. Whether the authors’ conclusions are justified given the data.

Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results. Thus, while reviewers are free to enter such comments on the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria. Manuscripts that pass Stage 2 peer review will be published immediately.

Manuscript Withdrawal

It is possible that authors with an IPA may wish to withdraw their manuscript following or during data collection. Possible reasons could include major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, manuscripts can of course be withdrawn at the authors’ discretion.

Incremental Registrations

Authors may add experiments to approved submissions. In such cases, the approved Stage 2 manuscript will be accepted for publication, and authors can propose additional experiments for Stage 1 consideration. Where these experiments extend the approved submission (as opposed to being part of new submissions), the Editorial team will seek to fast-track the review process. This option may be particularly appropriate where an initial experiment reveals a major serendipitous finding that warrants follow-up within the same paper. In cases where an incremented submission is rejected (at either Stage 1 or 2), authors will retain the option of publishing the most recently approved version of the manuscript. For further advice on specific scenarios for incremental registration, authors are invited to contact the Editorial office.

Tips for Avoiding Desk Rejection at Stage 1

Many Registered Report submissions are desk rejected at Stage 1, prior to an in-depth review, for failing to sufficiently meet the Stage 1 editorial criteria. In many such cases, authors are invited to resubmit once specific shortcomings are addressed, although major problems can lead to outright rejection. To help minimize the chances of authors’ submissions being desk rejected, we list below the top ten reasons why Stage 1 submissions are rejected prior to review.

1. The cover letter doesn’t make necessary statements concerning ethics, data archiving, and so forth (see above).
2. The protocol contains insufficient methodological detail to enable replication and prevent researcher degrees of freedom. One commonly neglected area is the criteria for excluding data, both at the level of animals/participants and at the level of data within animals/participants. In the interests of clarity, we recommend listing these criteria systematically rather than presenting them in prose.
3. Lack of correspondence between the scientific hypotheses and the pre-registered statistical tests. This is a common problem and severe cases are likely to be desk rejected outright. To maximize clarity of correspondence between predictions and analyses, authors are encouraged to number their hypotheses in the Introduction and then number the proposed analyses in the Methods to make clear which analysis tests which prediction. Ensure also that power analysis, where applicable, is based on the actual test procedures that will be employed to test those hypotheses; e.g., don’t propose a power analysis based on an ANOVA but then suggest a linear mixed effects model to test the hypothesis.
4. Power analysis, where applicable, fails to reach the minimum level stated in journal policy.
5. Power analysis is over-optimistic (e.g., based on previous literature but not taking into account publication bias) or insufficiently justified (e.g., based on a single point estimate from a pilot experiment or previous study). Proposals should be powered to detect the smallest effect that is plausible and of theoretical value. Pilot data can help inform this estimate but is unlikely to form an acceptable basis, alone, for choosing the target effect size.
6. Intention to infer support for the null hypothesis from statistically non-significant results, without proposing use of Bayes factors or frequentist equivalence testing.
7. Inclusion of exploratory analyses in the analysis plan. Manuscripts proposing exploratory analyses will usually be desk rejected until such analyses are removed because inclusion of exploratory “plans” at Stage 1 blurs the line between confirmatory and exploratory outcomes at Stage 2. Instead, such analyses can be included at Stage 2 and need not be pre-registered. Under some circumstances, exploratory analyses could be discussed at Stage 1 where they are necessary to justify study variables or procedures that are included in the design exclusively for exploratory analysis.
8. Failure to clearly distinguish work that has already been done from work that is planned. Where a proposal contains a mixture of pilot work that has already been undertaken and a proposal for work not yet undertaken, authors should use the past tense for pilot work but the future tense for the proposed work. At Stage 2, all descriptions shift to past tense.
9. Lack of pre-specified positive controls or other quality checks, or an appropriate justification for their absence (See Stage 1 criterion 5). We recognize that positive controls are not possible with all study designs, in which case authors should discuss why they are not included.
10. Where applicable, lack of power analysis within proposed positive controls that depend on hypothesis testing.