Open Science

How do we fix the field? How do we address the Replication Crisis?
Open Science: Preregistration, Open Materials, and Open Data.

As described below, Open Science can address the main problems plaguing the psychological research literature.

In addition, there are calls for a priori power analyses, larger samples, samples from more diverse populations of participants, confidence intervals rather than p-values, stricter p-values, Bayesian statistics, and more. The benefits and relative merits of these changes are varied and worthy of deep discussion, but here we will focus on Open Science.

Preregistration

Whatever else we do, preregistration is the core foundation of Open Science.

Preregistration provides transparency.
By writing down their plan, a researcher makes their intentions clear and concrete. Then, when their research goes out for publication, the researcher can point to the plan and say one of three things:

• We followed the plan. You can trust our work.
• We have deviated from the plan in these specific ways. You can assess our reasons for changing and judge our work for yourself.
• We have abandoned the original plan for a specific reason. Given this, you should read our work as exploratory rather than confirmatory.

Preregistration neither binds nor traps the researcher.
Some researchers worry that, if they preregister, they will have to do what they wrote down, even if it turns out their plan was ill-conceived. This is not the case! We are all learning all the time and, sometimes, a researcher may plan to analyse data one way, but later learn that their plan isn't the proper or the ideal way to analyze their data. The data may turn out differently than expected and may break some assumptions the researcher made. The researcher may learn a new technique that is better-suited to these data. No problem! The researcher can always change what they actually do. The preregistration does not force them to do anything, it merely keeps track of their plan and any changes so that they have to be transparent.

A preregistration puts researchers in a position where they would need to disclose that they have deviated from the preregistration. They would need to explain how and why they decided to deviate.

Why would anyone want to keep their changes a secret?

It seems like the two major answers are embarrassment and, to a much lesser degree, fraud.

In the rare case of fraud, preregistration is working as intended!
We want to prevent fraud, as rare as it is, and transparency provides a way to do that.

In the case of embarrassment, the researcher may worry that they could look foolish in front of their peers for changing their plans. While this concern is socially valid, it isn't scientifically valid. We can address the social concern by creating new social norms: more preregistrations will naturally reveal that lots of researchers change their minds. We are always learning and learning is something to be proud of, not to be embarrassed about. The more we make these sorts of changes clear, the more we dispel the illusion of perfection, which we know isn't accurate. The more we are open, the more socially acceptable openness becomes.

Open Materials and Open Data

Open Science also includes Open Materials and Open Data.

Open Materials means that you share all your stimuli, questionnaires, tasks, and programs.

Open Data means that you share all the data you analyse, anonymized of course.

Open Science is the way forward

Openness and transparency are the only way to do science properly.
I don't like saying "the only way", but I don't know any other proper way to do science!

I recognize that Open Science doesn't fix everything.
Nothing can fix the low-quality research that has already been published. Open Science doesn't address systemic issues, like "publish or perish". Open Science cannot fix historical or social issues, but it can provide a transparent foundation for research. Different areas will also need to address additional changes, whether that be increasing sample sizes or changing over to Bayesian statistics.

Open Science seems daunting!
If you've never preregistered a study before, planning everything in detail, all in advance, can feel disconcerting. What if you make a mistake? Isn't it a lot of extra work?

We addressed mistakes above: you can just report what you changed. No harm, no foul. You're allowed to deviate from a preregistration, you just need to explain why you changed your mind.

The preregistration isn't extra work, it just front-loads the work you'd have to do anyway. You will eventually have to define your statistical analyses: coming up with the analysis plan for a preregistration is just as much time and effort as coming up with the analysis plan after you've collected data and immediately before you run the plan. You come up with the same plan, you just do it sooner and write it down. In fact, doing it sooner means that you have a chance to catch any mistakes or notice any opportunities before collecting data, not after it's too late. Planning ahead and writing a preregistration can help you include a variable you might have otherwise overlooked or could help you rethink the format you'll need for the output.

Starting anything new is always a bit daunting.
Open Science is no different. You'll get used to it and future-you will thank past-you for putting in the time and effort that saves you when you come back to a project and think, "What was I doing again?" The preregistration has your plan so you'll never lose your place.

Index

Return to Psychology The State of the Field

Continue to Where do research ideas come from

Jump to Undergraduate Psychology