Reading Methods

When reading methods sections, read the original questionnaires if you can find them.
When a researcher reports that they used a questionnaire when measuring a construct of interest, read the full questionnaire and consider whether you believe that this questionnaire measures what what the authors claim to measure.

Reading Results

When reading results sections, if an author uses quadratic predictors in a regression and treat them as anything other than exploratory, you should be very dubious unless the author pre-registered this model. Quadratic effects are likely to be incorrect in most cases.

Reading Discussions

Discussion Section Red Flags

  • Discussion section refers to p-value results that were slightly above 0.05 as "marginal" or "trending". Recall that p < 0.05 is already a very liberal threshold compared to other sciences. This is not valid.
  • Discussion section refers to results where p-values were not statistically significant as evidence that the effect-size is zero, e.g. phrased as "there is no effect". This is not valid.
  • Discussion section treats post hoc follow-up tests as if they were a priori confirmatory tests. This is not valid.

Example: Reading Methods: Control Conditions

Check the control condition.
Is it reasonable? Does it match the claims?

The control condition defines what you can learn about mechanisms.
Lets say I want to test the effect of meditation on well-being as measured by the difference between pre-intervention and post-intervention on a questionnaire. Lets say you check the scale and you agree that it measures well-being. Lets also say the authors describe their meditation intervention and their control condition so completely that you believe you would be able to re-create their experiment (replicability).

When it comes to results, lets say the meditation group has higher well-being scores after the intervention compared to the control conditions.
What might those higher scores mean?

Control condition: None

We learn almost nothing. We learn that doing this meditation probably didn't make people worse, but we cannot know what made well-being increase because we don't have any control condition.

Control condition: Waitlist

One group is assigned to a list while the other meditates. This means that the control group is (a) spending time. We learn that doing something compared to nothing increased well-being. We don't know that the increase had anything to do with the meditation per se, though. The increase could have been for other reasons, e.g. that participants did any intervention, that they interacted with researchers, that they believed they would get better, etc.

Control condition: Tetris

One group plays Tetris while the other meditates. This means that the control group is (a) spending time (b) doing something. We learn that doing something like meditation increased well-being more than something like Tetris and that the increase in well-being wasn't just time passing. We don't know how or what specifically about the meditation. Maybe the meditation group had more social support, maybe it was time away from screens, or maybe it really was the meditation. We don't know.

Control condition: Introspective Journaling

One group is journaling about their day while the other meditates. Now we've got two conditions that are getting closer. The control group is (a) spending time (b) doing something (c) private and introspective. We learn that there is something different about reflecting on experience in journaling compared to the practice of meditating. As above, we learn that the increase in well-being wasn't just because time passed. We don't know how or what specifically about meditation made the difference, but the increase in well-being doesn't seem to be just about taking time or thinking about your day.

Control condition: A second kind of meditation

One group is doing a different kind of meditation while the other group meditates. We have two groups that are meditating and results show that one group's well-being increases more than the other group's well-being. These two conditions are very similar. This similarity lets us look at the differences in outcomes as potentially associated with the limited number of differences between interventions. We still don't know how, but we can start to understand which specific aspects of the two meditations may have caused the differences in well-being. Maybe one group is doing a practice where they focus on their breath and the other group is focusing on a mantra, but otherwise the meditation instructions are the same: that could lead us to conclude this the focus of attention itself is a key factor, which is something we couldn't conclude from the other control condition examples above.

Hopefully these detailed examples of different control conditions help illustrate how we have different limitations trying to learn about different properties by using different control conditions.

Theory Papers

For considering theory papers, you might also want to consider individual differences and edge-cases.

If some finding or claim goes against your intuition or just feels off, don't ignore yourself. Be willing to change your mind, but be critical of the evidence. Ask for clarification where you want it.

If some finding strongly supports your intuition or just feels right, still question yourself! As Richard Feynman said about science, "The first principle is that you must not fool yourself and you are the easiest person to fool."

Maintain a healthy skepticism. Even when you accept something scientific, don't think of it as a fundamental universal capital-T Truth. Think of it as a belief with decent evidence that could be overturned someday when better evidence or ideas come along. Retain intellectual humility. Don't forget that we're hairless monkeys hurtling through space on a damp rock.

Index

Return to Reviewing papers

Jump to Conference Advice