Cliché study critiques
When asked to critique a study, undergraduates often offer several clichés:
- Sample size
- WEIRD sample / culturally limited sample
- Lack of control for certain variables, especially any easy demographics (e.g. sex, gender, SES, etc.)
- Any perceived or imagined demographic imbalances (or not testing for differences by demographics)
With each of these, the most common cliché is the meta-critical issue that undergraduates fail to describe why their critique is relevant. I've seen criticisms of sample size when the sample size was 300 participants, which was totally adequate for the design. I've seen people say a study should have tested for gender differences when there was no theoretical reason to do so.
Students often regurgitate a few critiques they've heard could be limitations: they don't actually understand the underlying nature of research design and its flaws.
I would be very surprised to see a critique of a discussion section.
I don't think I've ever seen an undergraduate student critique that mentioned how an author extrapolated beyond what was a reasonable generalization from the data.
I think critiques of method sections are also important, but students should consider the whole package.
A study's methods determines what conclusions can be reached from the data that gets collected. The methodology put limits on what a reasonable discussion section can say so a decent critique should consider the methods and the discussion.
Imagine a study that doesn't find a significant effect with a null-hypothesis significance test, then frames this result in the discussion as evidence that "there is no effect". This interpretation would be incorrect and worthy of criticism: absence of evidence is not evidence of absence. The methods of the study failed to find evidence of an effect, but such a methodology is fundamentally different from a method that seeks evidence that there is no effect. Different statistical methods are needed to report "no effect".
A decent critique is holistic.
A decent critique needs to explain why the critique is relevant.
Small sample size could be relevant, but if someone just says "small sample", that doesn't actually demonstrate any critical reasoning. Similarly, some papers don't have proper control groups, but selecting an appropriate control group is non-trivial and is methodologically linked to the conclusions one wishes to (and is capable of) drawing from an experiment.
If I were to teach a research methods course, I would ask students to write Constraints on Generality (COG) statements for various papers, then discuss them. It is important to ensure that students learn about philosophy of science so they can understand the limitations of inquiry, generalization, and of different methodological approaches. Undergraduates should learn these skills so that, when they see an article in the news, they can be appropriately skeptical and draw informed conclusions.
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
Index
Return to Undergraduate Psychology
Continue to What do you wish you knew before grad school?
Jump to Graduate Psychology