Meehl Predicted Ego Depletion's Failures
New research with 3500+ participants finds no effect. Meehl predicted this in the 60's.
Over at my Psychology Today blog, I’ve written up the results of a new large-scale replication report on ego depletion—a modern research take on willpower. The headline result is that, despite getting buy-in from experts in the ego depletion literature, including video instructions and direct consultation by one of the world’s leading ego depletion researchers, no significant effect was found. Using Bayesian analysis, results indicate that a model with no effect is four times as likely as a model showing an effect.
Recently, I’ve been writing responses to a critique about the current Metascience movement that it is lacking theory. I’ve tried to push back against this by arguing that Metascience has intentionally tried not to impose a top-down view of what science should look like onto the psychology research community, and that instead it’s relied on reports by working scientists of what they take their own goals to be. Reflection on the sociology of science piece has also led me to think about what theory Metascience does rely on, and was the inspiration for my PT article.
Paul Meehl is as close to a philosophical guidepost for modern Metascience as anyone. He was a working psychologist who was well-regarded, highly published, and influential in the field—and he wrote about issues in philosophy of science in terms that resonated with psychologists, talking in terms that we understand. The problems he laid out, like so many laid out by previous generations of psychology critics and would-be reformers, were not refuted but allowed to fade away through inaction. Yet reading 50-year-old articles by Meehl (as I first started to during a summer break from my PhD coursework) reveals just how timely they still feel.
As I recapped some of Meehl’s arguments at PT:
The problem is in the way psychologists build up a body of evidence. They typically report results of studies that come out in the expected direction excitedly, as confirmation of the theory. When results don’t come out in the expected direction, some reason is typically found to dismiss them. As Meehl wrote, there is “a fairly widespread tendency to report experimental findings with a liberal use of ad hoc explanations for those that didn’t ‘pan out.’ This last methodological sin is especially tempting in the ‘soft’ fields of (personality and social) psychology, where the profession highly rewards a kind of ‘cuteness’ or ‘cleverness’ in experimental design.”
Meehl saw that it was entirely possible for a researcher to build an entire career publishing studies in an area without ever refining our understanding of the area one bit. Another quote from the PT piece:
Across a series of studies, a research group can claim that the theory was supported after every positive result, but that there was some alternate explanation that needs to be investigated for every negative result. It’s a “heads I win, tails you lose” method of theory building. Using this method, “a zealous and clever investigator can slowly wend his way through a tenuous nomological network, performing a long series of related experiments which appear to the uncritical reader as a fine example of ‘an integrated research program’ without ever once refuting or corroborating so much as a single strand of the network.” This method can’t ever move a theory closer to the truth, because it can’t rule out anything as wrong. As Meehl sees it, this pattern can lead to “a long publication list and … a full professorship” but with an enduring contribution of “hardly anything.”
You could say that Meehl predicted the ego depletion failure (I did in my headline!), but it’s probably more accurate to say that Meehl was observing a problem that was already present in the late 60’s—a problem that has never fully been dealt with. There are famous, influential psychology theories (and theorists) that just haven’t contributed that much to our understanding of anything. Science is a career with its own internal politics, and you can be very successful in science as a career while contributing very little to science as an intellectual endeavour.
Crucially, Meehl is laying out a theory for why we might expect results not to hold up in psychology. His 1967 article (which I quoted at PT) lays out specific features of the way psychology proceeds in theory development that will ultimately prevent it from being successful. These include features particular to psychology (such as the “crud factor” whereby it seems all self-reported psychology variables are at least a little correlated with each other), and issues in scientific education (such as not understanding that Popper’s modus tollens formulation means that null results—zero effects—should weigh much more heavily against a theory than they do). This is a sociological explanation for why psychology isn’t progressing more quickly, and I take it as implicit that this is basically the theory that most Metascientists adopt.
Now, you can definitely argue that the Meehl was too into falsificationist thinking, and that actually doing “critical tests” of current psychology theory is misguided. In fact, that’s a position that Meehl later advocated. In a 1990 essay, he wrote:
If the reader is impelled to object at this point ‘Well, but for heaven’s sake, you are practically saying that the whole tradition of testing substantive theories in soft psychology by null hypothesis refutation is a mistake, despite R.A. Fisher and Co. in agronomy,’ that complaint does not disturb me because that is exactly what I am arguing.
So the ego depletion failure—and the fact that hundreds of studies across decades of research are now being called into question—does have a theoretical explanation, but it doesn’t mean that we should just start running more carefully controlled null hypothesis tests. There are good reasons to believe that these sorts of “support versus doesn’t support” distinctions aren’t very helpful in getting a deeper understanding of what’s really going on, and we’d be better off doing research that isn’t based on whether a result is or isn’t significant.
Having a single up or down vote on ego depletion, as was done in this replication, might seem to be at cross-purposes with “real reform” or “deeper reform” as Meehl advocated in later writings, and as many modern thinkers like Berna Devezer, Iris van Rooij, and Olivia Guest have advocated. Yet I think that psychology needed these types of public failures to spur change and deeper thinking about how to do better science. As Maarten Derksen put it on twitter: “I think what the replication projects of the last decade have done is also to create problems, rather than solve them. In that way they have been very productive.” Research psychologists have been happy to go on using methods with unrefuted criticisms for generations, so perhaps what was needed was some outcry—both from within and without the scientific community—that this approach doesn’t work.
More concretely, what ego depletion’s failure to replicate shows is not that there is no such thing as “willpower” or “self-control” but rather that we can’t keep studying it the way we have been. The next step shouldn’t be rush out and figure out the next experimental paradigm that can give us significant results. Instead, we should step back to examine how we could respond to Meehl’s recommendation to make more specific predictions, and modern calls for thinking through theory a bit before deciding what study to set up. Replication failures don’t tell us that phenomena or effects aren’t real. They tell us that satisfaction with our approach isn’t.