Meta-analyses are supposed to combine the evidence on a topic from many studies (e.g., does melatonin help sleep?) to produce an answer. Many people even consider them the gold standard for evidence about scientific questions.
Unfortunately, taking a weighted average of many different studies sometimes doesn’t work because averaging the studies can be meaningless.
Suppose a meta-analysis on “meditation for depression” tries to average the results of a one-hour app-based mindfulness meditation and a one-week silent Vipassana meditation retreat. Obviously, these can’t be averaged. But, surprisingly often, meta-analyses try things like this. There is no such thing as “the effect of meditation on depression” – since “meditation” can mean a lot of different things, and the effect size may depend a lot on what precisely is meant by “meditation” (in terms of type, dosage, etc.).
An additional problem with meta-analyses is that they sometimes combine studies of very different quality. I think low-quality studies often add more noise and bias than signal (i.e., they have a low signal-to-noise ratio). So I don’t think they should be combined with high-quality studies.
I think that for a savvy reader of papers, sometimes a better approach than meta-analyses is to sort studies by quality (highest to lowest) and skim each, trying to update one’s view on the evidence study by study, taking into account how high-quality each appears to be.
A study with a larger number of study participants tends to be more reliable. A randomized controlled trial is better than a survey in most cases. A small p-value is more reliable than a bigger one. A study that tests the exact hypothesis of interest is better than one that tests something that’s only sort of like it. And so on.
Then, you can stop when you start to get to poorly-conducted studies.
An essential aspect of this process is ensuring you understand *exactly* what was studied in each paper since an intervention’s dosage, format, and duration can matter a great deal to an outcome. It’s not enough to know it was a study of “meditation” – what kind and for how long?
Of course, many people won’t have the expertise or time to go study by study – in that case, just try to be aware of what the meta-analysis is combining together if you need to rely on it! For instance, did it eliminate low-quality studies? Are the studies being combined comparable in the format, duration, dosage, and quality of the intervention being used?
It would be great if we could treat meta-analysis as providing definite answers to scientific questions – I wish that were the case! In reality, they have a lot of problems. In addition to those already mentioned, they often suffer from publication bias (i.e., studies that found no effect were not published, and hence the meta-analysis is missing them, meaning that the set of studies being average is biased towards ones that found an effect). There are methods to check for and correct for publication bias (so you should look for those when evaluating a meta-analysis), but even those methods are imperfect.
High-quality meta-analyses also include checks for “heterogeneity” to help make sure the studies aren’t too different from each other. Even these checks have problems, though (they tend to be used in a binary way to either accept or reject the studies being “too different,” and sometimes they find the studies are not “too different” even when the studies clearly are too unrelated to be averaged).
Of course, none of this means that meta-analyses aren’t useful at times – sometimes, they are really useful. One just has to use them with caution. High-quality meta-analyses have approaches to deal with many of these issues. It’s just that, pretty often, these issues are not addressed.
If you’re interested in the topic of problems with meta-analysis, Data Colada has a great series of essays on the topic: Meaningless Means – https://datacolada.org/105 by Uri Simonsohn, Leif Nelson, and Joseph Simmons.
This piece was first written on March 10, 2023, and first appeared on this site on July 16, 2023.
Comments