Systematic Reviews and Meta-Analyses: How to Read Them
When a supplement brand or a fitness influencer says "a meta-analysis proves it works", the phrase sounds final. In reality, systematic reviews range from rigorous syntheses that change clinical practice to quick compilations of weak studies dressed up in statistics. Our editorial team explains how to tell the difference and what to look for before you trust a headline.
Reviews, systematic reviews and meta-analyses
A traditional narrative review is an expert's summary of a field. It can be insightful, but the author chooses which studies to discuss, and there is no guarantee that inconvenient findings are included. Narrative reviews are useful for orientation, not for settling whether something works.
A systematic review attempts to remove that selectivity. It starts with a pre-specified question, often framed using PICO (population, intervention, comparison, outcome), defines inclusion criteria in advance, searches multiple databases comprehensively and documents every step. Ideally the protocol is registered beforehand, for example on PROSPERO, so readers can check whether the authors changed their plan after seeing the data.
A meta-analysis is a statistical technique that may or may not be part of a systematic review. It combines the results of several studies into a single pooled estimate, giving more weight to larger and more precise studies. A meta-analysis without a proper systematic search is a red flag: pooling a hand-picked set of trials simply produces a precise-looking number for a biased sample.
Reporting standards help. The PRISMA 2020 statement sets out what a well-reported systematic review should include, from the flow diagram showing how many records were screened and excluded to the methods used to assess risk of bias. If a paper lacks these elements, treat its conclusions cautiously.
Reading a forest plot
The forest plot is the signature graphic of a meta-analysis. Each horizontal line represents one study: the square marks its point estimate, the size of the square reflects its weight, and the line shows its confidence interval. At the bottom, a diamond represents the pooled result, with its width showing the pooled confidence interval.
The vertical line marks "no effect". If a study's confidence interval crosses that line, its result on its own is compatible with no difference between groups. If the pooled diamond does not touch the line, the combined estimate is statistically significant at the chosen level, usually 5%.
Statistical significance is not the same as practical importance. A meta-analysis of creatine might show a clear but small improvement in a one-repetition maximum. Whether a few kilograms over several weeks matters depends on your sport and level. Always look at the size of the effect and its units, not just the asterisk.
Pay attention to the effect measure. Standardised mean differences (such as Hedges' g) allow different outcome scales to be combined, but they are abstract. Conventional thresholds of 0.2, 0.5 and 0.8 for small, medium and large effects are rough guides, not laws, and should be interpreted in context.
Heterogeneity and why it matters
Heterogeneity describes how much the results of included studies differ beyond what chance alone would explain. In sports science, differences are common: trials use different doses, populations (untrained students versus elite rowers), exercise tests and durations. Combining very different studies can produce an average that represents nobody.
The I² statistic, introduced by Higgins and colleagues, estimates the percentage of variability due to genuine differences between studies rather than chance. Values around 25%, 50% and 75% are often described as low, moderate and high, but again these are conventions. A high I² is a signal to investigate, not necessarily to dismiss the analysis.
Good reviews explore heterogeneity through subgroup analyses or meta-regression, for example asking whether caffeine works differently in trained and untrained people. These analyses are observational in nature and are easily over-interpreted, particularly when many subgroups are tested and only one is highlighted.
Also check the model used. Random-effects models assume the true effect varies between studies and usually give wider confidence intervals, which is often more honest for diverse sports nutrition trials. Fixed-effect models assume one true effect and can overstate precision when that assumption is wrong.
Bias, quality and certainty of evidence
A meta-analysis is only as good as the studies it contains. If most trials are small, unblinded, industry-funded or poorly randomised, pooling them does not cancel out the flaws. Cochrane reviews assess each study's risk of bias using structured tools covering randomisation, blinding, missing data and selective reporting.
Publication bias is a particular issue in the supplement world. Small trials with positive results are more likely to be published than small null trials. Funnel plots and statistical tests such as the one described by Egger and colleagues can hint at this, though they are unreliable with fewer than about ten studies.
| Question | Reassuring sign | Warning sign |
|---|---|---|
| Was there a protocol? | Registered in advance (e.g. PROSPERO) | No protocol mentioned |
| How broad was the search? | Several databases, grey literature | One database, English only |
| Was bias assessed? | Formal risk-of-bias tool | No quality appraisal |
| How consistent are results? | Low heterogeneity or explained | High, unexplored heterogeneity |
| Who funded it? | Independent or declared, managed conflicts | Undeclared commercial links |
The GRADE framework, widely used by guideline developers including NICE, rates the overall certainty of evidence as high, moderate, low or very low. It considers risk of bias, inconsistency, indirectness, imprecision and publication bias. A meta-analysis reporting a positive result with "low certainty" evidence is telling you that future research could easily change the conclusion.
Critical appraisal tools such as AMSTAR 2 allow readers to judge the methodological quality of the review itself. As Ioannidis warned in 2016, the volume of redundant and conflicted meta-analyses has grown enormously, so appraisal is not optional.
A checklist for sports nutrition claims
When you meet a claim backed by "a meta-analysis", run through a few quick questions before accepting it:
- Does the population resemble you in age, sex, training status and sport?
- Was the outcome a real performance measure or a surrogate marker in the blood?
- How large was the effect, and is it meaningful in your context?
- How many studies and participants were included, and how good were they?
- Did the authors rate certainty, and do they acknowledge limitations?
- Are the doses and forms studied the same as those in the product you are considering?
It also helps to read the abstract last rather than first. Abstracts are written to be concise and often emphasise positive findings. The results section, forest plots and limitations paragraph give a much fuller picture.
Finally, look for consistency across independent reviews. Caffeine, creatine and nitrate, for instance, have been examined repeatedly by different groups with broadly similar conclusions. A single meta-analysis with a surprising result deserves more caution than a body of convergent syntheses.
Conclusions of the editorial team
Systematic reviews and meta-analyses sit near the top of the evidence hierarchy, but only when they are carefully conducted and transparently reported. The label alone guarantees nothing.
Learning to read a forest plot, check heterogeneity and look for a certainty rating takes a little practice, yet it is one of the most useful skills for anyone navigating the supplement market.
Above all, ask whether the evidence applies to you and whether the effect is large enough to matter. That simple habit filters out a great deal of marketing noise.
Related topics worth exploring on our site include the hierarchy of evidence in sports science, how to read a single randomised controlled trial, and how to spot misleading claims on supplement labels.
References
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.
- Higgins JPT, Thomas J, Chandler J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. Chichester: Wiley; 2019.
- Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–560.
- Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–634.
- Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–926.
- Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
- Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.
