Translated from the Portuguese original. Read the original (Português)
Systematic review and meta-analysis are not the same thing, although they almost always appear together. The systematic review is a method of searching and appraisal: it defines the question before starting, systematically and documentedly searches for the studies that exist on it — including the rarely cited ones and those that may never have been published —, judges the quality of each one and synthesizes the set so that another person can repeat the path and arrive at the same place. The meta-analysis is the statistical, and optional, step that combines the numbers from those studies into a single summary estimate, generally more precise than that of any single study. Done well, they are the most reliable thing we have for answering "what do we know so far?" Done poorly, they merely lend an air of rigor to a handful of fragile studies.
A systematic review is a secondary study that answers an explicitly formulated question by employing systematic and pre-specified methods to identify, select, critically appraise and synthesize the relevant primary studies, with the aim of minimizing bias and producing reproducible conclusions. Its hallmarks are the protocol pre-specified before execution — preferably registered in a public database —, the search documented across multiple databases, selection and data extraction done in duplicate by independent assessors, and a formal assessment of the risk of bias of each included study. The meta-analysis is the statistical procedure — not mandatory — that combines the effect estimates of the included studies into a weighted summary estimate, giving more weight to the more precise studies, and that quantifies how much the results diverge from one another (heterogeneity). The term meta-analysis was coined by education researcher Gene V. Glass in 1976; the practice of combining results from different studies is older and is usually traced back to Karl Pearson, who in 1904 pooled data on typhoid fever inoculation in British troops.
Two different things that tend to go together
Imagine you want to know whether a given intervention helps children develop language. You find one study saying yes, another saying it makes no difference, and a third with an intermediate result. Which one counts? The question is poorly framed: a single study is rarely the answer. The answer lies in the body of evidence as a whole — and getting there requires method, because cherry-picking the studies that confirm our impression is far too easy.
That is what the systematic review does. It treats the literature itself as the object of research: instead of participants, the "subjects" are the studies; instead of a treatment protocol, there is a search-and-selection protocol. And, as in any serious research, that protocol is decided before the data appear, precisely so that the result is not shaped by the path taken to get there.
The meta-analysis comes afterward, and only if it fits. It is the calculation: it takes the effect size measured in each study and combines them into a weighted average, in which larger and more precise studies carry more weight. From this follow four possible combinations, worth telling apart:
- Systematic review with meta-analysis — the most common combination, when the studies are similar enough that combining their estimates makes sense.
- Systematic review without meta-analysis — perfectly legitimate. When the studies measure things that are too different from one another, the review team presents a structured synthesis without combining them. Refusing to calculate is, here, a sign of rigor.
- Meta-analysis without systematic review — combining numbers from studies gathered without a systematic search. A very precise average of a biased sample of studies is still biased.
- Narrative review — the expert's article that discusses the literature without an explicit search or declared criteria. It can be excellent reading and highly valuable for teaching, but it offers no guarantee against selection bias: there is no way to know what was left out.
What makes a review "systematic"
It is not the size, nor the number of studies included, nor the journal it was published in. It is the method — and it has recognizable steps. Tap the tabs to see each one:
The question comes before everything — in writing
A review begins by delimiting who (population), what (intervention), compared to what (comparator) and with what measured result (outcome) — the framework known as PICO. These elements, plus the inclusion criteria and the analysis plan, form the protocol, which must be publicly registered at the start of the review, before the results are known, on platforms such as PROSPERO (maintained by the Centre for Reviews and Dissemination at the University of York and open since 2011). Prior registration exposes course changes made after seeing the results — swapping the outcome, broadening or narrowing the criteria until the conclusion comes out the way one wants.
Searching beyond where a casual search reaches
The search is carried out across several databases (such as MEDLINE/PubMed, Embase, PsycINFO, the Cochrane Library and, in Latin America, LILACS), with the full search strategy published — the terms, the operators, the dates — so that anyone can rerun it. Reviewers often go further: tracking the references of the articles found, searching theses, conference proceedings and trial registries, and contacting authors in pursuit of results that were never published. This insistence has a specific reason: studies with a negative result are more likely to never be published, and whoever only finds what is easy to find ends up painting an overly favorable picture.
Two people, working independently
Screening of titles and abstracts, reading of full texts, and data extraction are carried out by two assessors working separately, with disagreements resolved through discussion or by a third assessor. This is not bureaucracy: judging whether a study meets the criteria involves subtle decisions, and one person alone makes mistakes and is influenced by their own expectations without noticing. The entire path — how many records were found, how many were discarded and, at the full-text stage, why — is recorded in the flow diagram that opens practically every published systematic review.
Not every included study deserves the same trust
Each included study undergoes a formal assessment of risk of bias, with tools specific to each type of design — RoB 2 for randomized trials, ROBINS-I for non-randomized studies. Only then comes the synthesis, with or without meta-analysis. And the final step is to judge the body of evidence as a whole: the GRADE system rates the certainty of the evidence on four levels — high, moderate, low or very low — downgrading it in the face of risk of bias, imprecision, inconsistency between studies, indirect evidence or suspected publication bias.
How to read a forest plot
It is the symbol-figure of almost every meta-analysis, and decoding it is simpler than it looks. Tap the cards to see what each element means:
When studies disagree with one another
Studies rarely all point to the same number. Part of that variation is just sampling chance — the expected kind. But part may reflect real differences: a different age range, a different dose, a different comparator, a different measurement instrument, a different culture. This variation beyond chance is called heterogeneity, and it is information, not a flaw to be hidden.
The most widely reported measure for describing it is I², which estimates what proportion of the variation observed between studies is due to real differences rather than chance. The Cochrane Handbook offers a rough guide — values of 0% to 40% may not be important; 30% to 60%, moderate heterogeneity; 50% to 90%, substantial; and 75% to 100%, considerable — and warns that applying these cutoffs mechanically can be misleading: the importance of inconsistency also depends on the direction and magnitude of the effects and on the strength of the evidence for heterogeneity itself.
Before combining, one must decide — in the protocol, not after peeking at the results — what assumption to adopt about what the studies are estimating:
- Fixed-effect model — assumes that all studies are estimating the same true effect and that only chance separates them. It does not handle heterogeneity: it assumes it does not exist.
- Random-effects model — admits that each study estimates a slightly different effect and calculates the average of that distribution. It produces wider intervals than the fixed-effect model on the same data — that is, more honest about the uncertainty — and is the usual assumption when the studies are expected to differ from one another.
And it is worth not expecting an automatic gain in precision from a meta-analysis: when heterogeneity is high, the combined estimate's interval may end up no narrower than that of the single best study.
When the disagreement is large, the productive step is not to choose a different model: it is to investigate where it comes from, with subgroup analyses planned in the protocol (by age, by severity, by intensity of the intervention) or through meta-regression. The question stops being "does it work?" and becomes the most useful one of all: "for whom, under what conditions, and at what intensity?"
Not everything should be added togetherIf the studies measured incompatible outcomes, used interventions that only share a name, or very distinct populations, calculating an average of them produces a meaningless number — the famous problem of adding apples to oranges. A good review explains why it did not run a meta-analysis and presents a structured synthesis instead of forcing the calculation. The absence of a diamond is not a failure: it is often the most careful decision of the whole review.
Where a meta-analysis can mislead
Sitting at the top of the hierarchy of evidence describes the potential of the design, not the quality of what was actually done. These are the points where error creeps in:
- Garbage in, garbage out. Combining twelve studies with high risk of bias does not cancel out the bias — it is inherited by the final estimate, now with an appearance of precision that makes it even more persuasive. Precision and reliability are different things.
- Publication bias. If the studies that found no effect were left in the drawer, the review only finds the optimistic part of the literature. The classic tool for sniffing this out is the funnel plot, in which studies should be distributed symmetrically: the large ones clustered at the top, the small ones scattered at the base. An asymmetry — few small studies on one side — suggests that something was not published. The regression test proposed by Egger and colleagues in 1997 assesses this asymmetry; but the very authors of that line of research warn that asymmetry can also come from real heterogeneity, from quality differences between small and large studies, or simply from chance.
- Samples counted twice. When several articles report the same participants, including all of them gives extra weight to a single data set, as if they were independent participants.
- Invisible analytic choices. Which outcome to combine, which studies to exclude, which model to use — each decision can shift the result. That is what the protocol registered in advance and sensitivity analyses are for: they redo the calculation under other assumptions and show whether the conclusion holds up.
- An excess of reviews. The number of published systematic reviews and meta-analyses has grown much faster than the scientific literature as a whole — and a considerable share is redundant or of low quality, a problem documented by John Ioannidis. The existence of a meta-analysis on a topic does not settle the discussion; one needs to look at which one.
Tools for judging a review
You do not need to be a statistician to appraise a systematic review. There are public instruments, and each one answers a different question:
- PRISMA 2020 deals with how to report. Published in the BMJ in 2021 by Matthew Page and colleagues, replacing the 2009 version, it brings a 27-item checklist, a specific checklist for the abstract, and revised flow diagrams. When an article fails to report something PRISMA asks for — the search strategy, the excluded studies, the funding sources — the omission is already a warning sign.
- AMSTAR 2 (Shea and colleagues, BMJ, 2017) assesses how the review was conducted, across 16 items, some considered critical — such as prior registration of the protocol and taking risk of bias into account when interpreting results. The ROBIS tool serves a similar purpose, estimating the risk of bias of the review itself.
- GRADE answers the final question: how much confidence does the body of evidence deserve, outcome by outcome? It rates that certainty as high, moderate, low or very low — and the same review can come out with high certainty for one outcome and very low for another. That is why conclusions from Cochrane reviews tend to come with phrasings like "low-certainty evidence suggests that…": that caution is not a lack of conviction, it is information.
It is worth remembering where much of this arsenal comes from. Cochrane, an international organization founded in 1993, takes its name from the Scottish epidemiologist Archie Cochrane (1909–1988), who in his 1972 book Effectiveness and Efficiency argued that health resources should go to what well-conducted evaluations showed to work. Organizing what is already known stopped being a weekend task and became a methodological discipline in its own right.
Two common mistakes"There's a meta-analysis, so it's proven." The label says nothing about the quality of the included studies, about how much was left out of the search, or about who funded the work. And the opposite mistake: "a meta-analysis is just lumping everything into an average." The hard part is not the calculation — it is the exhaustive search, the decision about what is combinable, the assessment of risk of bias, and the explicit judgment of the certainty of the evidence. The statistics are the most visible step and, almost always, the simplest.
Test your understanding
Quick quiz
Three questions to reinforce the concept. You'll see the explanation right away.
Further reading
- Page, M. J., McKenzie, J. E., Bossuyt, P. M. et al. "The PRISMA 2020 statement: an updated guideline for reporting systematic reviews". BMJ, 2021;372:n71.
- Higgins, J. P. T., Thomas, J., Chandler, J. et al. (eds.). Cochrane Handbook for Systematic Reviews of Interventions. Cochrane. Available at training.cochrane.org.
- Shea, B. J., Reeves, B. C., Wells, G. et al. "AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both". BMJ, 2017;358:j4008.
- Higgins, J. P. T., Thompson, S. G., Deeks, J. J., Altman, D. G. "Measuring inconsistency in meta-analyses". BMJ, 2003;327:557–560.
- Egger, M., Davey Smith, G., Schneider, M., Minder, C. "Bias in meta-analysis detected by a simple, graphical test". BMJ, 1997;315:629–634.
- Guyatt, G. H., Oxman, A. D., Vist, G. E. et al. "GRADE: an emerging consensus on rating quality of evidence and strength of recommendations". BMJ, 2008;336:924–926.
- Ioannidis, J. P. A. "The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses". The Milbank Quarterly, 2016.
- Glass, G. V. "Primary, secondary, and meta-analysis of research". Educational Researcher, 1976;5:3–8.
- Pearson, K. "Report on certain enteric fever inoculation statistics". British Medical Journal, 1904.
- Cochrane, A. L. Effectiveness and Efficiency: Random Reflections on Health Services. 1972.
- PROSPERO — International Prospective Register of Systematic Reviews. Centre for Reviews and Dissemination, University of York.
This entry is educational and does not replace assessment, diagnosis or treatment by a qualified health professional. Every person is unique — for a specific case, seek individual guidance.