Meta-analysis, systematic review and umbrella review: what is the difference?

A systematic review systematically identifies, evaluates and synthesises studies. A meta-analysis is a statistical method used to combine their results. An umbrella review, also known as an overview of reviews, synthesises existing systematic reviews. These are therefore neither interchangeable terms nor successive levels of publication quality.

In the remainder of this article, we explain how these approaches differ, when study results can be combined and which mistakes should be avoided when planning your own research publication.

What is a systematic review and how is it conducted?

Suppose we want to determine whether a particular dietary intervention reduces the symptoms of a disease. We could identify several well-known papers and describe their conclusions. Alternatively, we could define the research question, eligibility criteria and literature-search strategy in advance, and then apply these rules consistently.

The latter approach forms the basis of a systematic review. Its purpose is to identify and evaluate the available studies that meet predefined criteria and to synthesise their findings. In a conventional systematic review of primary studies, we analyse publications in which researchers collected original data—for example, by conducting clinical or observational studies.

A review is systematic because its methodology is transparent and planned, not because its reference list is long. In practice, the process includes:

• a research question and protocol — defining the scope and methods before the review begins;
• searching and study selection — documenting the search for studies and applying predefined eligibility criteria;
• data extraction and risk-of-bias assessment — establishing what the studies found and which limitations may have affected their results;
• synthesis and interpretation — bringing the findings together and drawing conclusions that acknowledge uncertainty.

This process reduces the risk of selecting publications simply because they support an expected conclusion [1].

PRISMA 2020 is one of the guidelines used to report systematic reviews transparently. PRISMA is a reporting guideline; the mere presence of a study-selection flow diagram does not guarantee that a review was conducted correctly. The diagram documents the flow of records through the review, but it cannot replace a justification of the methodological decisions [2].

What is a meta-analysis and when can study results be combined?

A meta-analysis is a statistical method used to combine results from multiple studies. It may form part of a systematic review when the available data allow a pooled effect estimate to be calculated. Depending on the research question, this may be, for example, a mean difference or a risk ratio, together with its confidence interval. In commonly used methods, studies are assigned weights related to the precision of their estimates and the statistical model applied [3].

In our example, knowing that all the publications concern diet is not enough. Before combining their findings, we need to compare:

• the population — for example, the participants’ age and diagnosis;
• the intervention and control group — what was administered and what it was compared with;
• the outcome — which symptom or result was assessed and how it was measured;
• the follow-up period — whether the study examined a short-term effect or one maintained over a longer period.

For example, a reduction in pain severity after four weeks and an improvement in quality of life after one year may both be important outcomes of the same treatment. However, they answer different questions and should not be treated as interchangeable measurements of a single effect.

The fact that a calculation can be performed does not mean that the pooled result will be clinically meaningful. A random-effects model incorporates variation in effects between studies, but it cannot solve the problem of studies that are fundamentally incomparable [3].

Is a systematic review without a meta-analysis possible?

Yes. After searching and evaluating the literature, researchers may conclude that statistically combining the results would not be justified. Studies may differ too greatly in their methods, interventions or reporting. In other cases, the data required for the calculations may be unavailable.

A valuable synthesis can still be conducted. Studies can be organised by comparison and outcome; the magnitude and direction of effects can be presented; uncertainty can be described; and important limitations can be identified. The reasons for not conducting a meta-analysis should be clearly reported [4].

Returning to the dietary example, some studies may assess pain, others bloating and still others quality of life. A systematic review can show which outcomes are supported by evidence and where important gaps remain. A meta-analysis may be appropriate for some outcomes, whereas others may require synthesis without statistical pooling.

A meta-analysis should not be replaced by counting how many publications reported “statistically significant” and “non-significant” findings. Such vote counting ignores both the magnitude of the effect and the precision of the estimates. Studies with similar results may fall on opposite sides of the p = 0.05 threshold [4].

What is an umbrella review?

Over time, several systematic reviews may be published on a similar topic. The question then becomes: what can be concluded from this body of reviews, and to what extent are their findings consistent?

This is the purpose of an umbrella review, also known as an overview of reviews. It systematically identifies, evaluates and compares existing systematic reviews, with or without meta-analyses. The assessment may cover their scope, findings, methodological quality and currency. An umbrella review should not be limited to repeating the original authors’ conclusions [5].

In our hypothetical example, one review may concern adults, another children and a third only short-term symptom improvement. Similar titles do not necessarily mean that the reviews address identical questions. Conversely, conflicting conclusions may require closer examination of the data that were actually included.

A particularly important issue is the overlap of primary studies. Suppose three meta-analyses include eight, ten and twelve studies, respectively, but eight studies appear in all three. This does not give us thirty independent studies. Instead, part of the same evidence has been considered repeatedly.

An umbrella review is therefore not simply an average of the results of several meta-analyses. The dependence created by overlapping primary studies must be identified and taken into account [5].

Meta-analysis vs systematic review vs umbrella review

The table below summarises the key differences. The dietary example is provided solely for illustration and does not evaluate the effectiveness of any particular treatment.

Meta-analysis is a method of statistical analysis, whereas systematic reviews and umbrella reviews are designs used to organise and conduct evidence synthesis. These concepts may therefore be combined within a single research project [1,3,5].

Common mistakes when distinguishing meta-analyses, systematic reviews and umbrella reviews

Confusion often begins with terminology, but it can subsequently affect both the selection of publications and the way data are analysed. Six misunderstandings are particularly common:

1. Using “meta-analysis” and “systematic review” interchangeably. A systematic review describes the entire evidence-synthesis process, whereas meta-analysis is a statistical method that may be used within that process. The concepts are related, but they are not synonymous.
2. Treating a systematic review without a meta-analysis as a narrative review. What makes a review systematic is its approach to searching, study selection, appraisal and synthesis. The absence of a pooled numerical estimate does not, by itself, change the type of review.
3. Calling every paper with a forest plot a systematic review. A forest plot presents results. It does not demonstrate that the literature was searched and selected systematically.
4. Assuming that an umbrella review must be a “meta-analysis of meta-analyses”. An umbrella review may organise and critically appraise findings without calculating another pooled estimate.
5. Assuming that merely citing meta-analyses makes a paper an umbrella review. In a conventional systematic review, meta-analyses may provide background information or help identify primary studies. In an umbrella review, systematic reviews are themselves the units of searching, inclusion and appraisal.
6. Treating the three labels as a hierarchy of quality. The name of a publication does not establish the credibility of its conclusions. The methods and underlying evidence must still be evaluated.

These distinctions reflect the different roles of the systematic-review process, statistical synthesis and reviews of reviews [1,3,5].

Does an umbrella review provide stronger evidence than a meta-analysis?

The quality with which a review was conducted should be distinguished from the certainty of the evidence for a particular outcome.

A systematic review may be conducted rigorously and still leave considerable uncertainty about the effectiveness of an intervention. If the available studies are affected by bias or their results are inconsistent or imprecise, a reliable synthesis should make this uncertainty clear. It also matters whether the evidence applies directly to the patients of interest and whether some results may be unavailable because of selective publication. These domains are considered when the GRADE approach is used to assess the certainty of evidence for individual outcomes [6].

AMSTAR 2, in turn, is used to critically appraise systematic reviews of healthcare interventions. It helps identify methodological weaknesses; it is not intended to generate a simple numerical score that serves as an overall “quality rating” for a publication [7].

Suppose a dietary review was conducted correctly, but the evidence for its effect on pain comes only from small studies with wide confidence intervals. The authors may then draw a very cautious conclusion. This does not necessarily indicate a weakness in the review itself; it may accurately reflect the limitations of the underlying evidence.

How should you choose an approach for your own publication?

Start with the question that the publication is intended to answer. Only then choose the study design and analytical methods.

The following recommendations are practical implications of the distinctions discussed above.

If the results of individual clinical studies are of interest, a systematic review of primary studies will be the appropriate starting point. Meta-analysis can be planned conditionally for comparisons and outcomes for which pooling proves justifiable. The suitability of studies for synthesis should be assessed separately for each outcome.

If several reviews already exist, consider whether a critical synthesis of those reviews is needed. In some cases, updating an earlier systematic review with newly published studies may be more useful. The mere existence of several meta-analyses does not, by itself, justify conducting an umbrella review.

It is also worth returning to the situation described at the beginning: a folder contains both primary clinical studies and meta-analyses. They may serve different purposes in the project, but they should not automatically be entered into the same table as independent observations. The first step is to determine what the units of inclusion and analysis will be and whether the same underlying data might otherwise be counted more than once.

From a project-planning perspective, methodological consultation is most useful before the eligibility criteria have been finalised and data extraction has begun. At this stage, it is still possible to determine whether the question, available literature and planned synthesis are genuinely aligned.

References:

1. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358.
2. Lasserson TJ, Thomas J, Higgins JPT. Starting a review. W: Cochrane Handbook for Systematic Reviews of Interventions, chapter 1.
3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372.
4. Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA. Analysing data and undertaking meta-analyses. W: Cochrane Handbook for Systematic Reviews of Interventions, chapter 10.
5. McKenzie JE, Brennan SE. Synthesizing and presenting findings using other methods. W: Cochrane Handbook for Systematic Reviews of Interventions, chapter 12.
6. Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Overviews of Reviews. W: Cochrane Handbook for Systematic Reviews of Interventions, chapter 5.
7. Schünemann HJ, Higgins JPT, Vist GE, et al. Completing ‘Summary of findings’ tables and grading the certainty of the evidence. W: Cochrane Handbook for Systematic Reviews of Interventions, chapter 14.