Skip to main content

Systematic Reviews

What is Critical Appraisal?

Critical Appraisal

Critical appraisal simply put is the process of systematically looking at research papers to assess three important things: trustworthiness, value and relevance. When critically appraising a research paper the first step is to examine the study for any bias.

Bias can occur in the design or methodology of the study and this can distort the study's findings so that they do not accurately reflect the truth. It should be noted that no study is totally free from bias and for this reason  it is necessary to systematically check that the researchers have done all they can to minimise bias.

A study which is sufficiently free from bias is said to have internal validity. A study will be said to have external validity when it can be generalised to the clinical (or wider population) context.

Critical appraisal checklists provide a framework for interpreting and determining the reliability of the evidence. Checklists are designed to help you answer the questions - is the study unbiased, are the findings reliable, and are the findings valid?

Critical Appraisal Tools

AMSTAR 2

An updated version of AMSTAR that appraises systematic reviews, including ones based on non-randomised studies of healthcare interventions. Includes additional criteria such as inclusion of PICO, risk of bias in the evidence synthesis stage, causes and significance of heterogeneity, and justification of chosen study design. 'Yes' answers to questions denote positive results.

Joanna Briggs Institute (JBI)

JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers.

Critical Appraisal Skills Programme (CASP)

This set of eight critical appraisal tools are designed to be used when reading research, these include tools for Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.

These are free to download and can be used by anyone under the Creative Commons License.

Johns Hopkins Research Evidence Appraisal Tool

Contains questions used to evaluate an article's study design and level of evidence. This tool contains three questions that allow the review to determine a study's methodology. Uses a 16 item checklist for research studies and a 12 item checklist for systematic reviews and meta-analyses.

Types of Bias

Bias may result from systematic errors in the research methodology. This table from the Cochrane Handbook summarises the different types of bias. 

 

A common classification scheme for bias
Type of bias   Description   Collaboration’s ‘Risk of bias’ tool
Selection bias Systematic differences between baseline characteristics of the groups that are compared
  • Sequence generation
  • Allocation concealment
Performance bias   Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest
  • Blinding of participants, personnel and outcome assessors 
  • Other potential threats to validity 
Attrition bias Systematic differences between groups in withdrawals from a study. Incomplete outcome data
  • Blinding of participants, personnel and outcome assessors
Detection bias Systematic differences between groups in how outcomes are determined
  • Blinding of participants, personnel and outcome assessors 
  • Other potential threats to validity
Reporting bias Systematic differences between reported and unreported findings 
  • Selective outcome reporting (see also Chapter 10)

 

Books on Critical Appraisal

Tools for Assessing 'Risk of Bias'

The Cochrane risk of bias tool is now recommended for use within all Cochrane reviews, and is widely used by non-Cochrane reviews of randomized controlled trials.