
How to Appraise a Research Paper: CASP Checklist by Design
By Wasim Ghoor 15 min read Updated
To appraise a research paper, you judge whether its methods make its findings trustworthy, using a checklist matched to the study's design: CASP or JBI for most designs, RoB 2 for randomised trials, ROBINS-I for non-randomised studies of interventions and AMSTAR 2 for systematic reviews. For each study you record a judgement with a reason, not a score, and that judgement sets how much weight the study carries in your literature review.
Critical appraisal is the structured assessment of a study's design, conduct and reporting to decide how far its results can be trusted and how well they apply to your question. Critical appraisal is asked only of studies that have already passed screening for relevance.
| Study design in front of you | Checklist to use | What you record |
|---|---|---|
| Randomised controlled trial | CASP RCT checklist, or RoB 2 for a formal systematic review | Judgement per question, or RoB 2: low risk, some concerns, high risk |
| Cohort or case control study | CASP cohort or case control checklist, or JBI | Judgement with reasons; JBI adds include, exclude or seek further info |
| Non-randomised study of an intervention | ROBINS-I (formal reviews) | Low, moderate, serious or critical risk of bias |
| Qualitative study | CASP qualitative checklist, or JBI qualitative checklist | Judgement with reasons; JBI: yes, no, unclear, not applicable per item |
| Systematic review you cite as evidence | AMSTAR 2, CASP or JBI systematic review checklist | AMSTAR 2: high, moderate, low or critically low confidence |
| Text, opinion or policy document | JBI textual evidence checklists | Judgement with reasons |
What is critical appraisal of a research paper?
Critical appraisal of a research paper is a trust decision: does the study's design, conduct and reporting give you reason to rely on its findings for your question? Critical appraisal comes after screening, which only decided that the paper is about your question, and before synthesis, where you decide what the body of evidence supports.
Think of each included study as a witness rather than a fact. A court does not simply write down what a witness said. It asks how the witness was placed to know, what might have shaped the account, and how sure the witness was about the parts that matter. Critical appraisal asks the same three things of a study: how was the study placed to answer the question (design), what might have distorted the result (bias), and how precise is the result (the confidence interval)?
The Cochrane Handbook draws the line between bias and imprecision clearly: bias is systematic error, imprecision is random error from sampling, and “bias should not be confused with imprecision” (Cochrane Handbook, chapter 7, section 7.1.1). A small study can be at low risk of bias and still imprecise; a large study can be precise and still badly biased. Critical appraisal judges both, and a judgement about the study is never a verdict on the authors.
How do you appraise a research paper, step by step?
You appraise a research paper in six steps: identify the study design, choose the checklist for that design, answer each checklist question with a reason, form an overall judgement, write down what the study can and cannot support, and record all of it in your matrix. The steps take the same shape for every design; only the checklist questions change.
| Step | What you do | What you have at the end |
|---|---|---|
| 1. Identify the design | Read the methods section, not the abstract's label | A design name (cohort, RCT, qualitative, systematic review) |
| 2. Choose the checklist | Match the tool to the design (table above) | One named tool, used for every study of that design |
| 3. Answer each question | Give yes, no or can't tell, plus the reason from the paper | A completed checklist with page references |
| 4. Form an overall judgement | Weigh the questions that matter most for that design | Strong, moderate or weak (or low, moderate, high concern) |
| 5. State what it can support | Say which claims the study can carry | One sentence you can reuse in your chapter |
| 6. Record it | Add design, tool, judgement, limitation and support to your matrix | A row you never have to reopen the PDF for |
Step 1 matters more than students expect. Authors sometimes call a study “experimental” when participants chose their own group, which makes the study observational and changes the checklist. Step 3 is where a page reference saves you later: a judgement that says “no adjustment for working hours (p. 7)” can be checked by your supervisor, while “felt weak” cannot.
Which appraisal checklist fits which study design?
The appraisal checklist has to match the study design, because the threats to a trial and the threats to an interview study are different: randomisation and blinding are meaningless questions for a qualitative study, and researcher reflexivity is a meaningless question for a trial. For most master's and doctoral literature reviews, CASP is a sensible default; JBI covers more designs; RoB 2, ROBINS-I and AMSTAR 2 are for formal systematic reviews.
| Tool | Published by | Use it on | Structure and judgement | Source |
|---|---|---|---|---|
| CASP checklists | Critical Appraisal Skills Programme (UK) | Eight designs: systematic reviews, randomised controlled trials, cohort, case control, economic evaluations, diagnostic studies, qualitative studies, clinical prediction rules | Questions in three sections (are the results valid, what are the results, will the results help locally); the 2024 cohort checklist has 12 questions | casp-uk.net |
| JBI critical appraisal tools | JBI | Fifteen checklists, including RCTs, quasi-experimental, cohort, case control, analytical cross-sectional, prevalence, case series, case reports, qualitative research, systematic reviews and three textual evidence types | Yes, no, unclear or not applicable per item; overall include, exclude or seek further info (2020 qualitative checklist: 10 questions) | jbi.global |
| RoB 2 | Cochrane (Sterne et al., 2019) | Randomised trials | Five bias domains; each domain and the overall result judged low risk of bias, some concerns or high risk of bias | riskofbias.info; BMJ 2019;366:l4898 |
| ROBINS-I | Sterne et al., 2016 | Non-randomised studies of interventions | Seven bias domains judged against a hypothetical target trial: low, moderate, serious or critical risk | BMJ 2016;355:i4919 |
| AMSTAR 2 | Shea et al., 2017 | Systematic reviews of healthcare interventions | 16 items, 7 of them critical; overall confidence high, moderate, low or critically low | BMJ 2017;358:j4008 |
| GRADE | GRADE Working Group | A body of evidence on one outcome, after individual appraisal | Certainty high, moderate, low or very low; not a per-paper checklist | Cochrane Handbook, chapter 14 |
The five RoB 2 domains are bias arising from the randomisation process, bias due to deviations from intended interventions, bias due to missing outcome data, bias in measurement of the outcome, and bias in selection of the reported result (Cochrane Handbook, chapter 8). Reach for RoB 2 or ROBINS-I only when your review has to report risk of bias domain by domain, which is normal in a systematic review and unusual in a narrative dissertation chapter. Whichever tool you choose, name it in your methodology and use the same tool for every study of that design.
How is appraisal different from screening and peer review?
Screening decides whether a study is relevant to your question; critical appraisal decides whether a relevant study can be trusted; peer review is a journal's decision to publish. A paper can pass all three of screening, peer review and publication and still carry only weak support for the claim you want to make.
| Screening | Critical appraisal | Peer review | |
|---|---|---|---|
| Question asked | Does the study match my eligibility criteria? | Do the methods make the findings trustworthy for my question? | Is the manuscript fit for this journal? |
| Who does it | You, against your inclusion and exclusion criteria | You, with a design-specific checklist | A few reviewers chosen by the editor, usually without the raw data |
| Output | Include or exclude | A judgement with reasons and a statement of what the study can support | Accept, revise or reject |
| When | Before appraisal | After screening, before synthesis | Before publication |
Screening is set by your inclusion and exclusion criteria, and the answer is yes or no. Peer review is a useful filter, but a journal accepting a paper does not tell you how much weight one finding deserves inside your argument; published papers are corrected and retracted, which is why it is worth knowing how to check whether a paper has been retracted. To confirm that a source went through peer review in the first place, the Peer Review Checker in this guide's free tools checks any paper by DOI, link or title.
What does a CASP appraisal look like on one paper?
A CASP appraisal of one paper is the checklist's questions answered in order, each with a reason from the paper, ending in one sentence about what the study can support. The worked case below uses the CASP cohort study checklist (2024), whose 12 questions sit in three sections: are the results valid, what are the results, and will the results help locally.
Format illustration: the study below is invented to show the method. It has no real authors and no real figures.
Suppose a literature review asks whether structured writing support improves thesis completion among part-time postgraduate students, and one included paper is a single-university cohort study comparing students who attended a writing programme with students who did not.
- Did the study address a clearly focused issue? Yes: population, exposure and outcome are stated. Note the exact outcome, because “completed” and “submitted within the time limit” are different claims.
- Was the cohort recruited in an acceptable way? Students chose whether to attend. Self-selection is the most important line in this appraisal, because motivated students may both sign up and finish regardless of the programme.
- Was the exposure accurately measured? Attendance was self-reported, which is weaker than a register.
- Was the outcome accurately measured? Completion came from university records, which is objective.
- Were confounding factors identified and accounted for? The analysis adjusted for age and faculty but not for prior research experience or employment hours, two plausible drivers of completion for part-time students.
- Was follow-up complete and long enough? Record how many students were lost and ask whether the students who left were the ones struggling.
- What are the results? Record the effect size, not only whether the result was significant.
- How precise are the results? Record the confidence interval. A wide interval that crosses no effect says something a p value hides.
- Do you believe the results? Partly: the direction is plausible, but self-selection and the missing adjustments suggest the size of the effect is overstated.
- Can the results be applied to the local population? One institution and one programme design limit how far the finding travels.
- Do the results fit with other available evidence? Compare the direction with the other included studies.
- What are the implications for practice? State what the study could justify, and what it could not.
The output of the CASP appraisal is a usable sentence: “This study offers moderate support for the direction of the effect and weak support for its size, because participants chose their own group and the analysis did not adjust for prior research experience or working hours.”
Should you give studies a quality score?
No: record a judgement with reasons rather than adding up “yes” answers into a quality score. Adding answers implies every question carries equal weight, and the Cochrane Handbook notes that the choice of quality scale “could significantly influence the interpretation of the meta-analysis results” and that a combined summary score “is difficult to interpret” (Cochrane Handbook, chapter 7, section 7.1.2).
The AMSTAR 2 authors say the same of their own tool: “AMSTAR 2 is not intended to generate an overall score” (Shea et al., 2017). Unaddressed confounding in an observational study matters far more than a vague description of the setting, and a total of yes answers hides that difference. If your department or supervisor requires a score, report the score alongside the reasons, never instead of them.
How do you record appraisal in a literature review matrix?
You record critical appraisal as four extra columns in the literature review matrix or extraction table you already fill in while reading: design and tool, overall judgement, main limitation, and what the study can support. Recording appraisal during reading means you never reopen the PDFs for a second pass.
| Column | What goes in it | Example (format illustration) |
|---|---|---|
| Design and tool | The design and the checklist used | Cohort study; CASP cohort checklist (2024) |
| Overall judgement | Three levels is enough | Moderate |
| Main limitation | One specific problem, with a page reference | Self-selection into groups; no adjustment for working hours |
| What it can support | The claims the study can carry | Direction of effect only, not its size |
If you have not built the matrix yet, the system in how to organize research sources sets one up, and the Literature Review Matrix in this guide's free tools builds a comparison table from your own PDFs. The same columns slot into a data extraction table if your review extracts outcomes study by study. When a second person appraises with you, appraise two or three papers independently and compare first: disagreement usually means the two appraisers read a checklist question differently, which is better found on paper three than on paper thirty.
Do you need to appraise every study in a literature review?
A systematic review appraises every included study; a scoping review usually does not; a narrative literature review in a dissertation or thesis sits in between. The PRISMA extension for scoping reviews lists critical appraisal of individual sources as “Item 12 (Optional)” (Tricco et al., PRISMA-ScR, 2018), because a scoping review maps what research exists rather than what the research shows.
The review type therefore sets how formal the appraisal must be, which is covered in scoping vs systematic vs narrative review. In a narrative chapter, appraising the studies your argument leans on hardest strengthens the chapter even where no examiner requires it. Ask your supervisor which applies before you appraise thirty papers: the answer usually takes one line. Appraisal sits after the search strategy and the PRISMA flow diagram, and before synthesis.
How does appraisal change what you write?
Critical appraisal changes a literature review from a list of findings into an argument about what the evidence supports, because each study is cited with the weight its methods earn. The change shows in individual sentences.
Format illustration with placeholder citations:
Before appraisal: (Author, Year) found that writing support improved completion. (Author, Year) reported a similar effect. (Author, Year) also found benefits for part-time students.
After appraisal: Three studies report a positive association between structured writing support and completion, but the strength of that evidence varies. The only study to adjust for prior research experience found the smallest effect, while the two larger effects come from cohorts in which students selected the programme themselves (Author, Year; Author, Year; Author, Year). The direction is consistent; the size remains uncertain.
The second paragraph is the summary-to-synthesis move described in how to synthesize sources in a literature review, and appraisal supplies its raw material. Appraisal also earns three sentences in your methodology section, for example: “Included studies were appraised with the CASP checklist for each design by one reviewer, with judgements recorded during extraction. No study was excluded on appraisal grounds; quality was weighted in the synthesis and is reported per study in Appendix B.” Decide openly whether weak studies are excluded or kept and weighted; both are defensible, and deciding silently, study by study, depending on whether a result agrees with you is not.
What mistakes do students make in critical appraisal?
The five common critical appraisal mistakes are treating peer-reviewed papers as automatically sound, appraising without a tool, using the wrong checklist for the design, over-weighting sample size, and appraising without the judgement ever reaching the written chapter.
| Mistake | Why the mistake matters | Fix |
|---|---|---|
| Treating peer reviewed as sound | Every study is cited as equally reliable, so nothing has been appraised | Appraise each included study with a named checklist |
| Appraising by feel | “This one felt weak” is not reproducible and drifts with your opinion | Ask the same checklist questions of studies you agree and disagree with |
| Wrong checklist for the design | Marking a qualitative study down for no blinding is a category error | Identify the design from the methods section first |
| Over-weighting sample size | A large, badly recruited sample is a large biased study; size makes systematic error look more certain | Judge bias and precision separately (Cochrane Handbook, section 7.1.1) |
| Appraisal stuck in an appendix | A table nobody cites in the prose is wasted work | Let each judgement shape the sentence that cites the study |
How do you compare appraised papers with an AI assistant for writing backed by sources?
An AI assistant for writing backed by sources helps after you have appraised your papers, by lining up the details you need to weigh them, each linked to its passage. Litrevu is an AI assistant for writing backed by sources. It reads every paper, report and PDF you add, answers your questions with a citation to the exact passage behind each claim, compares sources side by side, and checks your own draft against them.
On appraisal work, the Litrevu workflow on your own papers runs like this:
- Add the papers you included. Upload the PDFs, or paste a PDF link, DOI or arXiv link.
- Ask the questions your checklist asks. “How were participants recruited in each study?” or “Which studies adjusted for confounders, and which confounders?” Litrevu answers with citations from your papers only, and says so when your papers do not cover a question.
- Open each citation. Every citation opens to the highlighted passage on the PDF page, so you can check the recruitment method or the adjustment list yourself before you write a judgement.
- Compare the studies in a table. Line up design, sample, findings and limitations side by side, with a citation in each cell.
- Check your draft paragraph. Paste the paragraph where you weigh the studies; each claim comes back Supported, Partly supported, Contradicted or Not in your papers.
The judgement stays yours: Litrevu finds and lines up the evidence in your papers, and you decide how much each study can bear. “Not in your papers” means your library does not cover a claim, not that the claim is wrong or that no study exists. Litrevu can also draft cited text into a document; many universities do not allow AI-written text in submitted work, even after you edit it, so check your university's policy and declare how you used AI.
Frequently asked questions
Is critical appraisal the same as peer review?
No. Peer review is a journal's decision, made by a few reviewers before publication, about whether a manuscript is fit to publish. Critical appraisal is your own structured judgement, made with a design-specific checklist, about how far a published study can support a claim in your review.
Do I need to appraise every study in a literature review?
A systematic review appraises every included study, while a scoping review usually does not: PRISMA-ScR lists critical appraisal as optional (Item 12). A narrative literature review in a dissertation or thesis falls in between, so ask your supervisor, and at minimum appraise the studies your argument relies on most.
Are CASP checklists free?
Yes. The Critical Appraisal Skills Programme describes its checklists as free to download, and the 2024 cohort study checklist carries a Creative Commons Attribution Non-Commercial Share Alike licence. CASP offers eight checklists, one per study design, as a printable form or a version to fill in electronically.
Should I give studies a quality score?
Usually not. The Cochrane Handbook explains that summary scores from quality scales are difficult to interpret, and the AMSTAR 2 authors state that AMSTAR 2 is not intended to generate an overall score. Record a judgement with reasons for each study; if your department requires a score, report the reasons alongside it.
What is the difference between CASP and JBI checklists?
CASP offers eight checklists with questions grouped into three sections: are the results valid, what are the results, and will the results help locally. JBI offers fifteen checklists, including prevalence studies, case series and textual evidence, with each item answered yes, no, unclear or not applicable and an overall decision to include, exclude or seek further information.
Which checklist should I use for a qualitative study?
Use the CASP qualitative studies checklist or the JBI checklist for qualitative research (the 2020 JBI version has 10 questions). Do not use a trial or cohort checklist on a qualitative study, because questions about randomisation and blinding do not apply to interview or focus group research.
How do I write up critical appraisal in my methodology?
Name the checklist used for each design, say who appraised the studies and when, and state whether any study was excluded on appraisal grounds. Three sentences are usually enough, with the per-study judgements reported in a table or appendix and reflected in how the chapter cites each study.
Where should you start with critical appraisal?
Start critical appraisal with one included paper: download the CASP checklist that matches its design, work through the questions with a reason for each, and write one sentence about what the study can and cannot support. Judging the evidence is your work and nobody can do it for you. You read the methods, you saw the confounder nobody adjusted for, and you are the one who has to defend the weight you gave each study. What helps is having the details in front of you: each study’s design, sample and limitations, with the passage behind each one.
That is where Litrevu helps. Add the papers you have already chosen and appraised, ask it to compare their aims, designs, samples, findings and limitations, and you get a table in which every cell cites the passage it came from. Open any cell to check it against the page, then put your own weighting on each study. Litrevu is free to start, no credit card required.
Try Litrevu free