# Risk of Bias Assessment in Systematic Reviews By the end of this guide you will be able to choose the correct risk of bias tool for any study design you encounter, complete a domain-by-domain assessment with a second reviewer, reach an overall judgment using published decision rules, generate publication-ready summary figures, and carry your judgments into a certainty-of-evidence rating such as GRADE. You will need the official tool spreadsheets (all are free downloads), spreadsheet software such as Excel or LibreOffice Calc, a second assessor for duplicate judging, and optionally the R statistical environment if you want to produce figures programmatically. Risk of bias assessment is the step that separates a systematic review from a simple summary. A systematic literature search may return fifty studies, but if the included studies systematically overestimate or underestimate an effect, the pooled answer inherits that distortion. Review teams therefore judge each included study against structured criteria before they synthesize anything. ## What Risk of Bias Actually Measures Bias is systematic error, a flaw in design, conduct, or analysis that pushes a study's result away from the truth in a consistent direction. It differs from imprecision, which is random error captured by [confidence intervals](/blog/data-analysis/confidence-interval-formula-examples) and sample size. A small study with flawless methods has low risk of bias but low precision. A huge study with broken randomization can be precise and wrong at the same time. In risk of bias assessment in systematic reviews, you judge internal validity, not the importance of the topic or the elegance of the paper. The question you ask is narrow and repeated for every study: if this study's protocol were followed perfectly as described, how likely is the reported effect to differ from the effect a flawless study of the same question would have produced? Formal tools exist because unstructured "quality scoring" is unreliable and invites disagreement. Structured tools break internal validity into named domains, force an explicit judgment for each domain, and require written justification. Current published reviews follow this pattern closely. A 2026 review of thromboprophylaxis timing in severe traumatic brain injury assessed risk of bias with ROBINS-I and then rated certainty of evidence with GRADE [1]. A review of intrarenal pressure control during flexible ureteroscopy matched tools to design, using RoB 2 for randomized studies, ROBINS-I for non-randomized studies, and the JBI checklist for case series [2]. The tool follows the design, never the other way around. ## Risk of Bias Versus Quality, Reporting, and Applicability Three distinctions trip up almost every beginner. **Risk of bias versus reporting quality.** A study can report its methods poorly and still have low risk of bias, and a beautifully reported study can conceal fatal flaws. Tools such as RoB 2 ask what happened, not what was written about it. **Risk of bias versus applicability.** Applicability (sometimes called external validity or concerns regarding applicability) asks whether the study population, interventions, or measures match the question your review is trying to answer. QUADAS-2 and PROBAST score risk of bias and applicability separately. A diagnostic study can be bias-free in a population that looks nothing like your patients, which is an applicability concern, not a bias concern. **Risk of bias versus certainty of evidence.** GRADE takes the risk of bias judgment as one input and combines it with imprecision, inconsistency, indirectness, and publication bias to rate how confident you can be in an estimate. Low risk of bias does not guarantee high certainty, and high risk of bias is only one way certainty can fall. Reviews consistently run these two steps in sequence, judging bias first and certainty second [3], [4], [5], [1]. ## Choose the Right Tool for the Study Design ![Decision path asking if a study is randomized, then whether it is diagnostic, a prediction model, or observational, ending at the matching tool.](/article-images/illustrations/risk-of-bias-assessment-systematic-review-tool-choice.webp) This is the single most consequential decision in the whole exercise. The table below summarizes the standard options. | Study design | Standard tool | Bias domains | Judgment scale | |--|--|--|--| | Randomized trials | Cochrane RoB 2 | 5 | Low risk, some concerns, high risk | | Non-randomized intervention studies | ROBINS-I | 7 | Low, moderate, serious, critical risk | | Diagnostic accuracy studies | QUADAS-2 or QUADAS-3 | 4, each with bias and applicability | Low or high risk | | Prediction model studies | PROBAST | 4, each with bias and applicability | Low or high risk | | Prognostic factor studies | QUIPS | 6 | Low, moderate, high risk | | Cohort and case-control studies (older reviews) | Newcastle-Ottawa Scale | 3 (selection, comparability, outcome) | Star rating up to 9 | | Case series | JBI critical appraisal checklist | Sequential yes or no items | Item-level yes or no | | Mixed or rapid reviews | MMAT (Mixed Methods Appraisal Tool) | 5 methodological categories | Item-level criteria | Published reviews illustrate how these pairings work in practice. Reviews of randomized trials of vitamin D supplementation, psychotherapy for obsessive-compulsive disorder, and non-pharmacological interventions for kinesiophobia after knee replacement all used the Cochrane RoB 2 tool [6], [7], [8]. A network meta-analysis of eosinophilic esophagitis treatments also applied RoB 2 to its randomized trials [9]. A review of cardiovascular predictors in multiple myeloma used QUIPS for its nine retrospective cohorts, one prospective cohort, and one case-control study [10]. A diagnostic accuracy review of clinical tests for lumbar foraminal stenosis used QUADAS-2 [11], while a protocol for biomarkers in periprosthetic joint infection pre-specified QUADAS-3 with an adapted QUADAS-C comparison for studies evaluating two tests in the same patients [12]. A 2026 review of lymphatic filariasis risk prediction models applied PROBAST to 26 modeling studies [13], and a machine learning review of glycemic prediction applied PROBAST with two independent assessors across 28 assessment units [14]. One caution: when a review mixes randomized and observational evidence, as in the bariatric surgery weight-regain literature, teams sometimes pair the Newcastle-Ottawa Scale with the JBI checklist rather than using ROBINS-I, which limits direct comparability of judgments across designs [3]. ## Risk of Bias Assessment in Systematic Reviews Step by Step The walkthrough below assumes a mixed review containing randomized trials and non-randomized studies, which is the most common real-world situation. Adapt the tool names if your evidence base is uniform. ### Step 1: Fix the Tool in a Registered Protocol Decide which tool applies to which design before you screen a single record, and register the decision. Most biomedical teams register in PROSPERO, and published protocols routinely state the assessment tool explicitly [12], [9]. Registration prevents the most damaging accusation in review methodology, which is that bias judgments were made after the authors saw the results. If your review will assess risk of bias per outcome rather than per study, say so in the protocol, because bias in a subjective outcome such as patient-reported pain can differ from bias in an objective outcome such as mortality within the same trial. ### Step 2: Build the Assessment Form Create a spreadsheet with one row per study and one column per domain, plus columns for the free-text justification and the overall judgment. For RoB 2 your columns are D1 through D5. ```text Study D1_randomization D2_deviations D3_missing D4_measurement D5_selection Overall Notes Trial 01 + + + x + ? ... Trial 02 + + - + + ? ... ``` Judgment codes for RoB 2 in most software are + (low), x (some concerns), and - (high). For ROBINS-I use L, M, S, and C. Lock the coding scheme down before the pilot, because inconsistent coding is the most common source of reconciliation chaos. ### Step 3: Assess Independently in Duplicate Two assessors judge every study independently, blind to each other's judgments, and a third person arbitrates unresolved conflicts. Published reviews treat duplicate assessment as standard practice rather than an upgrade. A review of hydration strategies during endurance exercise had two reviewers independently screen records, extract data, and assess risk of bias and certainty [4], and the glycemic prediction machine learning review specified two named assessors applying PROBAST independently [14]. Independent duplicate assessment is the strongest single safeguard against your own expectations influencing your judgments. Pilot the form on three to five studies first. If two trained assessors disagree on more than roughly a third of domain judgments during the pilot, the problem is usually an ambiguous form instruction rather than the tool itself, and you should revise the operational rules before scaling up. ### Step 4: Judge Each Domain With Signalling Questions RoB 2 works through signalling questions answered yes (Y), probably yes (PY), probably no (PN), no (N), or no information (NI), followed by a domain-level algorithm. Here is a worked example for Domain 1, bias arising from the randomization process, for a hypothetical trial described as "randomized" with a central telephone allocation service and a baseline imbalance in disease severity. ```text D1.1 Was the allocation sequence random? Response: Y (authors describe computer-generated sequences) D1.2 Was the allocation sequence concealed until participants were enrolled and assigned? Response: PY (central telephone service, described in methods) D1.3 Did baseline differences suggest a problem with randomization? Response: PN (imbalance explainable by chance, no adjustment made but adjustment is not mandatory) Algorithm for D1: no answers of N, PN, or NI on any question that could lead to high risk Overall domain judgment: Low risk of bias ``` Now change one fact. If the trial instead said "we randomized patients alternately on admission days," the answer to D1.1 becomes N, because alternation is predictable and a form of quasi-randomization. The algorithm then escalates the domain to high risk, and the free-text justification should quote the exact sentence from the paper that drove the judgment. Every domain judgment in your spreadsheet must be traceable to a quoted sentence or a stated absence. "The paper does not report allocation concealment" is a complete justification. "Methods seem okay" is not. ### Step 5: Reach the Overall Judgment ![Flowchart: any high-risk domain makes the study high risk, some concerns without high risk gives some concerns, all-low domains give low risk.](/article-images/illustrations/risk-of-bias-assessment-systematic-review-overall-judgment.webp) RoB 2 uses a conservative rule for the overall study judgment (or per-outcome judgment if you assessed by outcome): any domain rated high risk makes the overall judgment high risk, any domain rated some concerns without any high-risk domain makes the overall judgment some concerns, and only all-low domains produce an overall low-risk judgment. ROBINS-I follows the same conservative logic one level up, where any domain at serious or critical risk drags the overall rating down accordingly. You never average domains. One fatal flaw is fatal. ### Step 6: Resolve Disagreements and Measure Agreement Reconcile every mismatched judgment by discussion with the papers open, not from memory. Many teams also quantify agreement with [Cohen's kappa](/blog/research-skills/cohens-kappa-how-to-calculate-and-interpret), which corrects raw percent agreement for the agreement you would expect by chance. Here is the worked calculation for a 20-item pilot where both assessors agreed on 16 items. ```text Raw agreement (Po) = 16 / 20 = 0.80 Assessor A judged 15 low, 5 high Assessor B judged 13 low, 7 high Chance agreement (Pe) = (15/20)(13/20) + (5/20)(7/20) = 0.4875 + 0.0875 = 0.575 Kappa = (Po - Pe) / (1 - Pe) = (0.80 - 0.575) / (1 - 0.575) = 0.225 / 0.425 = 0.53 ``` A kappa around 0.53 in a pilot signals moderate agreement, which is common and fixable. The usual fix is writing explicit operational rules, for example "self-reported outcome measures without blinded assessment always score high risk in D4, regardless of what the authors claim." ### Step 7: Visualize Your Judgments The R package robvis converts your spreadsheet into the two figures most journals expect, a traffic-light plot showing each judgment cell by cell and a weighted bar chart showing the distribution of judgments per domain. ```r install.packages("robvis") library(robvis) rob_data <- read.csv("rob2_assessments.csv") head(rob_data) # Study D1 D2 D3 D4 D5 # 1 Trial 01 + + + x + # 2 Trial 02 + + - + + # 3 Trial 03 x + + + x rob_summary(rob_data, tool = "ROB2", weighted = FALSE) # Expected output: a stacked bar chart with one bar per domain # showing the share of low, some-concerns, and high judgments rob_traffic_light(rob_data, tool = "ROB2") # Expected output: a colored grid with one row per study, # one column per domain, and the overall judgment column at the right ``` The `tool` argument must match your spreadsheet's domain structure. Passing `"ROB2"` to data laid out for ROBINS-I produces a cryptic column-count error, so save one CSV per tool if your review mixes designs, exactly as reviews that pair RoB 2 with ROBINS-I and JBI checklists do in practice [2]. ### Step 8: Feed the Results Into Synthesis and Certainty Ratings Risk of bias judgments are inputs, not decoration. Standard options include restricting the primary analysis to low-risk studies, running a sensitivity analysis that removes high-risk studies and comparing estimates, and downgrading certainty in GRADE for the relevant domain. GRADE downgrades observational evidence starting at high risk by default and requires an explicit risk of bias judgment for every outcome in every evidence profile. The traumatic brain injury review tied ROBINS-I judgments directly into its GRADE certainty ratings [1], and the physical activity review for disadvantaged older adults paired an adapted MMAT with GRADE [5]. Whatever combination you choose, state the mapping from tool domain to GRADE downgrade in your methods, so a reader can reproduce the logic. ## Tool-Specific Judgment Notes ### ROBINS-I for Non-Randomized Studies ROBINS-I asks you to do something no trial-based tool requires: specify in advance the confounding domains and the covariates that a hypothetical randomized trial of the same question would balance. This specification belongs in the protocol, not in the assessment itself. Reviews of observational questions, such as the association between child abuse and lower urinary tract symptoms, applied ROBINS-I across 19 included studies spanning both pediatric and adult populations [15], and cardiovascular predictor reviews applying QUIPS face the same confounding-heavy structure even with a different tool [10]. The extra domains, especially confounding and classification of interventions, are where most judgments land at moderate or serious risk, which is why observational syntheses almost always report lower certainty than randomized syntheses. ### QUADAS-2 and QUADAS-3 for Diagnostic Accuracy QUADAS-2 evaluates four domains: patient selection, index test, reference standard, and flow and timing. The first three domains carry both a risk of bias rating and an applicability rating, because a perfectly executed study using an index test in the wrong clinical context is still unhelpful. QUADAS-3 modernizes the question bank and is the version pre-specified in current protocols, with an adapted QUADAS-C used when studies directly compare the accuracy of two index tests in the same patients [12]. In a published QUADAS-2 assessment of clinical tests for foraminal stenosis, only two of 15 included studies achieved an overall low-risk judgment, which is a realistic reminder that diagnostic accuracy literature is rarely as clean as its sample sizes suggest [11]. ### PROBAST for Prediction Models PROBAST organizes judgment into four domains: participants, predictors, outcome, and analysis. It applies to model development studies and to external validation studies, and it separates risk of bias from applicability concerns because a model validated in one country on one data source can be internally honest and externally useless. Reviews using PROBAST frequently find the analysis domain to be the weak point, since small samples, overfitting, and missing validation drive high-risk judgments [13], [14]. If your review covers machine learning prediction models, expect heterogeneous reporting of targets, horizons, and uncertainty, and judge the analysis domain against the tool's criteria rather than against the model type's fashion. ### Checklists for Cohorts, Case Series, and Mixed Methods The Newcastle-Ottawa Scale awards up to nine stars across selection, comparability, and outcome ascertainment, and its simplicity is also its weakness, because star totals collapse distinct flaws into one number. The JBI checklists for case series and prevalence studies use sequential yes-or-no items [2],. The MMAT covers five methodological categories and is often adapted when rapid reviews combine randomized, non-randomized, and qualitative designs [5]. When you use a checklist rather than a domain-based tool, resist the temptation to report only the total score. Report item-level results so readers can see which specific weakness drove your judgment. ## Where Risk of Bias Fits in the Review Workflow Risk of bias assessment sits in the middle of the pipeline, after study selection and alongside data extraction. It cannot be done well if the earlier steps were done loosely. A defensible systematic literature search means multiple databases, a documented search strategy with dates, and grey-literature sources where relevant, because reviews that search narrowly manufacture a biased evidence base before any bias tool is applied [16], [3], [4]. Once studies are included, the assessment runs in parallel with extraction, and the same duplicate-assessment rule applies to both. This is also where the distinction from a narrative exercise matters if you are learning how to write literature review sections. In a traditional narrative review, an author weighs studies by judgment and mentions credibility in passing. In a systematic review, credibility is measured with a named tool, per domain, per study, before any conclusions are drafted, and the results section reports the bias distribution across studies as a finding in its own right. If your evidence base ends up mostly at high or unclear risk of bias, that is a reportable result, not an embarrassment to hide. ## Practical Tips for Smoother Assessments **Quote first, judge second.** Write the supporting sentence from the paper before you pick the judgment. This reverses the natural order in which readers decide they dislike a study and then hunt for a justification. **Judge the study as described, not as you imagine it.** If a trial does not report allocation concealment, the answer is no information or probably no, depending on the tool's rules, and never "they probably did it right." **Assess per outcome when outcomes differ in objectivity.** Mortality cannot be influenced by lack of blinding, patient-reported itchiness can. RoB 2 supports outcome-specific judgments, and reviews of subjective outcomes such as quality of life after cervical spine surgery benefit from this granularity [17]. **Timebox reconciliation.** Agree as a team that any domain disagreement surviving a fifteen-minute discussion goes to the arbitrator with a one-paragraph memo from each assessor. **Keep your blank and completed forms.** Journals, guideline panels, and updating teams all ask for them, and re-creating judgments two years later from memory is impossible. ## Common Mistakes and Limitations The most frequent errors in risk of bias assessment in systematic reviews are predictable, and each has a specific fix. **Using the wrong tool for the design.** Applying an RCT checklist to a cohort study misses confounding entirely, the flaw that dominates non-randomized evidence. Match the tool from the table above before you start. **Scoring studies instead of domains.** A single overall "good" or "bad" label discards the information that a study measured its outcome flawlessly but lost a third of its participants. Domain-level judgments carry the content, and the overall judgment is a conservative summary of them. **Conflating risk of bias with certainty of evidence.** A review can pool low-risk studies and still rate the evidence as low certainty because of imprecision or inconsistency. Conversely, judgments at high risk of bias are one downgrade among several possible reasons for low certainty. Keep the two concepts in their separate columns. **Hiding low-risk-of-bias studies with unclear judgments.** "Unclear" used as a comfort zone turns a domain tool into a quality-free checklist. Every unclear judgment must cite exactly which piece of information was missing. **Letting results influence judgments.** Assessors who know that a study's result contradicts the emerging conclusion tend to scrutinize that study harder. Duplicate independent assessment plus protocol-fixed rules exists precisely to block this [4], [14]. **Ignoring the sensitivity of judgments to tool version.** QUADAS-2 and QUADAS-3, and RoB 1 versus RoB 2, can produce different judgments on the same paper. Name the version in your methods. Finally, accept the structural limitation of the whole enterprise. Risk of bias tools judge what is reported, and reporting is incomplete in most biomedical literature, so some judgments rest on absence of information. The judgments are structured expert opinion about internal validity, not measurements of truth, and a review's conclusions should always be read alongside its bias assessment rather than instead of it. ## Frequently Asked Questions ### What does risk of bias mean in a systematic review? It means the risk that systematic error in a study's design, conduct, or analysis distorted its reported result. The assessment judges internal validity domain by domain using a structured tool, separate from imprecision and separate from how important the topic is. ### Which risk of bias tool should I use? Match the tool to the study design. Use RoB 2 for randomized trials, ROBINS-I for non-randomized intervention studies, QUADAS-2 or QUADAS-3 for diagnostic accuracy, PROBAST for prediction models, and QUIPS for prognostic factor studies, with the table above as your reference. ### Can one person do the risk of bias assessment alone? Not reliably, and not by current standards. Duplicate independent assessment by two reviewers with arbitration of disagreements is the accepted method, and skipping it is one of the most common criticisms raised during peer review. ### What is the difference between risk of bias and study quality? Risk of bias concerns systematic error that distorts the result, while quality is a vaguer notion covering reporting and conduct. A study can be well reported and still biased, so structured domain tools have replaced generic quality scores in modern reviews. ### What is the difference between risk of bias and applicability? Risk of bias asks whether the study was done correctly, applicability asks whether it was done in a population, setting, or with measures that match your review question. Tools such as QUADAS-2 and PROBAST rate the two separately, and a study can score low on one and high on the other. ### How does risk of bias affect meta-analysis? High-risk studies can be excluded in a sensitivity analysis, the primary analysis can be restricted to low-risk studies, or certainty can be downgraded in GRADE. Whatever approach you choose must be declared in the protocol and applied consistently. ### Do I assess risk of bias in a narrative literature review? Narrative reviews do not use structured bias tools, which is one of the main reasons their conclusions carry less evidentiary weight. If you plan a formal synthesis, adopt a tool, because the structure is what makes your credibility judgments transparent and reproducible. ### Where do I record risk of bias assessments? Record judgments, justifications, and quotes in a structured spreadsheet with one row per study and one column per domain, and report the distribution in your results section with a traffic-light or bar-plot figure. Protocols registered in PROSPERO should pre-specify the tool so readers can check you followed your own plan. ## Sources 1. [Timing of pharmacological thromboprophylaxis in severe traumatic brain injury: a systematic review and meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43009626/) 2. [Intrarenal Pressure Monitoring and Control Strategies Using Suction in Flexible Ureteroscopy: An EAU Endourology Systematic Review of Technologies and Clinical Outcomes.](https://pubmed.ncbi.nlm.nih.gov/43012454/) 3. [Postoperative Behavioral and Psychological Factors Associated With Recurrent Weight Gain After Metabolic and Bariatric Surgery: A Systematic Review and Meta-Analysis.](https://pubmed.ncbi.nlm.nih.gov/43011444/) 4. [Effects of during-exercise oral hydration strategies on endurance performance and thermophysiological strain: a systematic review with meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43010950/) 5. [Promoting physical activity amongst disadvantaged and marginalised older adults: a rapid systematic review with meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43009851/) 6. [Effect of vitamin D supplementation on metabolic syndrome parameters in adults: a systematic review of randomized controlled trials exploring the role of baseline vitamin D status and supplementation dose.](https://pubmed.ncbi.nlm.nih.gov/43011259/) 7. [The impact of cognitive behavioral therapy and third-wave therapies on the quality of life of patients with obsessive-compulsive disorder: a systematic review and meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43010055/) 8. [Comparative Effectiveness of Non-Pharmacological Interventions on Postoperative Kinesiophobia in Total Knee Arthroplasty: A Systematic Review and Network Meta-Analysis of Randomized Controlled Trials.](https://pubmed.ncbi.nlm.nih.gov/43008900/) 9. [Comparative efficacy and safety of dupilumab and swallowed topical corticosteroid formulations in eosinophilic esophagitis: A systematic review and network meta-analysis of randomized controlled trials.](https://pubmed.ncbi.nlm.nih.gov/43012836/) 10. [Predictors of cardiovascular events in multiple myeloma: a systematic review and exploratory meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43011843/) 11. [Clinical findings associated with diagnosing lumbar foraminal stenosis: systematic review.](https://pubmed.ncbi.nlm.nih.gov/43009940/) 12. [Diagnostic performance and reported diagnostic thresholds of serum and synovial host-response biomarkers for periprosthetic joint infection in patients with inflammatory arthritis: protocol for a systematic review and diagnostic test accuracy meta-analysis.](https://pubmed.ncbi.nlm.nih.gov/43011413/) 13. [Lymphatic filariasis risk prediction models: A systematic review of modelling approaches, predictors, validation practices, and methodological quality.](https://pubmed.ncbi.nlm.nih.gov/43008045/) 14. [Machine Learning Models for Glycemic Prediction in Type 1 and Type 2 Diabetes: Systematic Review and Meta-Analysis.](https://pubmed.ncbi.nlm.nih.gov/43006686/) 15. [Child Abuse and Lower Urinary Tract Symptoms: A Systematic Review.](https://pubmed.ncbi.nlm.nih.gov/43012092/) 16. [Perioperative neurocognitive outcomes after robot-assisted pelvic surgery: a systematic review and meta-analysis of reported proportions, comparative risk and prognostic factors.](https://pubmed.ncbi.nlm.nih.gov/43009612/) 17. [Reported and Objective Outcome Measures Following Cervical Spine Surgery: A Systematic Review and Meta-Analysis.](https://pubmed.ncbi.nlm.nih.gov/43008897/)