Risk+of+Bias+Guidelines


 * ARCHE Risk of Bias (ROB) Guidelines **

Download guidelines: toc

**Types of Biases and ROB domains**
Allocation concealment || back to top]
 * ** Bias ** || ** ROB Domain ** ||
 * **Selection Bias** || Sequence generation
 * **Performance Bias** || Blinding study participants and personnel ||
 * **Detection Bias** || Blinding outcome assessors ||
 * **Attrition Bias** || Incomplete outcome data ||
 * **Reporting Bias** || Selective outcome reporting ||
 * **Other Bias** || “Other sources” of bias ||

**Sequence generation**

 * Was the allocation sequence adequately generated?
 * Randomization ensures that the groups being compared are balanced for known and unknown confounders.
 * Inadequate sequence generation can overestimate treatment effects by 12% (ROR 0.88, 95% CI 0.79, 0.99).[|[1]]

Statements that groups were blocked or stratified should **NOT** be considered sufficient for a ‘yes/low risk’ rating unless accompanied by a description indicating the sequence was computer generated. || back to top]
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low** || Description of a random component in sequence generation process (e.g., computer-generated random numbers, coin toss, random number table). ||
 * **Unclear** || Insufficient data is provided to make a judgment (e.g., only described as: random, randomly generated, randomized, etc.).
 * **High** || No randomization or inappropriate randomization (e.g. alternation, assignment based on birth date or day of hospital admission). ||

**Allocation concealment**

 * Was allocation adequately concealed?
 * Allocation concealment ensures that the randomization sequence is unknown to the person entering participants into a trial until allocation to an intervention has occurred.
 * Inadequate allocation concealment can exaggerate treatment effects by 18% (ROR 0.82, 95% CI 0.71, 0.94).[|[2]]

Sequentially numbered drug containers of identical appearance. Central allocation with description (i.e. pharmacy, web-based, call-in). Randomization sequence maintained by a 3rd party uninvolved in the investigation or off-site. Central allocation, but no further description is provided. Use context to determine whether this better fits in ‘low’ or ‘unclear.’ || Includes 1 or 2 of: opaque, sealed, sequentially numbered envelopes is described, but does not use **ALL 3** descriptors. Randomization sequence is kept on site (with no further description). || Any method in which the investigators could foresee the group assignment (e.g. alternation, date of birth, etc.). || back to top]
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low** || Sequentially numbered, opaque, sealed envelopes.
 * **Unclear** || No statement.
 * **High** || Open allocation schedule.

**Blinding**

 * Was knowledge of the allocated intervention adequately prevented during the study?
 * Blinding of key individuals in a trial can minimize performance and detection bias.
 * Studies not described as “double-blind” can overestimate treatment effects by 9% (ROR 0.91, 95% CI 0.83, 1.0).[|[3]]
 * Results are not consistent across studies.

Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken. Description of “double-blinding” with specification that the participants and study personnel are blinded. Description of the use of “double-dummy” or “matched placebo.” || The study did not address blinding. || Blinding attempted, but likely that it could have been broken and the outcome is likely to be influenced. || back to top]
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low** || No blinding or incomplete blinding, but the outcome is unlikely to be influenced by lack of blinding.
 * **Unclear** || Insufficient information to permit judgment.
 * **High** || No blinding or incomplete blinding and the outcome is likely to be influenced.

**Incomplete outcome data**

 * Were incomplete outcome data adequately addressed?
 * Per protocol analyses may yield more favourable estimates compared to intention-to-treat (ITT) analyses.
 * “Modified” ITT* may overestimate treatment effects by 15% (ROR 0.85, 95% CI 0.81, 0.88).[|[4]]
 * Results not consistent across studies.

No ITT, dropout rate <10%, and number and reasons for withdrawal balanced between groups. || Dropout rate <30%, but number and reason for dropouts very imbalanced between groups. Per protocol analysis only (unless under of patients switching groups is too small to make any important difference to the estimated intervention effect). ||
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low ROB** || ITT and dropout rate <10%.
 * **Unclear ROB** || ITT and/or dropout rate between 10 and 30%. ||
 * **High ROB** || Dropout rate ≥30%.
 * Modified ITT is not consistently defined, but may refer to how participants contribute outcome data (e.g., all participants available for follow-up at timepoint X), and whether all or a subset of randomized participants are included in the analysis (e.g., excluding participants that did not receive a specified minimum amount of the intended intervention). In assessing this domain, closely consider the description of what was done, the dropout rate, and the likelihood for bias.

back to top]

**Selective outcome reporting**

 * Is there an indication that the reports are free of selective outcome reporting (SOR)?
 * SOR occurs when a subset of the original variables recorded for investigation in the protocol is presented in the publication.
 * It may also be indicated by discrepancies in the primary outcomes proposed and those reported.
 * Statistically significant outcomes are more likely to be completely reported than non-significant outcomes (range of odds ratios 2.2 to 4.7).[|[5]]

No discrepancies between methods section and results section in the study report(s). ||
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low ROB** || No discrepancies between outcomes listed in the protocol and those described in the results.
 * **Unclear ROB** || N/A ||
 * **High ROB** || Outcomes are identified in the protocol or methods section that are not described in the results section of the report. ||

//When to search for protocols:// In a previous study, we found that searching for protocols had little yield, but often changed assessment when found. || Protocol registries: clinicaltrials.gov, ISRCTN, WHO, Australia/New Zealand registry || Compare the methods section with the results for: primary vs. secondary outcomes; first outcome reported in methods vs. first outcome reported in results; time points of outcome assessment || Request guidance from clinical leads to determine if there are key outcomes that should be reported according to the report topic. Decisions on searching for protocols and how to use them should be made at the outset of each review, based on the specific context.
 * Cochrane recommendations || Protocol should be compared with results.
 * Cochrane recommendations || Protocol should be compared with results.
 * Protocol or registry number is reported || Look for the protocol and compare outcomes in protocol with the results.
 * No protocol reported || Do not look for protocols.

back to top]

**Other sources of bias**
This is the default assessment. || back to top]
 * “Catch-all” domain.
 * Source of funding or early stopping should not be assessed in “other sources” of bias; however, continue to extract these items and report them in the results/discussion sections.
 * Conduct a sensitivity analysis for studies that received industry funding versus no industry funding.
 * Conflict of interest can be assessed in the “publication bias” domain of GRADE.
 * Specific “other sources” of bias will not be identified to assess for each project.
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low ROB** || No other sources of bias are identified.
 * **Unclear ROB** || N/A ||
 * **High ROB** || Other sources of bias such as: participant characteristics (baseline imbalances), study design characteristics (crossover, cluster randomized, blocked randomization in unblinded trials). ||

**Overall ROB assessment**
back to top]
 * ** ROB Decision ** || ** Criteria ** ||
 * **Low ROB** || All domains are assessed as low. ||
 * **Unclear ROB** || At least one domain is assessed as unclear and no domains are assessed as high. ||
 * **High ROB** || At least one domain is assessed as high. ||

[|[1]] Als-Nielsen B, Gluud LL, Gluud C. Methodological quality and treatment effects in randomised trials: a review of six empirical studies. 12th Cochrane Colloquium Oct 2-6, Ottawa, Ontario, Canada, 2004 [|[2]] Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457 [|[3]] Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457 [|[4]] Abraha I, Duca PG, Montedori A. Empirical evidence of bias: modified intention to treat analysis of randomised trials affects estimates of intervention efficacy. Z Evid Fortbild Qual Gesundhwes 2008; 102[Suppl VI]:9. [|[5]] Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008; 3(80):e3081.

Return to: **Tools**

media type="custom" key="17943382"