Prevalence of Avoidable and Bias-Inflicting Methodological Pitfalls in Real-World Studies of Medication Safety and Effectiveness

Bykov K, Patorno E, D’Andrea E, He M, Lee H, Graff JS, Franklin JM. Clinical Pharmacology & Therapeutics, July 14, 2021

Purpose

This paper focused on recognizing methodological flaws in real-world evidence (RWE) studies so that researchers can avoid these flaws as they conduct studies. The increasing availability of electronic health data has led to an abundance of studies that utilize these data sources to generate evidence on the effectiveness and safety of treatments. While real-world data (RWD) analysis can generate reliable and valid evidence, the potential for biased findings increases substantially in the absence of a rigorous and appropriate methodological approach to study design and analysis. Methodological problems in RWE hinders efforts to maximize the use of existing health care data and the ability to interpret and identify high-quality and valid RWE studies.

Methods

The prevalence of RWE studies with avoidable methodological flaws is unclear, so researchers from Harvard, Brigham and Women's and the National Pharmaceutical Council carried out an analysis of RWE studies to quantify the prevalence of avoidable methodological issues. They conducted a systematic review of 75 studies comparing the safety and effectiveness for cardiovascular, diabetes and osteoporosis medications. The researchers focused on RWE analyses done in health insurance claims databases, the most common RWD source in the United States. The study used a prespecified nine-item questionnaire to assess certain avoidable methodological flaws. Researchers focused their analysis on four pitfalls of RWE: time-related bias, depletion of susceptible individuals, adjustment for variables measured during follow-ups without appropriate statistical models, and reverse causation.

Findings

The researchers found that 95% of the studies they reviewed had at least one avoidable methodological issue known to incur bias and 81% had potentially at least one of the four major pitfalls:

• Time-related bias (57%)
• Potential for depletion of outcome-susceptible individuals (44%)
• Inappropriate adjustment for variables measured at follow-ups (41%)
• Potential for reverse causation (39%)

The researchers noted that it is unclear how much methodological flaws impact the validity of a study or whether these flaws result in substantial bias.

These findings provide an opportunity to improve the quality of RWD methods by recognizing these flaws ahead of time – not just after the study is complete. The authors believe that understanding the current state of RWE is essential to move the field forward, especially considering the rapidly growing availability of RWD and the increasing number of RWE studies coming out each year.