Research Methods 101: Observational Studies
This post is part of NPC's series on research methods.
This post is part of NPC's series on research methods.
An observational study is a very common type of research design in which the effects of a treatment or condition are studied without formally randomizing patients in an experimental design. These studies can be done prospectively, wherein data are collected about a group of patients going forward in time; or retrospectively, in which the researcher looks into the past, mining existing databases for data that have already been collected. These latter studies are frequently performed by using an electronic database that contains, for example, administrative, “billing,” or claims data. Less commonly, observational research uses electronic health records, which have greater clinical information that more closely resembles the data collected in an RCT. Observational studies often take place in "real world" environments, which allow researchers to collect data for a wide array of outcomes. Patients are not randomized in these studies, but the findings can be used to generate hypotheses for investigation in a more constrained experimental setting. Perhaps the best known observational study is the "Framingham study," which collected demographic and health data for a group of individuals over many years (and continues to do so) and has provided an understanding of the key risk factors for heart disease and stroke."
Observational studies present many advantages to the comparative effectiveness researcher. The study design can provide a unique glimpse of the use of a health care intervention in the “real world,” an essential step in gauging the gap between efficacy (can a treatment work in a controlled setting?) and effectiveness (does the treatment work in a real-life situation?). Furthermore, observational studies can be conducted at low cost, particularly if they involve the secondary analysis of existing data sources. CER often uses administrative databases, which are based upon the billing data submitted by providers during routine care. These databases typically have limited clinical information, may have errors in them, and generally do not undergo auditing.
The uncontrolled nature of observational studies allows them to be subject to bias and confounding. For example, doctors may prescribe a new medication only for the sickest patients. Comparing these outcomes (without careful statistical adjustment) with those from less ill patients receiving alternative treatment may lead to misleading results. Observational studies can identify important associations but cannot prove cause and effect. These studies can generate hypotheses that may require RCTs for fuller demonstration of those relationships. Secondary analysis can also be problematic if researchers overwork datasets by doing multiple exploratory analyses (e.g., data-dredging): the more we look, the more we find, even if those findings are merely statistical aberrations. Unfortunately, the growing need for CER and the wide availability of administrative databases may lead to selection of research of poor quality with inaccurate findings.
In comparative effectiveness research, observational studies are typically considered to be less conclusive than RCTs and meta-analyses. Nonetheless, they can be useful, especially because they examine typical care. Due to lower cost and improvements in health information, observational studies will become increasingly common. Patients and their providers will need to critically assess whether the described results are helpful or biased based upon how the study was performed.
Standards for Observational Studies
Efforts are underway to reduce bias in observational studies through rigorous, generally agreed upon methods and standards known as the GRACE Principles (Good ReseArch for Comparative Effectiveness). GRACE collaborators also are developing a GRACE checklist to "provide a validated tool for the assessment of observational CER quality and usefulness for decision-making." The checklist is "based on existing literature and guidance from experts with extensive experience in the conduct and utilization of observational CER." The GRACE principles, along with the checklist, should help to alleviate some of the concerns associated with observational studies.
Dr. Nancy Dreyer, Chief of Scientific Affairs and Senior Vice President at Outcome, a Quintiles Company, outlined how GRACE can help to ensure higher quality observational studies.
Payers remain challenged by observational studies because most of them do not have standards in place for the evaluation of this research. According to a recent study sponsored by NPC, payers consider observational studies very differently, which results in widely varying coverage options for patients. To assist payers, NPC, the Academy of Managed Care Pharmacy and the International Society for Pharmacoeconomics and Outcomes Research are collaborating on a toolkit that payers can use to determine the validity and applicability of an observational study to answer the research question at hand.
The debate over RCTs and observational studies and their role in answering questions is certain to continue. Dr. Robert Dubois, NPC’s chief science officer, outlines the pros and cons of both in this video.
For more on observational studies, be certain to check out NPC’s presentations at the International Society for Pharmacoeconomics and Outcomes Research 17th Annual International Meeting, which will be held in Washington, DC, on June 2-6, 2012.