Making Informed Decisions: Assessing the Strengths and Weaknesses of Study Designs and Analytic Methods for Comparative Effectiveness Research

Senior Editors: Velengtas, P (Outcome); Mohr, P, Messner, DA (Center for Medical Technology Policy).
Publication: February 2012
Size: 34 pages

Download the PDF

Comparative effectiveness research (CER) aims to close a gap in evidence by producing information that decision-makers can use to make informed treatment and coverage decisions. CER therefore must be designed to meet the real-world needs of decision-makers. This practical focus of CER introduces unique requirements for the design and implementation of studies. For example, tradeoffs of validity, relevance, feasibility, and timeliness must be considered in the context of the specific decision-makers and decisions. These unique considerations lead to questions concerning which study designs and methods are appropriate for CER questions.

Understanding which approach to conducting a CER study is best to use under which circumstances is a question of significant debate among researchers. Generally, CER approaches fall into two broad categories: experimental study designs and methods, and nonexperimental study designs and methods. In experimental designs, patients are randomized (assigned by chance, not by a physician’s decision) to a particular therapy based on the study protocol. In nonexperimental designs, patients and physicians make real-world treatment decisions, and patterns of care and outcomes are observed. Some argue that experimental study designs are needed to answer most CER questions, because randomization eliminates concerns regarding channeling bias (ie, the tendency of clinicians to prescribe specific treatments based on a patient’s prognosis), and achieves balance with regard to measured and unmeasured confounders (ie, extraneous variables that may play a role in the outcomes of interest). Others believe that nonexperimental studies, incorporating ways to address channeling bias and other confounding in the design and/or analysis, represent important alternatives to randomized studies. In practice, each research approach has advantages and disadvantages, and the research approach for a CER question should be selected based upon the specific features or characteristics of the study question.

The purpose of this document is to provide brief descriptions of both experimental and nonexperimental study designs and methods that may be used to address CER study questions. Each design or analytic topic is described, along with the strengths and limitations associated with the approach. Examples are provided to demonstrate the use of the described methods in the literature. While this document does not prescribe methodologies for specific CER research questions, it is part of a larger effort to develop a systematic approach to determining which methods are best able to address given CER questions. In its current form, this document provides information needed for researchers and consumers of the literature to understand the relative strengths and limitations of various CER design and analytic approaches, and how their use may affect study results and interpretation.

This document is organized into four sections: experimental study designs; experimental methods; nonexperimental study designs; and nonexperimental methods. It was published by the National Pharmaceutical Council, Outcome, and the Center for Medical Technology Policy.