NPC Comments on Revised PCORI Methodology Standards

NPC logo

March 28, 2016

Dr. Joe Selby
Executive Director
Patient-Centered Outcomes Research Institute
1828 L St., NW, Suite 900
Washington, DC 20036 

Dr. Robin Newhouse
Chair PCORI Methodology Committee
Attn: [email protected]

Dear Methodology Committee Members:

We welcome the opportunity to comment on the new and revised Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards published for comment on January 25, 2016. We believe that the methodological standards provide a fundamental base for evidence to improve and transform medical practice. These standards are an important element to guide researchers in the conduct of high-quality research, for decision-makers to distinguish credible and valid research, and for patients and other stakeholders to identify research that is trustworthy and useful.

The National Pharmaceutical Council (NPC), a health policy research organization dedicated to the advancement of good evidence and science and to fostering an environment in the United States that supports medical innovation, firmly believes that research should be conducted in a rigorous and transparent manner using common and agreed-upon methods. Founded in 1953, and supported by the nation’s major research-based pharmaceutical companies, NPC and the NPC member companies include a wealth of scientific and research expertise. We are pleased to offer comments regarding the latest revision and future opportunities for PCORI’s leadership on research that is useful for patient and provider decision-making.

Overall Comments:

Actionable and Measurable Standards:

The current version of the PCORI Methodology Standards identifies a minimum set of practices for conducting comparative effectiveness research (CER). However, it can be difficult if some standards are not “actionable” or “measurable.” For example, IR-5 specifies that researchers “provide sufficient information in reports to allow for assessments of the study’s internal and external validity,” but the criteria for “sufficient” remains in the eye of the researcher and the reader. In contrast, MD-2, which addresses statistical methods for dealing with missing data, outlines which methods are considered valid and which methods are discouraged. Actionable and measurable standards provide a target for decision-makers to judge the usability of the evidence produced and for researchers to aim.

Recommendation: We recommend the Methodology Committee engage in a review of the current methods standards to outline the criteria required to meet each standard. For standards in which there are not clear requirements, examples of ways in which the standards would be fulfilled would benefit all stakeholders.

Standards Associated with Patient-Centeredness:

We applaud the Committee on clarifying how patients and stakeholders should be involved in the prioritization of research, conduct of research, and the dissemination of research findings. As outlined in the standards, a broad set of stakeholders, including the biopharmaceutical industry, contribute to a more robust research process. Purchasers, payers, and industry communities seek to ensure that relevant questions are addressed, research findings are usable, and results are translated and implemented in practice.1 This engagement is an important aspect toward improving the relevance of CER questions and ultimately the impact of CER on health care decision-making.2

Standards for Data Integrity and Rigorous Analysis:

We commend the Committee on the recognition of the growing body of research standards and the need to increase research plan and analytic transparency. These are important steps forward. However, we believe there are other opportunities for the PCORI Methodology Committee to play a pivotal role in the nation’s research enterprise.

We commend the broader recognition of standards such as STROBE (for observational research) and SRQR and COREQ (for qualitative research). As outlined in the Affordable Care Act, the methodology standards “shall build on existing work on methodological standards for defined categories of health interventions and for each of the major categories of comparative clinical effectiveness research methods.” Good practices identified by professional societies and consortia such as RECORD (for observational studies), the GRACE Checklist (for observational, the CER Collaborative (for observational studies, indirect treatment comparisons, and modeling studies), CENT (for N-of-1 trials), CONSORT Extension (for pragmatic trials), and others are not referenced in this version of standards.3, 4 ,5 ,6 ,7, 8 ,9 Because stakeholders are increasingly looking to PCORI as a leader in the establishment of standards for CER, broader inclusion and recognition of good research practices is needed. 2

Recommendation: We recommend that additional methods or standards developed by other groups be considered and at a minimum referenced to increase stakeholder awareness or adopted in the PCORI methods.

Over the past decade, standards for research have proliferated among different research disciplines (e.g., biostatistics, pharmacovigilance, econometrics). This proliferation can increase the reach of standards across the types of research. However, it can also have unintended consequences. For example, a recent study compared and contrasted nine existing sets of standards or guidelines (based on 23 elements) for conducting observational studies. While most guidelines standards agreed on what elements were important (e.g., the need for a study protocol), there was disagreement 52% of the time on how the standards should be acted upon and addressed.10 This disagreement can contribute to variation in study quality, and create discrepancies adopted into care decisions. There is a need to identify common and agreed-upon methods for research through consensus-based approaches. An ongoing process enables agreement where quicker consensus is feasible and an iterative process for new, novel, or controversial methods.

Recommendation: Stakeholders perceive PCORI to be a leader in establishing CER standards. Few other organizations have the access to research experts from a variety of research disciplines and communities or experience in facilitating multi-stakeholder processes for prioritization as PCORI. We commend the Committee on the advancement of the standards. We believe the Committee and PCORI can offer their leadership to advance a set of common and agreed-upon methods through a consensus-based process that will form the foundation for research that the public can use and trust.

Finally, we commend the Methodology Committee on the recognition of the importance of research transparency. Without insight into which outcomes were pre-specified and the research analysis process used, the validity of study results may be questioned. The efforts to register studies before study start at clinicaltrials.gov and other sites seek to increase research transparency.

Recommendation: Some sites such as clinicaltrials.gov were developed for clinical trials. Although used for observational studies as well, many modifications are needed for these sites to account for features relevant for observational studies (e.g., propensity methods to balance confounding, analytic definitions to identify exposures). PCORI has an opportunity to collaborate with other entities such as the National Institutes of Health to enhance existing research infrastructure which encourages transparent, reproducible, and credible research.

We offer these suggestions to improve the rigor, transparency, and conduct of research to inform patient and provider decision-making. We propose the development of more actionable and measurable standards, broader inclusion of all research methods, and a consensus-based process to identify a common and agreed-upon set of methods for the field of CER. The National Pharmaceutical Council looks forward to further dialogue and engagement with the PCORI Board, Methodology Committee, and staff.

Sincerely,

Jennifer S. Graff, PharmD
Vice President, Comparative Effectiveness Research
National Pharmaceutical Council

Robert W. Dubois, MD, PhD
Chief Science Officer
National Pharmaceutical Council

---------------------------------------------------------------------

  1. Concannon TW, Khodyakov D, Kotzias V, Fahey G, Graff J, Dubois RW. Employer, Insurer, and Industry Perspectives on Patient-Centered Comparative Effectiveness Research: Final Report. Santa Monica, CA: RAND Corporation, 2015. http://www.rand.org/pubs/research_reports/RR1242.html.
  2. 2016 Comparative Effectiveness Research and the Environment for Health Care Decision-Making. March 2016. Available at: http://www.npcnow.org/system/files/research/download/NPC_ChartPack_v04_highres.pdf.
  3. Benchimol EI, Smeeth L, Guttmann A, et al. RECORD Working Committee. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Med. 2015;12(10):e1001885.
  4. Dreyer NA, Velentgas P, Westrich K, Dubois R. The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. J Manag Care Pharm;2014:20(3):301-8.
  5. Berger M, Martin B, Husereau D, et al. A questionnaire to assess the relevance and credibility of observational studies to inform healthcare decision making: an ISPOR-AMCP-NPC good practice task force report. Value Health. 2014; 17:143-156.
  6. Caro JJ, Eddy DM, Kan H, et al. A modeling study questionnaire to assess study relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:174–82.
  7. Jansen J, Trikalinos TA, Cappelleri JP, et al. Indirect treatment comparison/network meta-analysis study questionnaire to assess study relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health 2014;17:157–173.
  8. Vohra S, Shamseer L, Sampson M, et al. CENT group. CONSORT extension for reporting N-of-1 trials (CENT) 2015 Statement. BMJ. 2015;350:h1738.
  9. Zwarenstein M, Treweek S, Gagnier JJ, et al. CONSORT group; Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.
  10. Morton C, Costlow R, Graff JS, Dbois RW. Standards and guidelines for observational studies: quality is in the eye of the beholder. J of Clin Epi. 2016;71;3-10.