February 5, 2019
Division of Dockets Management (HFA-305)
[FDA-2018-N-4000]
www.regulations.gov
Food and Drug Administration
5630 Fishers Lane Rm 1061
Rockville, Maryland 20852
RE: Framework for a Real-World Evidence Program; Availability
Submitted electronically via: www.regulations.gov FDA-2018-N-4000
Dear Sir or Madam:
Thank you for the opportunity to provide input on issues related to the Framework for a Real-World Evidence Program; Availability.[1] At the National Pharmaceutical Council (NPC), we share in the Food and Drug Administration’s view that “real-world evidence provides us with a potential source of information that can complement, augment and expand our understanding of how best to use medical products—improving what we know about our medical care.” When done correctly, real-world data (RWD) sources and real-world evidence (RWE) can provide meaningful information and accelerate clinical knowledge. When done poorly or with poor-quality data, misinterpretations and poor health outcomes for patients are possible. Therefore, we welcome the FDA’s comprehensive review of regulations and policies to consider the potential use of RWE and the multifaceted approach through demonstration projects, stakeholder engagement, internal processes, and guidance documents to advance the FDA’s Real-World Evidence Program.
NPC is a health policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. We are supported by the major U.S. research-based biopharmaceutical companies and focus on the issues of evidence, access, innovation, and the value of medicines for patients. Our research helps inform critical health policy debates and supports the achievement of the best patient outcomes in the most efficient way possible.
We applaud the FDA’s consideration of RWD and RWE to inform regulatory decisions. Our comments focus on the following three areas:
- Stakeholders Agree that Observational Studies Can Complement Existing Clinical Trial Evidence and Ensure Relevant, Timely and Reliable Information to Guide Decision-Making
- Standards and Best Practices for the Conduct of Observational Studies Exist and Can Be Foundational to Future Guidance
- Methods and Standards for Observational Studies Exist
- Protocol Submission Is an Important, But Insufficient, Step in Developing Transparent, Reproducible and Credible Observational Studies
- The Multifaceted RWE Program Requires a Clear Vision, Timelines for Various Programmatic Elements, Clear Definitions and a Centralized Repository
- Stating the Vision for RWE Can Stimulate the Advancement of RWD and RWE Tools
- Timelines for Programmatic Elements and Clear Consideration of Comments Can Facilitate Broader External Engagement
- Modifications to the Foundational Definitions of RWD and RWE Are Needed
- A Harmonized and Centralized Repository of Multiple FDA Guidance Documents Related to RWE Is Useful for Both Internal and External Stakeholders
We urge the FDA to carefully consider all comments received and prioritize finalization of this Framework and the development of subsequent guidance in an expeditious manner.
- Stakeholders Agree that Observational Studies Can Complement Existing Clinical Trial Evidence and Ensure Relevant, Timely and Reliable Information to Guide Decision-Making
Across health care, RWE is being used to better understand how treatments work across care settings and more diverse patient populations. Improved evidence can enable more reliable treatment decisions, improve health system efficiency, and optimize care delivery and patient outcomes. For stakeholders making daily decisions about which treatments will be recommended, reimbursed or prescribed, stakeholders are looking for not just more evidence, but better evidence. NPC conducted several studies that examined, developed and tested frameworks to understand the types of evidence, including pragmatic clinical trials and observational studies, stakeholders desire to inform their decision-making.
- To harmonize the evidence payers desire for coverage and formulary decisions with the evidence researchers develop, NPC and AcademyHealth identified and tested a framework for evidence development with payers. In the study, Developing Evidence That Is Fit for Purpose: A Framework for Payer and Research Dialogue, the framework tested three clinical scenarios and the associated payer decisions (e.g., cover, determine formulary tier placement, implement utilization management processes, etc.) to determine if new evidence would be meaningful, and if so, which types of evidence. Payers sought impactful, high-quality evidence from a variety of data sources and designs, including observational studies and pragmatic clinical trials, if the studies were fit for the clinical condition, contextual factors and payer decision. Observational studies were sought to address effectiveness, long-term safety, care coordination, impact on quality measures and identify treatment differences among specific patient populations.[2]
- To assess the patient community’s knowledge about RWE, their level of trust in RWE and interest in using it in decision-making, NPC and the National Health Council collaborated to convene representatives from patient organizations in Patient-Community Perspectives on Real-World Evidence: Enhancing Engagement, Understanding, and Trust. While most participants were initially unaware of RWE and its actual or potential uses, the patient community found potential in using RWE and were surprised this evidence was not already deeply embedded in health care decision-making. In many cases, the patient community recognized the ability of RWE to answer questions that matter most to them: “How does this treatment work in patients who look like me.”[3]
NPC recommends that FDA prioritize the evaluation of the potential role for observational studies to contribute to the evidence of drug product effectiveness.
- Standards and Best Practices for the Conduct of Observational Studies Exist and Can Be Foundational to Future Guidance
- Methods and Standards for Observational Studies Exist
We appreciate the FDA’s consideration of the reliability of RWD (e.g., data accrual and data quality control) and the relevance of the underlying data. However, we also recommend that the FDA consider not only the potential study design but the analytic approaches to transform RWD into high-quality RWE. Over the past five to 10 years there has been a proliferation of best practices and standards for conducting observational studies developed by a variety of disciplines (e.g., epidemiology, pharmacovigilance, and health economics) and organizations (e.g., professional societies, cross-stakeholder efforts, and governmental and non-governmental organizations). Among these efforts:
- The NPC-funded and multi-stakeholder developed Good ReseArch for Comparative Effectiveness (GRACE) Principles (www.graceprinciples.org) and 11-item validated GRACE Checklist (Appendix A) were established to guide good practices for the design, conduct, analysis and reporting of observational studies for researchers and others. [4],[5] The GRACE effort has been cited by several organizations, including the International Society of Pharmacoepidemiology, the Journal of Managed Care and Specialty Pharmacy and the United Kingdom’s National Institute for Health and Clinical Excellence.
- Similarly, the Comparative Effectiveness Research Collaborative (www.cercollaborative.org), composed of the Academy of Managed Care Pharmacy, the International Society for Pharmacoeconomics and Outcomes Research and NPC, developed questionnaires and online tools to guide reviewers and enable uniformity and transparency in the evaluation and use of evidence for coverage and health care decision-making. The 33-item CER Collaborative checklist for evaluating observational studies can help end-users assess the relevance of a study and credibility of the RWD and the RWE methods for their decision. [6] (Appendix B) These tools have been accessed by over 4,000 professionals in the pharmacy, biopharmaceutical industry and managed care fields.
In addition to these NPC-led efforts, other standards for conducting and reporting observational studies have proliferated. To asses whether there was agreement across these standards and guidelines, researchers at NPC and the University of Pittsburgh compared and contrasted nine existing guidelines and standards for analyzing observational studies across 23 common methodological elements (e.g., study protocol, data analysis, and so forth). The researchers found there was general agreement at a high level on the basic elements required (e.g., the need for a study protocol); however, there was variation in how various guidelines and standards recommend those elements should be conducted (e.g., how to handle missing data).[7] (Appendix C and D)
Any development of guidance on standards for observational studies should build upon the foundation of these good practices. These standards complement and add to the FDA Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data. In totality, these guidelines can help identify areas where consensus exists, when further alignment and external stakeholder engagement is needed and if important gaps need to be addressed. As described in Morton et al., “Quicker consensus across stakeholders may be feasible for elements which are already considered important across the nine standards/guidelines. More effort and time would be required where there is disagreement on how to best address the element or where the element has been addressed by just a few organizations. Finally, there may be methods which were not prioritized, but may rise to the level of prioritization when considered by various stakeholders.” A set of methods and standards for analyzing observational studies developed in consultation with multiple stakeholders, including RWE scientists from the biopharmaceutical industry, and utilized by the FDA may encourage other stakeholders to accept high-quality studies and disregard poor-quality studies benefiting not only regulatory decisions but decisions by all stakeholders.
As outlined in the Framework, the FDA has extensive experience using RWD sources to monitor the safety of products; however, the activities to incorporate these methods and tools into more routine elements to assess efficacy and effectiveness is less established. Scientists in the private sector have experience applying observational study designs for safety and effectiveness. Recognizing there are additional elements to consider in the FDA’s three-factor approach, the biopharmaceutical industry also can provide use cases to more clearly delineate the appropriate contexts for RWE.
NPC recommends that the development of guidance on the standards for observational studies should build upon the foundation of existing good practices and be developed in consultation with experts, including RWE scientists in the biopharmaceutical industry and professional research societies. While there is general agreement among researchers and other stakeholders on the best methods and standards for conducting observational studies, further efforts are needed. FDA can advance these efforts.
- Protocol Submission Is an Important, But Insufficient, Step in Developing Transparent, Reproducible and Credible Observational Studies
Central to the credibility of observational studies is the transparency of research methods used to conduct these analyses. As outlined in the Framework, submission of study protocol to the FDA before execution is critical for ensuring the hypotheses were pre-specified and increasing the confidence in observational study results intended for regulatory decision-making. Despite the call for RWE to be “transparent, reproducible, disclosed, accurate and valid,” there is little agreement on how to achieve these elements.
To help researchers identify the steps required to make observational study methods transparent and allow end-users to assess whether they can rely on various studies for decision-making, NPC and AcademyHealth undertook a collaborative project. Through this collaboration, we worked with key stakeholders and opinion leaders to identify recommendations and convened a multi-stakeholder roundtable to refine these recommendations. The resulting six recommendations include:
- The RWE hypothesis statement should be pre-specified and logged in a repository. The origin of the hypothesis should be described.
- The RWE analysis plan should be pre-specified and logged in a repository. Deviations from the analysis plan should be documented, and the rationale for changes should be provided.
- Steps should be documented to assure that the study data used is feasible, appropriate and of high quality for the research question.
- Sensitivity analyses should be performed on key definitions and outcomes.
- Data coding should be provided upon request, along with a natural language description of the coding logic to allow other research teams to replicate the study in an appropriate secondary dataset.
- Access to summary tables of data should be provided.
These recommendations can serve as a guide to help the FDA and other end-users assess whether the methods used in observational studies are reliable. However, there are practical concerns and considerations for the FDA to consider before developing reporting requirements for retrospective and prospective observational study designs.
First, as noted in the Framework, there are concerns associated with whether study results from existing datasets are based on the initial question of interest or if the questions evolved to provide more favorable results. Clinicaltrials.gov was established to promote transparency by requiring clinical trials to be registered publicly. While nearly one-fifth of all studies included in clincialtrials.gov are observational studies, another subset of observational studies, using registry designs, are included in the Registry of Patient Registries (RoPR) repository. As we found in the NPC/AcademyHealth interviews, there is no optimal location to pre-specify and publicly record hypotheses for observational studies. Some organizations have developed their websites to track studies conducted or sponsored by their organizations. Given the concerns with multiple analyses, public specification with time and date stamps may help identify when hypotheses were specified and whether and how the analytic plans changed. However, existing public repositories are not available nor optimal for the unique aspects of observational studies. FDA leadership with other external stakeholders would be useful.
Second, the Framework cites FDA’s perspective on using RWD for safety studies in the Pharmacoepidemiologic Guidance to test pre-specified hypotheses and emphasizes that investigators should submit protocols to FDA before study initiation. While we agree that the submission of protocols before study initiation and final reports upon completion is helpful, we contend that protocol submission to the FDA is insufficient. For example, analysis plans should be part of the protocol or submitted separately. Detailed analysis plans enable end-users to understand how a study was conducted and evolved. This plan should specify elements including the numerous curation and analytic approaches used to identify the target study population, outcomes, interventions and comparators, outcome variables and statistical methods for handling missing data, as well as ways to mitigate bias and confounding.
Changes in analysis plans are not uncommon among randomized controlled trials if data discrepancies are found based on blinded data. Similarly, changes to the analysis plan would not be unique to observational studies due to data aberrations such as changes in diagnosis codes due to reimbursement changes which may occur over time. However, high-quality studies track specific changes and the rationale in a transparent manner. In addition, the analysis plan and report should test the impact of assumptions on study findings (e.g., the identification of a patient population based on medical diagnosis alone vs. the identification of a patient population based on both medical diagnosis and prescription use).
We recommend FDA, along with other stakeholders, work to more clearly articulate the requirements for transparent, reproducible evidentiary or hypothesis-testing studies. Greater clarity will benefit not only observational studies developed for regulatory decisions but also hypothesis-testing observational studies designed for other purposes.
- The Multifaceted RWE Program Requires a Clear Vision, Timelines for Various Programmatic Elements, Clear Definitions and a Centralized Repository
- Stating the Vision for RWE Can Stimulate the Advancement of RWD and RWE Tools
As the FDA has outlined, advancing the use of RWD into regulatory-quality RWE is a key strategic priority. Accomplishing this goal requires accessible and interoperable data, trusted curation methods, credible analytics, new cutting-edge data skills, and cultural adoption. These changes necessitate a clear and specific vision. Stating the vision and objectives clearly and explicitly is essential for stakeholders to evaluate the intent and contribute to the proposed effort. Without clarity, the use of RWD and RWE tools may be stymied because the regulatory vision remains uncertain. We recognize that certain data or technology developments may require maturation; however, it would be helpful for all stakeholders to understand FDA’s ultimate vision for its RWE Program including a more robust outline of which regulatory decisions RWE may be useful to address.
NPC recommends articulating the vision for the RWE Program to allow all stakeholders to contribute to the proposed effort.
- Timelines for Programmatic Elements and Clear Consideration of Comments Can Facilitate Broader External Engagement
Timelines for the various programmatic elements outlined in the Framework priorities should be announced in advance to allow external engagement to occur. Delineation of the time table for the programmatic elements provides interested researchers and stakeholders with ample opportunity to contribute research and set aside needed resources to provide input during future engagement opportunities. In addition, clarity in how the comments FDA receives will be considered and weighed can build credibility and trust. This clarity is vital not only for guidance documents but also for the Framework and public meetings and other forums to engage stakeholders.
NPC recommends that timelines for various programmatic elements be established including timelines to allow the consideration of external comments before finalizing the Framework and future guidance documents.
- Modifications to the Foundational Definitions of RWD and RWE Are Needed
We appreciate the importance of clarifying the differences between RWD and the curation and analytic approaches required to transform RWD into meaningful and usable RWE. Because these definitions are foundational to a common understanding, we recommend external engagement to develop consensus prior to finalizing these definitions. We provide specific examples where greater clarity is sought.
- It is unclear how and when the transition from RWD to RWE occurs (e.g., pg. 7 of the Framework). While RWE requires appropriate curation and analytics, illustrative examples of the evolution from data to evidence and greater clarity on when elements are RWD and when these elements are RWE would be helpful. For example, to create similar patient cohorts for historical controls appropriate statistical methods should be applied to RWD. Would this information that has undergone analysis and curation be considered RWD or RWE?
- As outlined, RWE is the “clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD.” There is growing evidence of the association of social determinants of health such as health-related behaviors (e.g., smoking or diet), socioeconomic factors (e.g., access to specialized care centers or distance to care) and environmental factors (e.g., pollen or air quality) with health outcomes. Limiting RWE to the clinical evidence may undermine the consideration of factors which should be addressed to mitigate confounding and bias.
- RWE can play an important role to assess the benefits and risks of pharmaceutical interventions, surgeries, medical devices, care delivery programs, or even the decisions to treat. For example, RWE may compare a pharmaceutical intervention with a surgical procedure, which is not a “medical product.” We recommend replacing “medical product” with “intervention” to allow the full range of relevant intervention strategies to be considered as comparators.
NPC recommends broader stakeholder engagement to clarify the foundational RWD and RWE definitions and to advance common terminology across regulatory and to inform non-regulatory uses.
- A Harmonized and Centralized Repository of Multiple FDA Guidance Documents Related to RWE Is Useful for Both Internal and External Stakeholders
The Framework outlines multiple existing guidance documents which address the use of electronic data, oversight issues related to RWD, use of patient-reported outcome measures and best practices for conducting and reporting pharmacoepidemiologic safety studies. For organizations seeking to apply various guidance, clarity in the criteria and standards for implementing RWE across documents is critical. As outlined, the RWE Subcommittee of the Center for Drug Evaluation and Research’s Medical Policy and Program Review Council will address cross-cutting issues. However, a central repository for guidance related to RWD and RWE would allow a more complete view of the guidance documents for harnessing these tools for both those internal and external to the agency.
NPC recommends that FDA develop a process to ensure a harmonized view across documents and create a central repository to enable a consistent application of RWE tools across divisions, regulatory decisions, and biopharmaceutical manufacturers.
In closing, NPC shares the FDA’s goal to complement, augment, and expand our understanding of how to best use medical products using RWE tools. As the volume, variety, velocity and veracity of RWD sources grows, ongoing development and engagement will be needed to harness the value of these data sources across a product’s life cycle. The implications for the Framework extend beyond regulatory decision-making as other health care stakeholders consider the FDA to be an independent and unbiased arbiter of good evidence and science. Definitions, standards, and demonstration projects may help inform population health decision-making. And most importantly, the use of RWE may allow patients to find someone that "looks like me" as assurance of how a treatment might benefit them personally.
NPC appreciates this opportunity to comment on the Framework for FDA’s Real-World Evidence Program. We look forward to participating in the dialogue to advance the development of good evidence and science. We would be pleased to expand upon our comments, share our research or engage in public meetings and other stakeholder forums. Thank you for the opportunity to provide these comments.
Respectfully submitted,
Robert W. Dubois, MD, PhD
Chief Science Officer
National Pharmaceutical Council
[1] U.S. Food & Drug Administration. Framework for FDA’s Real-World Evidence Program. December 2018.
[2] Sabharwal RK, Graff JS, Holve E, Dubois RW. Developing Evidence that is Fit for Purpose: A Framework for Payer and Research Dialogue. Am J Manag Care. 2015;21(9):e545-e551.
[3] Oehrlein E, Graff JS, Harris J, Perfetto EM. Patient-Community Perspectives on Real-World Evidence: Enhancing Engagement, Understanding, and Trust. The Patient. 2019; January 22.
[4] Dreyer NA, Schneeweiss S, McNeil BJ, et al. GRACE Principles: Recognizing High-quality Observational Studies of Comparative Effectiveness. Am J Manag Care. 2010; 16(6):467-71.
[5] Dreyer NA, Velentgas P, Westrich K, Dubois R. The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. J Manag Care Spec Pharm. 2014;20(2):301-8.
[6] Berger ML, Martin BC, Husereau D et al. A Questionnaire to Assess the Relevance and Credibility of Observational Studies to Inform Health Care Decision Making: An ISPOR-AMCP-NPC Good Practice Task Force Report. Value Health. 2014;17(2):143-56.
[7] Morton SC, Costlow MR, Graff JS, Dubois RW. Standards and Guidelines for Observational Studies: Quality is in the Eye of the Beholder. J Clin Epidemiol. 2016;71:3-10.