NPC Comments on PCORI Proposal for Peer Review of Primary Research and Public Release of Research Findings

NPC logo

November 7, 2014

Dr. Joe Selby
Executive Director
Patient-Centered Outcomes Research Institute
1828 L St., NW, Suite 900
Washington, DC 20036

Dear Dr. Selby: 

The National Pharmaceutical Council (NPC) welcomes the opportunity to comment on the Patient-Centered Outcomes Research Institute’s (PCORI) Proposal for Peer Review of Primary Research and Public Release of Research Findings. Funded by the nation’s innovative biopharmaceutical companies, NPC is a health policy research organization dedicated to the advancement of good evidence and science and to fostering an environment in the United States that supports medical innovation. We believe that to inform treatment decisions, research must be transparent, reproducible and credible.

The goals of the proposed approach aim to develop and implement a process for peer-review of research and make research findings publicly available in a form and format useful to patients and clinicians. These goals align with principles for comparative effectiveness research (CER) developed by thought leaders at the outset of the nation’s increased investment in CER. These principles are focused on transparency, consideration of uncertainty in study results, generalizability and transferability of study findings across patients, settings and geographies.* Our recommendations to enhance the draft proposal are meant to ensure these goals and principles are met.

PCORI’s Proposed Research Peer-Review Process

1)      Registration

We applaud PCORI’s leadership to encourage the transparency, reproducibility and credibility of research by registering research projects at public sites prior to study conduct. In addition to registration at the Registry of Patient Registries (RoPR), we recommend analyses using clinical registries infrastructure should be encouraged to register their anlaysis prior to conduct at clinicaltrials.gov. There has been increasing registration of analyses using observational studies from various data sources including clinical registries over the past several years.** Registration of a study prior to analysis helps to ensure the analyses were pre-specified, the validity of the analytic process, and the certainty of the validity of the research.

Recommendation: In addition to the registration of clinical registries on RoPR, specific research questions should be registered prior to analysis on clinicaltrials.gov.

2)    Production of a draft final report, scientific abstract, and standalone results table.

We appreciate the recognition in the legislative language and PCORI peer-review proposal that patient differences matter. Differences in how treatments work in individual patients or sub-populations of patients is an important element to include in the research report results and discussion. However, patient response can vary based upon many factors–not just biology and comorbidities as outlined in the peer-review proposal. What works best for individual patients may be associated with the care setting, a patient’s education, sociodemographic characteristics or preferences. These characteristics–not just the biology–should be included in the discussion section of the final report.

Recommendation: The discussion section of the draft final report should describe findings based upon sub-populations that may differ based upon genetic, biologic, sociodemographic or patient preferences.

Conducting multiple analyses across various patient subgroups can result in an increased risk that treatment differences are simply due to chance. It is important that the methods section outlines which analyses are pre-specified, which analyses are adequately powered, which analyses are exploratory, and which were conducted in a post-hoc manner after the initial results.

Recommendation: Clarity on which analyses are pre-specified, and which are hypothesis-generating, should be included in the methods section, the results section and the discussion section of the draft final report.

It is rare that a single study has immediate impact. In part, this is because research often is cumulative. To increase adoption, a study must be put into context based upon prior knowledge. This is also supported by PCORI’s methodology guidelines which propose the conduct of a gap analysis or systematic review prior to the study. The discussion section of the draft report should describe how the findings are similar or dissimilar to prior studies and known evidence. In areas where PCORI may be funding several topics in one area, summarizing the totality of the evidence is important. For example, PCORI has funded eight studies assessing asthma treatment options for African Americans and Hispanics or Latinos. Summarizing the lessons individually and across these eight studies together will be important a critical part of the peer-review process for patients, providers and other stakeholders.

Recommendation: The discussion section of the draft final report should describe how the findings relate to prior studies and known evidence. Additional information on the generalizability and transferability across patients, settings, geographies and systems of care should be included. These elements should be incorporated in the medical and lay abstracts.

The Affordable Care Act is clear about the need to note potential conflicts of interest. Direct and indirect conflicts should be transparent regardless of the stakeholder and stakeholder ties for two reasons. First, PCORI’s research portfolio extends beyond comparing the red pill and blue pill. It focuses on health systems, treatment strategies and patient engagement. Second, nearly every PCORI-funded project will have a vested party. For example, a researcher in a hospital system assessing a quality improvement program to reduce readmissions who is also reimbursed based upon admissions may have a conflict. Similarly, a health system assessing the effectiveness of MRI imaging for cancer surveillance and who is also reimbursed for these costly procedures may have a conflict. Because all or nearly all PCORI-funded projects have patient and stakeholder engagement, there will be multiple and conflicting areas of interest.

Recommendation: Because no stakeholder (patient, provider or policy maker) can truly be objective or without bias, we recommend the research entity, the investigators conducting the research and the stakeholder partners should list all conflicts, not just those with ties to industry.

3)    PCORI peer review process of the draft final report.

PCORI has a thoughtful group of experts ready in the Methodology Committee and via the merit review process. However, simply having a nationally recognized set of experts to conduct peer-review is likely insufficient due to the volume of reviews and the broad expertise needed. Additional tools, training and clarity on which elements will be required in the peer review process will be needed. Other similar exercises with experts found that expert agreement on whether a study was “sufficiently credible” or “insufficient” resulted in agreement only 50% of the time.***

Recommendation: Training, tools and objective criteria will be needed to ensure consistent and objective peer-review. To ensure peer-review is consistent across studies, experts and time, inter-rater reliability should be evaluated periodically.

Making Research Findings Publicly Available

1)    Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public.

To ensure that the research findings are comprehensible and accessible to patients, consumers and the general public, PCORI plans to review and edit the 500 word medical professional abstract and results table at an 8th-grade reading level. However, for patients and consumers who are older, may not speak English as their first language, or vary in education levels, study results written at an 8th-grade reading level may be difficult to comprehend. We recommend the readability be lowered to 5th- or 6th-grade levels to account for variation in health literacy. Patients and the lay public also will seek to understand: for whom do the study results apply?; and does the study confirm or refute other studies or improve my certainty? Patients and patient groups can help to ensure that the research is translated in a way that is useful to patients.

Recommendation: Information for patients and the general public should be written with 5th- or 6th-grade readability level. Study results should be put into context for the patient based upon how the study fits with prior studies and known evidence and to clarify for whom the study results apply. Patients or patient representatives should be involved in the review of the lay abstract prior to public availability to ensure the information is useful for patients and consumers.

2)    Submission of results table to ClinicalTrials.gov and/or other sites as appropriate with a link to the abstract and standalone results table on PCORI’s website.

Having all information available at a single site may provide patients, providers and others with a seamless way to learn about study results. Posting in two separate sites (e.g., PCORI’s website and clinical trials.gov) can be a burden to researchers and challenging to ensure the consistency of information and information release. An alternative approach could create a single source input with two-way communication to both ClinicalTrials.gov and the PCORI website via additional IT intraoperability. This may not be currently feasible due to the database, the need for additional programming, or funding. However, based upon the legislative language which allows for contracts with preference to the Agency for Healthcare Research and the National Institutes of Health, PCORI funding could enable two-way information exchange through IT solutions.

Recommendation: Working in collaboration with National Library of Medicine (and if needed, providing funding) could allow for a seamless input across platforms. This IT solution could decrease researcher burden, avoid duplication, and allow patients to find information in a more seamless manner.

Questions About the Proposal as a Whole

To ensure research is comprehensible and accessible to patients, providers, payers and policy makers, the research findings will need to be translated and disseminated in preferred language and communication channels. It is important that the checks and balances, which apply to a 500-page draft report, also apply to the 500-word abstract, the 50-character tweet and all the derivative communication products in between. The need for these checks and balances are two-fold. Firstly, the safeguards ensure that individual dissemination products are fair and balanced. Secondly, this ensures consistency of messaging across various dissemination products, communication channels and audiences.

Recommendation: Peer-review of all communication and dissemination materials is needed, not just the final report.

NPC appreciates the opportunity to comment on PCORI’s Proposal for Peer Review of Primary Research and Public Release of Research Findings. We offer these suggestions to ensure that the research is considered transparent, reproducible and credible by all stakeholders. NPC looks forward to further dialogue and engagement with the PCORI Board, Methodology Committee and staff.

Sincerely,

Jennifer S. Graff
Director, Comparative Effectiveness Research
National Pharmaceutical Council

-------------------------------------------------------------------------------------------------------------

* Luce BR, Drummond MF, Dubois RW et al. Principles for planning and conducting comparative effectiveness research. JCER. 2012; 1(5):431-44.

** ClinicalTrials.gov. Trends Chars, Maps. Accessed at http://clinicaltrials.gov/ct2/resources/trends#TypesOfRegisteredStudies.

*** Dreyer NA, Velentgas P, Westrich K et al. The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. J Manag Care Pharm. 2014;20(3):301-8.