What Does “Good” Look Like? New Study Explores Standards and Guidelines for Observational Studies

For Immediate Release
Contact:
Andrea Hofelich, [email protected], 202-827-2078
Allison Hydzik, [email protected], 412-647-9975

(Washington, DC, December 7, 2015)—Many organizations provide guidance on how to evaluate the quality of observational studies, but a new paper shows these standards and guidelines often lack agreement. This can lead to variations in funding priorities, research methods, publication and ultimately, the use of evidence in decision-making. The gaps identified in this paper are increasingly relevant as insights from real-world clinical experience are being sought to improve patient care and reduce costs.       

“Without a ‘gold-standard’ for conducting observational studies, the quality of the research is in the eye of the beholder,” said Sally C. Morton, PhD, Professor and Chair of the Department of Biostatistics at the University of Pittsburgh’s Graduate School of Public Health and the paper’s corresponding author. “If the full promise of real-world evidence and big data for health care is to be realized, a common set of agreed-upon minimum standards and guidelines is needed to promote consistency in study quality and broader application of findings to treatment decisions.”

Experts from the National Pharmaceutical Council (NPC) and the University of Pittsburgh Graduate School of Public Health’s Department of Biostatistics conducted the peer-reviewed study, “Standards and Guidelines for Observational Studies: Quality Is in the Eye of the Beholder,” which was published online in The Journal of Clinical Epidemiology. The researchers reviewed, compared and contrasted nine sets of standards and guidelines developed by public, private and professional societies in the U.S. and Europe. These nine sets outline how to conduct observational or real-world studies that leverage information from electronic health records, administrative claims, patient registries or data networks.  Authors evaluated the presence and agreement of 23 methodological elements (such as the need to use a study protocol, how to link data from different sources, or handle missing information) and compared how actionable each element was.

The authors found that, out of the 23 methodological elements, 14 (61 percent) were addressed by seven or more standards and guidelines, reflecting general agreement that these elements are important. However, for all but two of these 14 elements, there was disagreement on how the element should be addressed or acted upon. The remaining 9 elements varied in whether the sets of standards and guidelines agreed that the element was important or was included at all. Just over half (57 percent) of the 23 methodological elements were considered actionable.

“This lack of common standards impacts multiple stakeholders in the health care landscape,” said Jennifer Graff, PharmD, NPC’s vice president for comparative effectiveness research. “Understanding how treatments work in the real-world and among typical patients requires high-quality evidence. 

“Without a common set of standards, it’s problematic for funders to know which research proposals to fund. Researchers struggle to understand which methods to follow. Journal editors and reviewers have difficulty publishing high-quality evidence. Most important, patients, providers and health plans are challenged to identify which research to use when making treatment and coverage decisions,” Dr. Graff said.

To create a consensus-based approach and engage stakeholders from different disciplines, the authors made four key recommendations:

• Gain Alignment and Consensus—While audiences and research focuses may differ among disciplines and types of stakeholders, good practices in conducting observational studies should not. 
• Identify the Level of Consensus—Minimum standards and guidelines may serve better than best practices – this approach would allow stakeholders to develop new standards and guidelines that would raise the bar for study conduct gradually over time. 
• Accomplish Consensus—Gaining consensus would likely need to occur via multiple and periodic meetings among stakeholders.
• Encourage or Enforce—While enforcement of minimum requirements would be needed, the broadest adoption of common standards would most likely be achieved with voluntary approaches. 

“The importance of real-world data in a learning health care system, where better outcomes and value is the goal, is clear. But as the appetite for credible research grows, stakeholders will feel this pain due to the lack of common standards and guidelines with increasing acuity,” said Dr. Graff. “We hope our work will inform a constructive dialogue and initiate meaningful steps to close this gap and identify what ‘good’ looks like.”  

About the National Pharmaceutical Council
The National Pharmaceutical Council is a health policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. Founded in 1953 and supported by the nation’s major research-based pharmaceutical companies, NPC focuses on research development, information dissemination, and education on the critical issues of evidence, innovation and the value of medicines for patients. For more information, visit www.npcnow.org and follow NPC on Twitter @npcnow.

About the University of Pittsburgh Graduate School of Public Health
The University of Pittsburgh Graduate School of Public Health, founded in 1948 and now one of the top-ranked schools of public health in the United States, conducts research on public health and medical care that improves the lives of millions of people around the world. Pitt Public Health is a leader in devising new methods to prevent and treat cardiovascular diseases, HIV/AIDS, cancer and other important public health problems. For more information about Pitt Public Health, visit the school’s Web site at www.publichealth.pitt.edu.