NPC Public Comments on ICER 2020 Value Assessment Framework

June 10, 2019

Steven D. Pearson, MD, MSc, FRCP
President
Institute for Clinical and Economic Review
One State Street, Suite 1050
Boston, MA 02109 USA

RE: Public Input for 2020 Value Assessment Framework

Submitted electronically via: [email protected]

Dear Dr. Pearson:

The National Pharmaceutical Council (NPC) shares your interest in translating “evidence into policy decisions that lead to a more effective, efficient, and just health care system.” [1]  NPC launched the Going Below The Surface initiative [2] with this view in mind. NPC appreciates ICER’s call for public input for the 2020 Value Assessment Framework. [3]  

NPC is a health policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. NPC is supported by the major U.S. research-based biopharmaceutical companies. We focus on research development, information dissemination, education and communication of the critical issues of evidence, innovation and the value of medicines for patients. Our research helps inform critical health care policy debates and supports the achievement of the best patient outcomes in the most efficient way possible.

As stated in NPC’s Guiding Practices for Patient-Centered Value Assessment (Guiding Practices), [4] and in our prior comments on the ICER Value Assessment Framework, [5,6],  we believe value assessments can be an important tool for the complex decisions organizations and patients face when considering treatment options. Assessments that adhere to the Guiding Practices can support optimal value for patients.

ICER’s prior framework revisions [7] made changes that created greater alignment with the Guiding Practices, and we look forward to seeing even greater alignment with the 2020 revision. There are three broad areas of improvement that are needed:

• improvements to the framework itself,
• improvements to the assessment process, and
• an expanded focus to include the entire health care system.

Detailed suggestions for improvement in these three areas are presented below.
   

1. Improvements to the Framework

NPC recommends improvements to eight areas of the ICER Value Assessment Framework:

• Include societal perspective as a base case
• Use a collaborative model to achieve a realistic base case
• Increase subgroup analyses
• Include real-world evidence
• Quantitatively integrate additional benefits/contextual considerations
• Lay the groundwork to replace or augment cost-per-QALY based methodology
• Leave budget impact assessment to the end user
• Eliminate assessment of affordability and use of artificial affordability threshold

The motivation for each of these recommendations is presented below.

1. Include Societal Perspective as a Base Case

NPC’s prior comments recommended the use of a “societal perspective” for ICER’s cost-effectiveness analyses (CEA). This recommendation is echoed in the recommendations from the Second Panel on Cost-Effectiveness in Health and Medicine (Second Panel), which emphasizes the importance of the societal perspective as at least one of the base cases for CEA. [8] Guiding Practice VIII emphasizes the importance of utilizing methods based on established health economic methodologies, such as the those of the Second Panel.

The societal perspective can incorporate important factors such as productivity and caregiver burden. A societal perspective will ensure that all patient- and societal-focused benefits are included, not just those that will be accrued by the payer.

NPC recommends that ICER include the societal perspective as a co-base case in all reviews and include the results from the societal perspective base case in all result summaries such as press releases and report-at-a-glance documents.

2. Use a Collaborative Model to Achieve a Realistic Base Case

The Guiding Practices underscore the importance of ensuring that the foundation for all assessment results, the base case, is realistic (Guiding Practice X). NPC’s members have pointed to some examples of contested base case assumptions from individual reviews:

• Population. ICER may choose the entire population in its models rather than the target population identified in the label. At launch, data are only available for the target population. Additionally, the expanded population leads to an artificially high estimate of budget impact.
• Comparator. ICER may choose a placebo comparator rather than a more realistic real-world comparator. Focusing on a placebo often leads to an inflated estimate of the incremental cost per quality adjusted life year (QALY).
• Model. In the rheumatoid arthritis review, ICER chose to use a Markov model, while the majority of recently published models, included NICE’s model, used an individual patient simulation (IPS) approach. The Markov model led to significantly different results.

Achieving a realistic base case can be aided by a collaborative and transparent model development process, such as the one used by the Advisory Committee on Immunization Practices (ACIP). [9] In the ACIP process, a manufacturer’s model can be presented at the public meeting alongside an ACIP model. A collaborative process that explores different base cases can promote consensus on realistic base case assumptions.

NPC recommends that ICER pilot a collaborative and transparent model development process like the ACIP process that allows for a manufacturer’s model to be presented alongside ICER’s model, highlighting differences in the base case assumptions or modeling approach.

3. Increase Subgroup Analyses

Guiding Practice XIV speaks to the importance of recognizing that patients are heterogeneous and respond to treatments differently. Subgroup and scenario analyses should be built into the assessment process to capture this heterogeneity, including the estimation of value-based prices for the various analyses. Reporting a single value-based price for the average patient implies a false sense of precision and generalizability to the end-user. To avoid this false impression, the full range of estimated value-based prices should be reported in result summaries. Subgroup analysis should not be performed if sample size is inadequate, however.

NPC recommends that ICER include more subgroup and scenario analyses in its assessments. Value-based prices should be estimated for these analyses, and the full range of value-based prices should be included in result summaries such as press releases and report-at-a-glance documents.

4. Include Real-World Evidence

Guiding Practice XXII emphasizes that assessments should use the best available evidence. ICER’s assessments rely heavily on evidence from traditional randomized clinical trials (RCTs). Traditional RCTs are designed to answer whether a treatment can work and are not designed to answer whether treatments work in real-world populations and settings. Real-world patient data can be better suited to inform payer decision making and answer for whom the treatment works.

The evidence landscape is evolving due to higher quality real-world data sources, improved curation and analytic methods, and broader use of real-world evidence (RWE). With the growing focus on personalized medicine, smaller patient populations make RCTs harder and more expensive to conduct.

Recently, in response to provisions of the 21st Century Cures Act, the Food and Drug Administration (FDA) released the Framework for FDA’s Real-World Evidence Program as a guide for how to evaluate RWE as a way to “complement, augment and expand our understanding of how best to use medical products—improving what we know about our medical care.” [10] Just as the FDA is evolving in their use of real-world data, we encourage ICER to adopt a similar stance on broader use of real-world evidence and data, not only relying on these alternative evidence sources when high-quality RCTs are not available, or pre-emptively limiting the study type when determining study inclusion and exclusion criteria based on blunt estimates of study quality such as sample size.

NPC recognizes that RWE will not be available for many new drugs at launch, but it is often available for products that have been on the market for a while and can be useful for therapeutic area class reviews or for updated reviews. This RWE evidence can be used to improve the evidence base in ICER’s assessments. For example, a high quality prospective observational study could bridge the gap in long-term, consistent RCT evidence.

Finally, improved methods to incorporate observational studies in the ICER Evidence Rating Matrix are needed. As described in the ICER Evidence Rating Matrix: A User’s Guide case study describing leukotriene inhibitors, decision-makers would make a different comparative clinical effectiveness rating if the evidence synthesis relied solely on RCTs rather than the totality of the evidence. [11] Limited consideration of RWE studies can alter not only the magnitude of the net health benefit, but also the level of evidence certainty when RWE studies complement rather than compete with RCTs.

NPC recommends that ICER increase the use of real-world evidence in its assessments for all outcomes, utilize existing good standards for evaluation of real-world evidence, and enhance the integration of real-world evidence in evidence synthesis and rating.

5. Quantitatively Integrate Additional Benefits / Contextual Considerations

Guiding Practice XIII states that “measurement of value should include a broad array of benefits that are important to patients and society.” While ICER seeks to identify these benefits, they are merely listed in the reports as ‘Additional Considerations.’ They are not measured or formally incorporated into the assessment results.

ICER’s current approach leaves the consideration of these factors up to the discretion of the voting panel, which may not have the expertise or appropriate context to meaningfully evaluate them. Moreover, this valuation approach that is heavily dependent upon the perspectives and decisions of a small group, is not transparent or consistent. This approach is insufficient to incorporate the impact of these important patient-centered factors.

NPC recognizes that there is no gold standard methodology for quantitatively incorporating these factors into assessment results. However, it is critical that these methodologies, such as multiple-criteria decision analysis (MCDA), be developed and tested.

NPC recommends that ICER partner with researchers such as Chuck Phelps and organizations such as Center for Enhanced Value Assessment (CEVA), Pharmaceutical Value (pValue), and the Innovation and Value Initiative (IVI) to lead the way towards the piloting of a consistent and transparent methodology to quantitatively incorporate these important factors in ICER’s value assessments.

6. Lay the Groundwork to Replace or Augment Cost-Per-QALY Based Methodology

As detailed in NPC’s prior comments, use of the QALY poses several significant concerns, primarily ethical considerations, methodologic issues and disease-specific considerations. [12] ICER itself has identified key problems with the QALY. [13] These concerns serve to heighten the importance of the development and testing of alternative value assessment methodologies to replace or augment ICER’s current approach, as addressed in the preceding section.

If the QALY is used (despite the limitations noted above), it should be recognized that no single cost-per-QALY threshold can or should be universally applicable, as thresholds are likely to vary by decision-maker, population, and disease. Neumann, Cohen, and Weinstein state:

"…it is impossible to find a single threshold to represent society's willingness to pay for QALYs gained, because different approaches yield different values, each of which is based on different assumptions, inferences, and contexts. Searching for a single benchmark is at best a quixotic exercise because there is no threshold that is appropriate in all decision contexts." [14]  

Evidence exists that willingness to pay for life-saving conditions is more than that for minor conditions, [15] and even higher for rare and ultra-orphan disease. Under its “highly specialised technology” process, the National Institute for Health and Care Excellence (NICE) proposes to use cost-per-QALY thresholds for ultra-orphan diseases that are 5-10 times its standard level. [16] Willingness to pay for oncology suggests thresholds that are much higher than what ICER currently uses as an upper bound. [17,18] Some real-world coverage decisions in the U.S. are similarly consistent with higher WTP thresholds. These signals of potentially higher thresholds in the U.S. should not be ignored, especially given the absence of a multi-stakeholder evaluation process to determine true societal WTP in the U.S.

The reality is that we do not know what society’s willingness to pay (WTP) is in the U.S. for various diseases and scenarios. Guiding Practice XIX recommends a multi-stakeholder evaluation process reflecting societal values be used to set specific thresholds, and this has not occurred.

NPC recommends that this uncertainty about U.S. WTP for various diseases and scenarios be addressed in the short run by using higher upper bound cost-per-QALY thresholds, and in the long run by developing and testing alternative value assessment methodologies to replace or augment the cost-per-QALY based methodology.  

7. Leave Budget Impact Assessment to the End User

Budget Impact Guiding Practice II states that “budget impact assessments should be separate from value assessments.” Budget impact assessment (BIA) is a measure of resource use, not a measure of value, and it has no role in value assessment.
NPC recognizes that budget impact is an important construct for individual payers in their decision-making process. Payers should estimate budget impact for their own populations. ICER’s estimates of national budget impact are not relevant to these payers or useful for their decision-making purposes.

NPC recommends that ICER’s value assessments should focus solely on value. The estimation of national budget impact should be eliminated from assessments, leaving budget impact estimation to the end user.

8. Eliminate Assessment of Affordability and Use of Artificial Affordability Threshold

Budget Impact Guiding Practice VI says that “a BIA is simply an assessment of budget impact, and should not be judged against artificial affordability caps.” Affordability is an important concept for society. Evaluating affordability involves making assessments and trade-offs at an overall health system level (i.e., a broad assessment of all investments in a health care system) and beyond the health system (i.e., spending on health care versus other societal considerations, such as education, police, and roads).

A comprehensive approach to affordability requires considerations of concepts such as disinvestment and tradeoffs, needs to be informed by cultural and societal values as well as health and non-health needs, and requires broad stakeholder involvement. ICER’s current approach to assessing affordability—setting a potential budget impact threshold that may trigger an “affordability alert”—is not a comprehensive consideration of the health care system, does not consider societal values, and does not adequately measure affordability.

Not only would an affordability assessment require decisions about health care spending versus non-health care spending, it also would require societal decisions about intra-health care spending. This would force decision-makers to make trade-offs regarding spending on the elderly versus the young, rare disorders versus common diseases, and curative therapies versus prolonging life or quality-of-life enhancement, as well as allocations among medications, surgery, hospital care, and physician services.

ICER’s current approach of setting a uniform budget impact threshold based on a fixed portion of drug expenditures creates an artificial affordability threshold that could have negative, unintended consequences. An analysis by IQVIA (formerly IMS) and NPC demonstrates that ICER’s affordability threshold could lead to an inefficient allocation of health care resources. [19] If the affordability threshold—which is not based on value—causes us to shift this spending away from high-value drugs, it could be shifting money toward lower-value care that is less efficient, paradoxically reducing the value of our health care dollar.

Another unintended consequence of an artificial affordability threshold is the disincentivization of the development of drugs for broad populations with unmet need. Predicted budget impact will increase as the predicted number of patients increases, causing a treatment for a broad population—particularly one with unmet need—to be more likely to trigger an “affordability alert” threshold. However, a comprehensive affordability assessment that considers societal values and the broader public health perspective would likely result in a higher spending allocation for such a treatment.

Finally, the ICER threshold equation assumes that the allocation of health care spending among drugs, hospital care, imaging and physician care is the “correct” allocation across resources. For example, perhaps more resources should be spent on drugs and less on imaging for optimal resource allocation, or vice versa. The derived threshold assumes that the current allocation is optimal, an unproven assumption that is likely incorrect.

NPC recommends that ICER eliminate its assessment of affordability as well as its use of an artificial affordability threshold.

2. Improvements to the Assessment Process

NPC recommends improvements to four areas of the ICER Value Assessment Process:

• Include broader results in press releases and report-at-a-glance documents
• Enable full transparency and reproducibility by making the ICER model publicly available
• Extend length of time for review
• Establish a disease-specific working group of clinicians for each assessment

More details about these recommendations are presented below.

1. Include Broader Results in Press Releases and Report-at-a-Glance Documents

As noted above in section I.C., reporting a single value-based price (VBP) for the average patient implies a false sense of precision to the end-user. Although ICER’s VBP results appear to have a range, this range relates solely to the use of different thresholds. For any given threshold, however, there is the implication that the VBP point estimate is accurate. There is uncertainty around this estimate due to uncertainty in the data and model assumptions, and ICER should provide ranges around each of these point estimates to indicate this uncertainty.

This uncertainty extends beyond the VBP estimates for the base case. There are different estimates for each analysis beyond the base case, such as those with different subgroups, different scenarios, and different perspectives (such as the societal perspective in section I.A.). The VBP estimates and confidence intervals for these additional analyses should be reported in result summaries so the full extent of uncertainty is recognized by the end user.

NPC recommends that ICER include broader results in summaries such as press releases and report-at-a-glance documents. These results should include the societal perspective as a co-base case and a full range of potential value-based prices, and the confidence intervals around these prices.

2. Enable Full Transparency and Reproducibility by Making the ICER Model Publicly Available

Guiding Practice IX emphasizes that transparency and reproducibility are necessary for demonstrating credibility and validity of assessments. NPC’s prior comments have highlighted the lack of transparency and reproducibility inherent in ICER’s models.
NPC commends ICER for introducing a pilot program to share models with manufacturers in 2018. [20] While this was a clear step in the right direction, we agree with the limitations to ICER’s approach that have been noted by others: models should be available to all stakeholders rather than subject to restricted access; models should be fully available for use and customization rather than only available for review; and model sharing should not include confidentiality agreements that restrict the ability to share and discuss the models freely with all stakeholders. [21] We also recommend conducting live Q&A sessions to discuss the assessment model, where stakeholders can ask clarifying questions to the model developers until the model is explained to the extent that allows accurate reproduction.

NPC strongly recommends and underscores the need for full access and transparency—down to the equation level—to enable reproducible results and support fully informed stakeholder collaboration.  

3. Extend Length of Time for Review

Guiding Practice IV notes that public comment periods need to be long enough to allow for comprehensive review of materials and submission of comments. NPC acknowledges that ICER has previously extended the time for stakeholders to submit comments on scoping documents and reports, but the amount of time is still far too short for most stakeholders and is inconsistent with timelines used by other HTA bodies and the government.

Patient groups have reported difficulty with reviewing assessment reports, identifying key issues and concerns, and developing constructive comments in such a limited amount of time. Greater effort needs to be made to meaningfully elicit the patient’s voice proactively and throughout the assessment process.

NPC recommends that additional time should be included for meaningful review and feedback by all interested stakeholders.

4. Establish a Disease-Specific Working Group of Clinicians for Each Assessment

Individual reviews are lacking in guidance from sufficient disease-specific clinical expertise. This could be addressed by mirroring the ACIP process noted above in I.B. For each drug evaluation, ICER could convene a working group of 8-12 clinicians with expertise in the disease or therapeutic area under review. This working group would provide clinical guidance on the selection of comparators, model parameters, and evidence to include or exclude, and they would benefit from seeing both ICER and industry models as they make these decisions. Since these choices are critical to the final assessment results, having a working group of clinical experts to provide direction will improve the credibility and accuracy of the results.

NPC recommends that ICER convene a working group of clinicians with disease-specific expertise for each review. Similar to the ACIP process, this working group would see both ICER and industry models and provide direction on important choices such as selection of comparators, model parameters, and evidence base.

5. Expanded Focus on the Entire Health Care System

Guiding Practice VII states that “value assessments should focus broadly on all aspects of the health care system, not just on medications.” This point is critical to the achievement of ICER and NPC’s shared goal, noted in this letter’s first sentence: a more effective, efficient, and just health care system. Drug spending accounts for only 16% of the U.S. health care dollar. [22] To truly achieve a more just, more efficient, and more effective health care system, value assessments must consider the other 84%, too. Since drugs represent a limited portion of the overall health care budget, ICER’s impact on the health care system would increase if its agenda was less concentrated and considered other interventions.

NPC recommends that ICER expand the focus of its assessments to the entire health system and conduct a proportionate share of value assessments for other parts of the health care system. 

***

We appreciate this opportunity to provide input on potential changes to ICER’s 2020 Value Assessment Framework. NPC’s continued engagement with ICER signifies our commitment to the critical dialogue necessary to ensure the development of high-quality, meaningful value assessment tools that help patients, physicians, payers, and others make informed decisions about all aspects of their health care treatments and services.

Respectfully submitted,
 
Robert W. Dubois, MD, PhD
Chief Science Officer
 

Kimberly Westrich, MA
Vice President, Health Services Research

 

 References

[1] About ICER. https://icer-review.org/about/. Institute for Clinical and Economic Review. Accessed May 3, 2019.

[2] Going Below the Surface. https://www.goingbelowthesurface.org/. Going Below the Surface. Accessed May 27, 2019.

[3] ICER Seeks Public Input for 2020 Value Assessment Framework. Institute for Clinical and Economic Review. https://icer-review.org/announcements/icer-seeks-public-input-for-2020-value-assessment-framework/. May 2, 2019. Accessed May 3, 2019.

[4]Guiding Practices for Patient-Centered Value Assessment. National Pharmaceutical Council. http://www.npcnow.org/sites/default/files/npc-guiding-practices-for-patient-centered-value-assessment.pdf. Accessed May 3, 2019.

[5] NPC Comments on Proposed Improvements to ICER’s Value Assessment Framework. National Pharmaceutical Council. http://www.npcnow.org/sites/default/files/160909npc-comments-ICER-value-framework-final.pdf. September 12, 2019. Accessed May 3, 2019.

[6] NPC Comments on Proposed Updates to ICER’s Value Assessment Framework. National Pharmaceutical Council. https://www.npcnow.org/newsroom/commentary/npc-comments-proposed-updates-icers-value-assessment-framework. Accessed May 3, 2019.

[7] Overview of the ICER value assessment framework and update for 2017-2019. Institute for Clinical and Economic Review. http://icer-review.org/wp-content/uploads/2017/06/ICER-value-assessment-framework-Updated-050818.pdf. Accessed May 26, 2019.

[8] Sanders GD, Neumann PJ, Basu A, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses. JAMA. 2016;316():1093-1103. doi:10.1001/jama.2016.12195.

[9] Advisory Committee on Immunization Practices (ACIP). Centers for Disease Control and Prevention. https://www.cdc.gov/vaccines/acip/index.html. Accessed May 27, 2019.

[10] Framework for FDA’s Real-World Evidence Program. U.S. Food & Drug Administration. https://www.fda.gov/media/120060/download. December 2018. Accessed June 6, 2019.

[11] ICER Evidence Rating Matrix: A User’s Guide. http://icer-review.org/wp-content/uploads/2016/02/Rating-Matrix-User-Guide-UPDATED-06.30.17.pdf. Institute for Clinical and Economic Review. July 18, 2017. Accessed June 5, 2019.

[12] Pettitt DA, Raza S, Naughton B, et al. The Limitations of QALY: A Literature Review. J Stem Cell Res Ther. 2016;6(4) 1-7. http://www.omicsonline.org/open-access/the-limitations-of-qaly-a-literature-review-2157-7633-1000334.pdf. Accessed May 26, 2019.

[13] Pearson SD. A Framework to Guide Payer Assessment of the Value of Medical Treatments. Institute for Clinical and Economic Review. http://icer-review.org/wp-content/uploads/2014/01/Value-Assessment-Framework-DRAFT-6-13-14.pdf. June 12, 2014. Accessed May 26, 2019.

[14] Neumann PJ, Cohen JT, Weinstein MC. Updating Cost-Effectiveness — The Curious Resilience of the $50,000-per-QALY Threshold. N Engl J Med 2014;371:796-797. doi: 10.1056/NEJMp1405158.

[15] Gross SD. Assessing Cost-Effectiveness in Healthcare: History of the $50,000 per QALY Threshold. Expert Rev. Pharmacoeconomics Outcomes Res. 2008;8(2), 165-178.

[16] Raftery J. NICE’s proposed new QALY modifier for appraising highly specialised technologies The BMJ Opinion. http://blogs.bmj.com/bmj/2017/04/18/nices-proposed-new-qaly-modifier-for-appraising-highly-specialised-technologies/. April 17, 2018. Accessed June 6, 2019.

[17] Nadler E, Eckert B, Neumann P. Do Oncologists Believe New Cancer Drugs Offer Good Value? The Oncologist. 2006 Feb;11(2):90-5. doi: 10.1634/theoncologist.11-2-90.

[18] Seabury SA, Goldman DP, Maclean JR, et al. Patients Value Metastatic Cancer Therapy More Highly Than Is Typically Shown Through Traditional Estimates Health Affairs. 2012 Apr;31(4):691-699.

[19] Ciarametaro M, Abedi S, Sohn A, Fan Ge C, Odedara N, Dubois RW. Concerns around budget impact thresholds: not all drugs are the same. Value in Health. 2017 Feb;20(2):230-233. doi: 10.1016/j.jval.2016.11.031.

[20] Institute for Clinical and Economic Review Announces New Program to Make Available Draft Executable Economic Models During Drug Assessment Review Process. Institute for Clinical and Economic Review. http://icer-review.org/announcements/model-transparency-program. March 30, 2018. Accessed May 26, 2019.

[21] IVI responds to ICER’s new pilot program. Innovation and Value Initiative. http://www.thevalueinitiative.org/ivi-responds-to-icers-new-pilot-program. April 3, 2018. Accessed May 26, 2019.

[22] Kleinrock M, et al. Reconciling the Seemingly Irreconcilable: How Much Are We Spending on Drugs? Value in Health. https://www.valueinhealthjournal.com/article/S1098-3015(18)36310-1/fulltext