NPC Comments on Proposed Changes for ICER 2020 Value Assessment Framework

NPC logo

October 18, 2019

Steven D. Pearson, MD, MSc, FRCP
President
Institute for Clinical and Economic Review
One State Street, Suite 1050
Boston, MA 02109 USA

RE: Proposed Changes for 2020 Value Assessment Framework

Submitted electronically via: [email protected]

Dear Dr. Pearson:

The National Pharmaceutical Council (NPC) appreciates ICER’s call for public comment on the proposed updates to the 2020 Value Assessment Framework methods and procedures.[1]  NPC especially appreciates the addition of language to ICER’s conceptual structure that emphasizes “Fair Access” and “Future Innovation.” In the past, the media reporting and payer use of ICER’s reviews has focused almost exclusively on the “Fair Price” piece of ICER’s conceptual structure, not recognizing that the structure is a three-legged stool: without payer commitment to fair access, future innovation will suffer. When innovation suffers, so do patients, health outcomes and society.

NPC is a health policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. NPC is supported by the major U.S. research-based biopharmaceutical companies. We focus on research, development, information dissemination, education and communication of the critical issues of evidence, innovation and the value of medicines for patients. Our research helps inform critical health care policy debates and supports the achievement of the best patient outcomes in the most efficient way possible.

As stated in NPC’s Guiding Practices for Patient-Centered Value Assessment (Guiding Practices),[2] and in our June 10, 2019 comments on changes needed to ICER’s 2020 Value Assessment Framework,[3] we believe value assessments can be an important tool for the complex decisions organizations and patients face when considering treatment options. Assessments that adhere to the Guiding Practices can support optimal value for patients.

Although ICER’s proposed updates provide several incremental improvements toward aligning with the Guiding Practices, further revision and refinement of the framework is necessary to truly support optimal value for patients. There are three broad areas of improvement that are needed: improvements to the framework itself, improvements to the assessment process, and an expanded focus to include the entire health care system.

Detailed suggestions for improvement in these three areas are presented below.

I.    Improvements to the Framework

NPC’s June 10, 2019 comments called for improvements to eight areas of the ICER Value Assessment Framework:

  • Include societal perspective as a base case
  • Use a collaborative model to achieve a realistic co-base case
  • Increase subgroup analyses
  • Include real-world evidence
  • Quantitatively integrate additional benefits/contextual considerations
  • Lay the groundwork to replace or augment cost-per-quality-adjusted life year (QALY)-based methodology
  • Leave budget impact assessment to the end user
  • Eliminate assessment of affordability and use of artificial affordability threshold

ICER acknowledged six of these areas in the proposed updates to the 2020 framework and made changes in four of them. NPC believes more substantial changes are needed in all eight areas and has added a ninth area: “Do not cross-reference with German Evidence Ratings.”

A.    Recommendation: Include Societal Perspective as a Co-Base Case

ICER acknowledged this recommendation and declined to include the societal perspective as a co-base case. ICER wrote, “Decision-makers in the U.S. health care system are not responsible for making trade-off decisions that involve broader societal resources.” NPC strongly disagrees with this statement; states, employers and other payers do indeed face such trade-offs and broader societal decisions.

NPC’s prior comments recommended the use of a “societal perspective” for ICER’s cost-effectiveness analyses (CEA). This recommendation is echoed in the recommendations from the Second Panel on Cost-Effectiveness in Health and Medicine (Second Panel).[4]  NPC’s Guiding Practice VIII emphasizes the importance of utilizing methods based on established health economic methodologies, such as the those of the Second Panel. The societal perspective can incorporate important factors such as productivity and caregiver burden. A societal perspective will ensure that all patient- and societal-focused benefits are included, not just those that will be accrued by the payer.

ICER has stated it will include a societal perspective as a co-base case when the “societal costs of care for any disease are large relative to the direct health care costs.” ICER should clarify the criteria for this as there is overlap between treatments for ultra-rare conditions, “Single or Short-term Transformative Therapies (SSTs)” and traditional therapies.

NPC recommends that ICER include the societal perspective as a co-base case in all reviews and include the results from the societal perspective co-base case in all result summaries, such as press releases and report-at-a-glance documents.

B.    Recommendation: Use a Collaborative Model to Achieve a Realistic Base Case

ICER did not acknowledge this recommendation explicitly, but is proposing to add a new section on “Controversies and Uncertainties” to “expand discussion of any alternative model structures or inputs suggested by manufacturers or other stakeholders that differ importantly from the base case.” NPC supports the addition of this section but urges ICER to go further than merely expanding discussion of alternative models. Qualitative discussion of alternative models is not enough; a quantitative exposition of the way cost-effectiveness results change under these models is needed, too.

The Guiding Practices underscore the importance of ensuring that the foundation for all assessment results, the base case, is realistic (Guiding Practice X). Achieving a realistic base case can be aided by a collaborative and transparent model development process, such as the one used by the Advisory Committee on Immunization Practices (ACIP).[5] In the ACIP process, a manufacturer’s model can be presented at the public meeting alongside an ACIP model. A collaborative process that explores different base cases can promote consensus on realistic base case assumptions and the resultant range of model results.

NPC recommends that ICER pilot a collaborative and transparent model development process like the ACIP process that allows for a manufacturer’s model to be presented alongside ICER’s model, highlighting differences in the base case assumptions, modeling approach and results. These differences should be included in press releases and report-at-a-glance documents.

C.    Recommendation: Increase Subgroup Analyses

ICER acknowledged this recommendation but declined to make any changes to subgroup analysis. NPC believes the need for appropriate subgroup analyses continues to be important.

Guiding Practice XIV speaks to the importance of recognizing that patients are heterogeneous and respond to treatments differently. Subgroup and scenario analyses should be built into the assessment process to capture this heterogeneity, including the estimation of value-based prices (VBPs) for the various analyses. At a minimum, ICER should describe the heterogeneity of the population and estimate what proportion of the disease population are represented in their analyses to allow transparency in understanding where results would and would not apply.  Reporting a single value-based price for the average patient implies a false sense of precision and generalizability to the end-user. To avoid this false impression, the full range of estimated value-based prices should be reported in result summaries. Subgroup analysis should not be performed if sample size is inadequate, however.

NPC recommends that ICER include more subgroup and scenario analyses in its assessments. VBPs should be estimated for these analyses, and the full range of VBPs should be included in result summaries such as press releases and report-at-a-glance documents.

D.    Recommendation: Include Real-World Evidence (RWE)

NPC is pleased that ICER has acknowledged this recommendation and reaffirmed its commitment to using RWE. NPC will be watching to see the extent to which this commitment translates into action, particularly when it comes to instances where RWE is discarded in favor of the exclusive use of randomized clinical trials (RCTs).

Guiding Practice XXII emphasizes that assessments should use the best available evidence. ICER’s assessments rely heavily on evidence from traditional RCTs. Traditional RCTs are designed to answer whether a treatment can work and are not designed to answer whether treatments work in real-world populations and settings. Real-world patient data can supplement RCT data to inform payer decision-making and answer for whom the treatment works.

NPC recognizes that RWE will not be available for many new drugs at launch, but it is often available for products that have been on the market for a while and can be useful for therapeutic area class reviews or for updated reviews. This RWE evidence can be used to improve the evidence base in ICER’s assessments. For example, a high-quality prospective observational study could bridge the gap in long-term, consistent RCT evidence. Consideration of RWE studies can alter not only the magnitude of the net health benefit, but also the level of evidence certainty when RWE studies complement, rather than compete, with RCTs.

ICER also stated in its proposed updates it will “seek opportunities to generate new RWE for incorporation in reviews.” NPC does not believe that ICER should undertake the generation of RWE. ICER does not have the resources to expand into this arena. While ICER should encourage others to conduct high-quality RWE that fills gaps in the knowledge base, we do not believe this is an appropriate role for ICER.

NPC recommends that ICER increase the use of real-world evidence in its assessments for all outcomes, utilize existing good standards for evaluation of real-world evidence, and enhance the integration of real-world evidence in evidence synthesis and rating. NPC recommends against ICER generation of RWE.

E.    Recommendation: Quantitatively Integrate Additional Benefits/Contextual Considerations

ICER acknowledged this recommendation but declined to quantitatively incorporate additional dimensions of value. Guiding Practice XIII states that “measurement of value should include a broad array of benefits that are important to patients and society.” While ICER seeks to identify these benefits, they are merely listed in the reports as “Additional Considerations.” They are not quantitatively incorporated into the assessment results.

ICER’s current approach leaves the consideration of these factors up to the discretion of the voting panel, which may not have the expertise or appropriate context to meaningfully evaluate them. This approach is insufficient and does not quantitatively incorporate the impact of these important patient-centered factors. NPC recognizes that there is no gold standard methodology for quantitatively incorporating these factors into assessment results. However, it is critical that these methodologies, such as multiple-criteria decision analysis (MCDA), be developed and tested.

ICER noted it attempted to pilot MCDA in 2009-2010 and found it difficult to use. NPC urges ICER to pilot MCDA in its 2020 assessment process. MCDA methods have evolved over the past decade, and the time period in which ICER piloted MCDA pre-dates ICER’s value assessment framework by many years.

NPC recommends that ICER partner with researchers at organizations such as the Center for Enhanced Value Assessment (CEVA), Pharmaceutical Value (pValue), and the Innovation and Value Initiative (IVI) to pilot a consistent and transparent methodology to quantitatively incorporate these important factors in ICER’s value assessments.

F.    Recommendation: Lay the Groundwork to Replace or Augment Cost-Per-QALY Based Methodology

As detailed in NPC’s prior comments, use of the QALY poses several significant concerns, ethical considerations, methodologic issues and disease-specific considerations.[6] ICER itself has identified key problems with the QALY.[7] These concerns serve to heighten the importance of the development and testing of alternative value assessment methodologies to replace or augment ICER’s current approach, as addressed in the preceding section.

If the QALY is used (despite the limitations noted above), it should be recognized that no single cost-per-QALY threshold can or should be universally applicable, as thresholds are likely to vary by payer, population and disease.[8] Evidence exists that willingness to pay for life-saving conditions is more than that for minor conditions,[9] and even higher for rare and ultra-orphan disease. Under its “highly specialised technology” process, the National Institute for Health and Care Excellence (NICE) proposes to use cost-per-QALY thresholds for ultra-orphan diseases that are 5-10 times its standard level.[10] Willingness to pay for oncology suggests thresholds that are much higher than what ICER currently uses as an upper bound. [11,12] Some real-world coverage decisions in the U.S. are similarly consistent with higher willingness to pay (WTP) thresholds.

These signals of potentially higher or variable thresholds in the U.S. should not be ignored, especially given the absence of a multi-stakeholder evaluation process to determine true societal WTP in the U.S. The reality is that we do not know what society’s WTP is in the U.S. for various diseases and scenarios. Guiding Practice XIX recommends a multi-stakeholder evaluation process reflecting societal values be used to set specific thresholds, and this has not occurred.

ICER acknowledged this recommendation and raised the upper bound in the core framework to $200,000 per QALY. However, ICER lowered the upper bound for ultra-rare disorders from $500,000 per QALY to $200,000 per QALY. While NPC supports an increase of the upper bound in the core framework, it is disappointing that the upper bound will not be used for value-based price benchmarks and will not be presented in report-at-a-glance summaries. It is even more disappointing that the upper bound for ultra-rare disorders has been dramatically lowered, given the evidence presented above that there is higher WTP for such conditions.

NPC recommends that this uncertainty about U.S. WTP for various diseases and scenarios be addressed in the short run by using a wide range of cost-per-QALY thresholds in analyses and including the results from the full range in press releases and report-at-a-glance documents. In the long run, this should be done by developing and testing alternative value assessment methodologies to replace or augment the cost-per-QALY based methodology. NPC further recommends that the upper bound threshold for ultra-rare diseases remain (at least) at $500,000 per QALY.  

G.    Recommendation: Leave Budget Impact Assessment to the End User

ICER acknowledged this recommendation but declined to drop the estimation of national budget impact from its assessments.  Budget Impact Guiding Practice II states that “budget impact assessments should be separate from value assessments.” Budget impact assessment (BIA) is a measure of resource use, not a measure of value, and it has no role in value assessment.
NPC recognizes that budget impact is an important construct for individual payers in their decision-making process. ICER’s estimates of national budget impact, however, are not relevant to these payers or useful for their decision-making purposes. Payers should estimate budget impact for their own populations, which are smaller and different than the overall U.S. population.

ICER proposed the addition of per-patient budget impact estimates, which are more useful to a payer seeking to use an ICER report than a hypothetical estimate of national budget impact. However, the addition of this per-patient budget impact highlights the false sense of precision that is inherent in ICER reviews, as mentioned above in section I.C. Cost per patient will vary by payer, both broadly (e.g., Medicaid, Medicare, employers) and narrowly (e.g., different commercial plans). While NPC believes a per-patient budget impact estimate is preferable to a national budget impact, if included, this estimate should include ranges so the individual payer can customize their own estimation of budget impact.

NPC recommends that ICER’s value assessments should focus solely on value. The estimation of national budget impact should be eliminated from assessments, leaving budget impact estimation to the end user. If included, per-patient budget estimates should include ranges to accommodate the variability across payers who will use these estimates.

H.    Recommendation: Eliminate Assessment of Affordability and Use of Artificial Affordability Threshold

ICER did not acknowledge this recommendation in its proposed updates. NPC strongly believes the estimation of affordability and use of an artificial affordability threshold is inappropriate and has unintended consequences.

Budget Impact Guiding Practice VI says that “a BIA is simply an assessment of budget impact and should not be judged against artificial affordability caps.” Affordability is an important concept for society. Evaluating affordability involves making assessments and trade-offs at an overall health system level (i.e., a broad assessment of all investments in a health care system) and beyond the health system (i.e., spending on health care versus other societal considerations, such as education, police and roads).
A comprehensive approach to affordability requires considerations of concepts such as disinvestment and tradeoffs, needs to be informed by cultural and societal values as well as health and non-health needs, and requires broad stakeholder involvement. ICER’s current approach to assessing affordability—setting a potential national budget impact threshold that may trigger an “affordability alert”—is not a comprehensive consideration of the health care system, does not consider societal values, and does not adequately measure affordability.

ICER’s current approach of setting a uniform budget impact threshold based on a fixed portion of drug expenditures creates an artificial affordability threshold that could have negative, unintended consequences. An analysis by IQVIA and NPC demonstrates that ICER’s affordability threshold could lead to an inefficient allocation of health care resources.[13] If the affordability threshold—which is not based on value—causes us to shift this spending away from high-value drugs, it could be shifting money toward lower-value care that is less efficient, paradoxically reducing the value of our health care dollar.

Another unintended consequence of an artificial affordability threshold is the disincentivization of the development of drugs for broad populations with unmet need. Predicted budget impact will increase as the predicted number of patients increases, causing a treatment for a broad population—particularly one with unmet need—to be more likely to trigger an “affordability alert” threshold. However, a comprehensive affordability assessment that considers societal values and the broader public health perspective would likely result in a higher spending allocation for such a treatment.

NPC recommends that ICER eliminate its assessment of affordability and its use of an artificial affordability threshold.

I.    Recommendation: Do Not Cross-Reference with German Evidence Ratings

A new issue that has emerged with the 2020 proposed updates is ICER’s proposition to introduce evidence ratings designed to crosswalk to the German categories of “added benefit.” Guiding Practice XXVII cautions against misuse of value assessments by an unintended audience. ICER notes the many important distinctions between the methods used by ICER and the German system, which all serve to underscore why such a cross-reference could be misleading and is ill-advised. Drug comparisons in Germany occur in an environment that is not analogous to the U.S. due to differences in factors such as population, treatment options and comparators, endpoints, treatment guidelines, and health care resource utilization.

ICER’s creation of a “rough algorithm” that relies on different methodologies and assumptions from the German system yet provides output in the same format, is described as a way to “spur further dialogue and calibration of evidence assessments across important pharmaceutical markets.” However, translating ICER’s assessments into the German categories in an approximated manner, laden with implicit assumptions, creates the opportunity for far more than dialogue: it creates the opportunity for misuse by creating the false impression that this output could be used in the German market.

NPC strongly recommends that ICER reconsider this proposal and drop the cross-reference with German Evidence Ratings from its 2020 Value Assessment Framework update.

II.     Improvements to the Assessment Process

NPC’s June 10, 2019 comments called for improvements to four areas of the ICER Value Assessment Process:

  • Include broader results in press releases and report-at-a-glance documents
  • Enable full transparency and reproducibility by making the ICER models and inputs publicly available
  • Extend length of time for review
  • Establish a disease-specific working group of clinicians for each assessment

ICER acknowledged and made changes to two of these areas in the proposed updates to the 2020 framework. NPC believes more substantial changes are needed in all four areas.

A.    Recommendation: Include Broader Results in Press Releases and Report-at-a-Glance Documents

ICER did not acknowledge this recommendation in its proposed updates. NPC believes it is critical that the uncertainty and full range of results are highlighted in summary documents so that end users will not have a false sense of the precision of ICER’s estimates and hence misuse them.

As noted above in section I.C., reporting a single value-based price (VBP) for the average patient implies a false sense of precision to the end-user. Although ICER’s VBP results appear to have a range when reported in summary documents, this range relates solely to the use of different thresholds. For any given threshold, however, there is the implication that the VBP point estimate is accurate. There is uncertainty around the VBP estimate due to uncertainty in the data and model assumptions, and ICER should provide ranges around each of these point estimates to indicate this uncertainty.

This uncertainty also extends beyond the VBP estimates for the base case. There are different estimates for each analysis beyond the base case, such as those with different subgroups, different scenarios, and different perspectives (such as the societal perspective in section I.A.). The VBP estimates and confidence intervals for these additional analyses should be reported in result summaries so the full extent of uncertainty is recognized by the end user and considered in their decision making.

NPC recommends that ICER include broader results in summaries such as press releases and report-at-a-glance documents. These results should include the societal perspective as a co-base case and a full range of potential VBPs, as well as the confidence intervals around these prices.

B.    Recommendation: Enable Full Transparency and Reproducibility by Making the ICER Models and Inputs Publicly Available  

Guiding Practice IX emphasizes that transparency and reproducibility are necessary for demonstrating credibility and validity of assessments. NPC’s prior comments have highlighted the lack of transparency and reproducibility inherent in ICER’s models.

NPC commends ICER for introducing a pilot program to share models with manufacturers in 2018.[14] While this was a clear step in the right direction, we agree with the limitations to ICER’s approach that have been noted by others: models should be available to all stakeholders rather than subject to restricted access; models should be fully available for use and customization rather than only available for review; and model sharing should not include confidentiality agreements that restrict the ability to share and discuss the models freely with all stakeholders.[15] We also recommend conducting live Q&A sessions to discuss the assessment model, where stakeholders can ask clarifying questions to the model developers until the model is explained to the extent that allows accurate reproduction.

ICER acknowledged this recommendation but is making minimal change beyond expanding the pilot program to now include all ICER reviews. While it is a positive that manufacturers will have broader model access for all reviews, the limitations noted in the above paragraph remain. All stakeholders should have unrestricted access to the models.

NPC strongly recommends and underscores the need for full access and transparency—down to the equation level—to enable reproducible results and support fully informed stakeholder collaboration.  

C.    Recommendation: Extend Length of Time for Review

ICER acknowledged this recommendation and is extending the review time for large class reviews by nine weeks. NPC appreciates this extension but notes that review time continues to be inconsistent with timelines used by the government and other health technology assessment (HTA) bodies. Guiding Practice IV notes that public comment periods need to be long enough to allow for comprehensive review of materials and submission of comments.

Patient groups have reported difficulty with reviewing assessment reports, identifying key issues and concerns and developing constructive comments in such a limited amount of time. Greater effort needs to be made to meaningfully elicit the patient’s voice proactively and throughout the assessment process.

NPC recommends that additional time should be included for meaningful review and feedback by all interested stakeholders, consistent with that allowed by other HTA bodies and the government.

D.    Recommendation: Establish a Disease-Specific Working Group of Clinicians for Each Assessment

ICER did not acknowledge this recommendation. NPC urges ICER to reconsider the need for a disease-specific working group of clinicians for each assessment. Individual reviews are lacking in guidance from sufficient disease-specific clinical expertise.

This could be addressed by mirroring the ACIP process noted above in section I.B. For each drug evaluation, ICER could convene a working group of 8-12 clinicians with expertise in the disease or therapeutic area under review. This working group would provide clinical guidance on the selection of comparators, model parameters and evidence to include or exclude, and they would benefit from seeing both ICER and industry models as they make these decisions. Since these choices are critical to the final assessment results, having a working group of clinical experts to provide direction will improve the credibility and accuracy of the results.

NPC recommends that ICER convene a working group of clinicians with disease-specific expertise for each review. Similar to the ACIP process, this working group would see both ICER and industry models and provide direction on important choices such as selection of comparators, model parameters and evidence base.

III.    Expanded Focus on the Entire Health Care System

ICER did not acknowledge this recommendation. NPC believes this is an oversight that does the U.S. health system a grave disservice. Guiding Practice VII states that “value assessments should focus broadly on all aspects of the health care system, not just on medications.” This point is critical to the achievement of ICER’s goal of “a more effective, efficient, and just health care system.”

Drug spending accounts for only 16% of the U.S. health care dollar.[17] To truly achieve a more just, more efficient and more effective health care system, value assessments must consider the other 84%, too. Since drugs represent a limited portion of the overall health care budget, ICER’s impact on the health care system would be magnified if its agenda was less concentrated and considered other interventions proportionately.

NPC recommends that ICER expand the focus of its assessments to the entire health system and conduct a proportionate share of value assessments for other parts of the health care system.  

***

We appreciate this opportunity to provide input on proposed changes to ICER’s 2020 Value Assessment Framework. NPC’s continued engagement with ICER signifies our commitment to the critical dialogue necessary to ensure the development of high-quality, meaningful value assessment tools that help patients, physicians, payers and others make informed decisions about all aspects of their health care treatments and services.

Respectfully submitted,

Robert W. Dubois, MD, PhD
Chief Science Officer

Kimberly Westrich, MA
Vice President, Health Services Research

References

  1. ICER Proposes Updates to 2020 Value Assessment Framework Methods and Procedures. Institute for Clinical and Economic Review.  https://icer-review.org/announcements/2020vaf_proposed_updates/. August 21, 2019. Accessed September 22, 2019.
  2. Guiding Practices for Patient-Centered Value Assessment. National Pharmaceutical Council. http://www.npcnow.org/sites/default/files/npc-guiding-practices-for-patient-centered-value-assessment.pdf. Accessed September 22, 2019.
  3. NPC Public Comments on ICER 2020 Value Assessment Framework. National Pharmaceutical Council. https://www.npcnow.org/newsroom/commentary/npc-public-comments-icer-2020-value-assessment-framework. June 10, 2019. Accessed September 22, 2019.
  4. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses. JAMA. 2016;316():1093-1103. doi:10.1001/jama.2016.12195.
  5. Advisory Committee on Immunization Practices (ACIP). Centers for Disease Control and Prevention. https://www.cdc.gov/vaccines/acip/index.html. Accessed September 22, 2019.
  6. Pettitt DA, Raza S, Naughton B, et al. The Limitations of QALY: A Literature Review. J Stem Cell Res Ther. 2016;6(4) 1-7. Doi: 0.4172/2157-7633.1000334.
  7. Pearson SD. A Framework to Guide Payer Assessment of the Value of Medical Treatments. Institute for Clinical and Economic Review. http://icer-review.org/wp-content/uploads/2014/01/Value-Assessment-Framework-DRAFT-6-13-14.pdf. June 12, 2014. Accessed September 22, 2019.
  8. Neumann PJ, Cohen JT, Weinstein MC. Updating Cost-Effectiveness — The Curious Resilience of the $50,000-per-QALY Threshold. N Engl J Med 2014;371:796-797. doi: 10.1056/NEJMp140