NPC Comments on Proposed Updates to ICER’s Value Assessment Framework

March 30, 2017

Steven D. Pearson, MD, MSc, FRCP
President
Institute for Clinical and Economic Review
One State Street, Suite 1050
Boston, MA 02109 USA

RE: Proposed Updates to ICER’s Value Assessment Framework

Submitted electronically via: [email protected]

 

Dear Dr. Pearson:

The National Pharmaceutical Council (NPC) shares your interest in promoting a dynamic, innovative health care system and in placing scientific methods of evidence analysis and transparency at the heart of value assessment processes. With this view in mind, NPC appreciates ICER’s call for public comments on the proposed updates to the ICER Value Assessment Framework.[i]

NPC is a health policy research organization dedicated to the advancement of good evidence and science, and to fostering an environment in the United States that supports medical innovation. NPC is supported by the major U.S. research-based biopharmaceutical companies. We focus on research development, information dissemination, education and communication of the critical issues of evidence, innovation and the value of medicines for patients. Our research helps inform critical health care policy debates and supports the achievement of the best patient outcomes in the most efficient way possible.

As stated in NPC’s Guiding Practices for Patient-Centered Value Assessment (Guiding Practices),[ii] and in our September 12, 2016, comments on the ICER Value Assessment Framework,[iii] we believe value assessments can be an important tool for the complex decisions organizations and patients face when considering treatment options. Assessments that adhere to the Guiding Practices can support optimal value for patients. Although ICER’s proposed updates provide several incremental improvements toward aligning with the Guiding Practices, further revision and refinement of the framework is necessary.

The constructs most critically misaligned with the Guiding Practices are the use of:

• models that are not fully transparent and reproducible,
• an artificial budget threshold to examine affordability,
• the as-yet-untested and subjective methodologic approach to incorporating “other benefits or disadvantages and contextual considerations” into an assessment, and
• a quality-adjusted life year (QALY) threshold range that has not been agreed upon in the U.S.

Other key areas are highlighted below.

Note that our comment format is analogous to that of our September 12, 2016, comments, with sections on “Long-Term Value for Money,”  “Short-Term Affordability” and “Assessment Process.”

  

1. Long-Term Value for Money

Several of ICER’s proposed updates to estimation of “long-term value for money” are aligned with the Guiding Practices. Specifically, steps have been taken toward improved transparency and consistent use of sensitivity analysis, and net price will be used instead of WAC in cost-effectiveness analyses. However, there is an important misalignment with the Guiding Practices in the approach to quantitative incorporation of “other benefits or disadvantages and contextual considerations” and in the applicability of the QALY threshold range to the U.S. health care system.

1. Full Transparency and Reproducibility  

NPC appreciates that there is a section on “economic model transparency” in the proposed updates. ICER’s publication of its detailed analysis plan on the Open Science Framework (OSF) could be a positive step toward transparency. NPC notes that this step has actually already been implemented, and the analysis plans for the Rheumatoid Arthritis (RA), Multiple Sclerosis (MS), and Osteoporosis reviews were all published on OSF.

Unfortunately, this step was not sufficient to achieve ICER’s stated intention: “to provide enough information for an experienced researcher to be able to replicate the economic model and analyses.” Experienced researchers among our member companies have reported that they are unable to replicate the models and analyses from these three reviews. Additional steps are needed to achieve full transparency and reproducibility.

ICER previously referenced intellectual property (IP) concerns as a barrier to full transparency. IP could be protected through the use of non-disclosure agreements, such as those used by some health technology assessment (HTA) bodies, for its reviews. ICER could also institute at least one live question and answer session specifically about each assessment model and its assumptions. In this session, interested stakeholders could ask clarifying questions of the model developers until the model is explained to an extent that allows accurate reproduction.

NPC strongly recommends and underscores the need for full transparency—down to the equation level—to enable reproducible results and support fully informed stakeholder collaboration. 

2. Routine Use of Sensitivity Analyses

NPC appreciates the many areas that are identified for potential scenario/sensitivity analysis in the proposed updates, including:

• an evaluation of the heterogeneity of treatment effect for key clinical outcomes,
• the influence of lower utilities for individuals with chronic severe conditions on cost per QALY findings,
• the impact of price changes on long-term cost-effectiveness ratios when there is a high likelihood of a major change to pricing within 12-24 months of a review, and
• the incorporation of workplace productivity (which should be integrated into the numerator of the cost-effectiveness ratio).

NPC supports ICER’s concern about lower utilities among individuals with chronic severe conditions and in “seek[ing] patient input and public comment on which scenario analysis should serve as the base case within the ICER report” when these lower utilities are found to make a substantive difference. While we agree this is an important scenario for patient input, we believe that all patients should be able to provide input into their own base case.  Ideally, in a patient-centric system, individuals with any condition would provide the major voice into their own base case.

NPC’s previous comments recommended the routine use of sensitivity analyses. We note that references to scenario/sensitivity analyses in the proposed updates are accompanied by an “if feasible” qualifier. We urge ICER to drop the feasibility qualifier and routinely conduct sensitivity analysis. Some form of sensitivity analysis should always be feasible, and stakeholders will certainly help ICER identify these opportunities if they are not readily apparent to ICER.

3. Assessment Perspective

NPC’s prior comments recommended the use of a “societal perspective” for ICER’s cost-effectiveness analyses (CEA). This recommendation is echoed in the recommendations from the Second Panel on Cost-Effectiveness in Health and Medicine, which emphasizes the importance of the societal perspective as at least one of the base cases for CEA.[iv] ICER reviews, however, will continue to use only the “health system perspective” as the base case and will not include the recommended “societal perspective” as a base case.

We do note and appreciate the fact that ICER strives to include these additional societal constructs in its approach to quantitatively integrating “additional benefits or disadvantages” and “contextual considerations,” but this as-yet-untested approach is not a substitute for adopting the gold standard of including the societal perspective as a base case.

The societal perspective can allow for many additional constructs to be considered, such as “value of innovation” and “value of scientific spillover.” A societal perspective will ensure that all appropriate cost-offsets are included and not just those that will be accrued by the payer. Taking a broader societal view is particularly germane to a societal-level goal such as “sustainable access to high-value care for all patients.”

4. Integrating “Additional Benefits or Disadvantages” and “Contextual Considerations”

ICER seeks to include a broad array of factors that are important to patients and society. The March 24, 2017, Atopic Dermatitis draft evidence report includes the newly proposed presentation format for these factors. NPC supports the enhanced emphasis and clarity that these factors gain in ICER’s new presentation format.

However, ICER proposes an as-yet-untested methodology to incorporate these factors quantitatively. While NPC strongly supports the concept of including these factors in a more robust manner than the qualitative method that was used previously, we have significant concerns about the proposed method, which is not “based on established health economic methodologies, consistent with accepted standards,” as called for in the Guiding Practices.

The proposed approach is confusing even to those with an extensive health economic methodologic background, yet its implementation is left to the voting panels of ICER’s core programs: CTAF, New England CEPAC, and Midwest CEPAC. The voting panels are unlikely to have the clinical and health economic expertise or appropriate context to evaluate and combine these factors into a meaningful score for a given therapeutic area and patient population. The fact that a voting panel’s assessment would require a disclaimer indicating that ratings are from one group of people on one particular day is a cautionary flag as to the assessment’s consistency and reliability.

Of particular concern is the application of this subjective approach to the determination of the relevant QALY threshold for a treatment, which is the threshold used to calculate a “value-based price benchmark” for the treatment and evaluate whether the treatment is labeled as “low,” “intermediate,” or “high” value in ICER’s report. This approach has no methodologic basis and the mapping of scores to QALY thresholds appears to be arbitrary.

Additionally, the approach is biased toward thresholds at the lower end of the QALY range. Since reviews are often performed early in a treatment’s lifecycle, it is less likely that evidence will already be available to demonstrate the achievement of many of these additional factors. This imposes a disadvantage on newer products/treatments that have smaller evidence bases, increasing the potential for a low score from the voting panel. As noted above, a lower score means a lower QALY threshold, a lower “value-based price benchmark” and a higher likelihood of being labeled “low” value.

This subjective approach is not a good approach for quantitatively incorporating these important factors. While it is important to give quantitative credit for these factors, NPC believes it must be done in a methodologically sound manner that will produce consistent, transparent, reproducible results.

5. Quality-Adjusted Life Year (QALY)

NPC’s prior comments include a full discourse on the serious limitations of the QALY, which we will not repeat in detail here. Use of the QALY poses several significant concerns, primarily ethical considerations, methodologic issues and disease-specific considerations.[v] ICER itself has identified key problems with the QALY.[vi]

Cost-effectiveness analyses can be done with other methods that are not QALY-based, and NPC supports ICER’s intention to include “analyses of cost per life-year gained and certain other ‘cost per consequences’ when relevant.” NPC recommends that “cost per consequences” be included as a routine part of all reviews. If ICER does not see an opportunity to include a measure of “cost per consequence” in a specific review, then it will be especially important to listen to stakeholder suggestions for ways to include one.

ICER’s current assessments apply the threshold range of $100,000 to $150,000 per QALY; the proposed updates drop the lower bound to $50,000 per QALY. Although ICER references several data points to suggest $50,000 is an appropriate lower bound threshold for the U.S., it has not been agreed upon, nor is there consensus around this critical element. The reality is that we do not know what society’s willingness to pay (WTP) is in the U.S. for various diseases and scenarios. The Guiding Practices recommend a multi-stakeholder evaluation process reflecting societal values be used to set specific thresholds, and this has not occurred.

WTP for oncology suggests thresholds as high as $300,000 per QALY.[vii],[viii] Some real-world coverage decisions in the U.S. are similarly consistent with higher WTP thresholds. These signals of potentially higher thresholds in the U.S. should not be ignored, especially given the absence of a multi-stakeholder evaluation process to determine true societal WTP in the U.S. The upper bound of the range should be increased to allow for their consideration.

NPC also strongly suggests reconsideration of the new lower bound of $50,000. The evidence that this lower bound is relevant in the U.S. is far from conclusive, and the evidence that it is broadly relevant is even more tenuous. Using such a small and likely irrelevant lower bound is of even greater concern given the likelihood of the approach discussed in the prior section (D) to bias threshold selection toward the lower end of the threshold range.

 

2. Short-Term Affordability

Some of ICER’s proposed changes to the evaluation of “short-term affordability” are in line with the Guiding Practices and NPC’s prior comments, such as the incorporation of a range of potential uptake rates and a range of prices in the assessment of budget impact. We appreciate these changes. However, there continues to be critical misalignment with our Guiding Practices on the application of an artificial affordability threshold.

ICER has taken steps to separate budget impact assessment and value, with both revised terminology (e.g., dropping the use of “health system value” for budget impact) and a new depiction of the framework that more clearly separates the two pieces. However, the estimation of a value-based price benchmark based on budget impact keeps the two pieces intertwined.

1. Estimation of Budget Impact Scenarios

An estimate of budget impact is a necessary but insufficient part of evaluating affordability. NPC offers these comments on estimating potential budget impact with the explicit caveat—as per the Guiding Practices—that it is not appropriate to hold a budget impact estimate up against an artificial affordability threshold.

NPC’s prior comments called for realistic estimates of utilization and price, and recommended sensitivity analysis for both. ICER’s proposed use of a range of both uptake rates and potential prices (including net price) to create a variety of budget impact scenarios addresses the need for sensitivity analysis and moves toward more realistic estimates. For individual reviews, NPC strongly encourages ICER to rely on guidance from stakeholders to assess how realistic price and utilization estimates actually are, and adjust as needed.

Given the challenges inherent in attempting to predict a national “unmanaged” uptake rate, ICER’s decision to drop this approach from its draft reports appears to be an appropriate one. However, NPC does not support ICER’s proposal to simply shift the responsibility for the prediction of a hypothetical “unmanaged” scenario to the Policy Roundtable participants. The Policy Roundtable will not be in a better position to predict an imaginary “unmanaged” rate.

2. Separate Budget Impact from Affordability

Short-term budget impact is a measure of resource use and should remain separate from affordability. Budget impact is an important concept for payers; affordability is an important concept for society. Evaluating affordability involves making assessments and trade-offs at an overall health system level (i.e., a broad assessment of all investments in a health care system) and beyond the health system (i.e., spending on health care versus other societal considerations, such as education, police, and roads).

A comprehensive approach to affordability requires considerations of concepts such as disinvestment and willingness to pay, needs to be informed by cultural and societal values as well as health and non-health needs, and requires broad stakeholder involvement. ICER’s current approach to assessing affordability—setting a potential budget impact threshold that may trigger an “affordability alert”—is not a comprehensive consideration of the health care system, does not consider societal values, and does not adequately measure affordability.

Not only would an affordability assessment require decisions about health care spending versus non-health care spending, it also would require societal decisions about intra-health care spending. This would force decision-makers to make trade-offs regarding spending on the elderly versus the young, rare disorders versus common diseases, and curative therapies versus prolonging life or quality-of-life enhancement, as well as allocations among medications, surgery, hospital care, and physician services. ICER’s current framework and stakeholder input process do not incorporate these broader factors required to assess affordability. Therefore, its focus should not extend beyond an assessment of budget impact; the assessment of affordability should be eliminated.

3. Artificial Affordability Thresholds Are Inappropriate

The Guiding Practices state that an assessment of budget impact should not be judged against artificial affordability caps. As noted above, an affordability assessment needs to look broadly at all health care spending that is relevant to achieving a given health outcome. ICER’s approach focuses narrowly on a particular treatment and determines whether spending on that treatment might exceed a fixed portion of drug expenditures.

ICER’s current approach of setting a uniform budget impact threshold based on a fixed portion of drug expenditures creates an artificial affordability threshold that could have negative, unintended consequences. An analysis by IMS and NPC demonstrates that ICER’s affordability threshold could lead to an inefficient allocation of health care resources.[ix] It shows that one-third of spending on new drugs launched between 2003 and 2014 would have exceeded ICER’s threshold of $904 million, and hence would not have been available to spend on those drugs. If the affordability threshold—which is not based on value—causes us to shift this spending away from high-value drugs, it could be shifting money toward lower-value care that is less efficient, paradoxically reducing the value of our health care dollar.

Another unintended consequence of artificial affordability thresholds is the disincentivization of the development of drugs for broad populations with unmet need. Predicted budget impact will increase as the predicted number of patients increases, causing a treatment for a broad population —particularly one with unmet need—to be more likely to trigger an “affordability alert” threshold. However, a comprehensive affordability assessment that considers societal values and the broader public health perspective would likely result in a higher spending allocation for such a treatment.

Finally, the ICER threshold equation assumes that the allocation of health care spending among drugs, hospital care, imaging and physician care is the “correct” allocation across resources. For example, perhaps more resources should be spent on drugs and less on imaging for optimal resource allocation, or vice versa. The derived threshold assumes that the current allocation is optimal, an unproven assumption that is likely incorrect.

NPC recommends that the budget impact scenarios should be presented on their own, without an affordability threshold. Customizable scenarios that could be adapted by the various and diverse payers in the U.S. could be of even greater utility.

4. Using Artificial Thresholds to Derive Prices Confounds Budget Impact and Value

As noted above, and consistent with the Guiding Practices and NPC’s prior comments, ICER’s proposed revisions move toward separating budget impact from value. However, this separation is negated by using the budget impact threshold to calculate a value-based price benchmark (VBPB).

When the Policy Roundtable’s estimate of the (hypothetical) unmanaged uptake rate (UP) is larger than ICER’s estimate of the unmanaged uptake rate that would cross ICER’s budget threshold at “net price” (UI), ICER intends to issue an “affordability and access alert.” As part of this alert, ICER will calculate a VBPB that correlates with the Policy Roundtable’s unmanaged uptake rate estimate (UP).

ICER’s stated goal for the alert is to send a signal to stakeholders that there may be a short-term affordability challenge. Since the alert is “triggered” if UP > UI, the signal can be sent without taking the extra step of calculating a VBPB. Taking this unnecessary extra step of deriving a price using UP suggests a false level of precision around the Policy Roundtable’s subjective estimate. NPC strongly recommends eliminating the VBPB estimation.

 

3.  Assessment Process

ICER has listened to stakeholders and made many changes to the assessment process over the past year, including focused outreach to multiple sclerosis and rheumatoid arthritis patients. There are, however, areas where the assessment process can be made more robust.

1. Include Broader Expertise on Voting Panel

It is important for multiple voting panel members to have clinical expertise in the disease area under discussion to improve the clinical accuracy of their determinations. The recent inclusion of patient and clinician representatives alongside the panel is helpful for broadening the panel perspective, but these representatives do not actually get to vote alongside the panel. Transparency around how panel members and the patient/clinician representatives are evaluated and selected is needed.

Voting panel members also should have received some level of (independent) training on the fundamentals of cost-effectiveness and value assessments. It has been apparent at past meetings that some members of the voting panels were unfamiliar with the concept of an incremental cost-effectiveness ratio and how to interpret it. Independent training will be even more critical if ICER proceeds with the proposed approach to integrating additional benefits/disadvantages and contextual considerations, which is largely reliant on the voting panel.

2. Extend Length of Time for Review and Seek Broader Feedback

ICER has extended the time for stakeholders to submit comments on scoping documents and reports, but the amount of time is still far too short for most stakeholders and is inconsistent with timelines used by other HTA bodies and the government. Patient groups, in particular, have reported difficulty with reviewing assessment reports, identifying key issues and concerns, and developing constructive comments in such a limited amount of time. Additionally, smaller companies, which lack the manpower to quickly address these labor-intense requests for information, may be at a disadvantage relative to larger firms with more resources and experts in residence. Form should follow function, and sufficient time should be allowed for meaningful review and feedback by all interested stakeholders.

3. Review Assessments Regularly

The Guiding Practices recommend that assessments be reviewed regularly and updated to keep pace with and account for medical innovation. ICER’s plan to include “newly prominent language specifying the review is time limited” is a step in the right direction, as is the intended “New Evidence Update” for PCSK9s and ICER’s indication that a formal process for updating reports is currently under development.

NPC strongly reiterates its recommendation that ICER develop a clear process for managing the evolution of evidence and that assessments be revised as significant new evidence becomes available. If an assessment is no longer reflective of current evidence and ICER does not have the bandwidth to update the report, it will be important to flag the assessment as out of date and remove the report from the website.  

4. Increase Comment Transparency

In our prior comments, NPC recommended that all comments and their disposition should be publically available, with ICER providing the rationale for issues that it has chosen not to address. We commend ICER for adopting this approach with the public comments from the Rheumatoid Arthritis draft report. Continuing to do this is important for transparency and stakeholder engagement.

5. Expand Topics and Topic Selection Process

ICER’s topic selection process should include multiple stakeholders with significant patient representation. ICER currently conducts a disproportionate share of drug evaluations.Given that drugs still represent a limited portion of the overall health care budget, ICER’s impact on the health care system would increase if its agenda was less concentrated and considered other interventions.

ICER’s review topics are largely “forward-looking,” which puts ICER in the position of conducting reviews prematurely, before a sufficient body of clinical evidence and economic information is available. Unapproved products should be out of scope for assessments given the uncertainty surrounding labeling, pricing, etc.

“Backward-looking” reviews aimed at identifying low-value care in all areas of the health care system would have the benefit of a more complete body of evidence and could contribute to the creation of a significant amount of headroom in health care spending. Such a contribution would be well-aligned with ICER’s stated goal of “sustainable access to high-value care for all patients.”

All stakeholders would benefit from earlier notice of upcoming topics. Earlier notice allows stakeholders to set aside resources for an upcoming review, which is particularly crucial for small companies and patient groups. In 2016, a press release was issued with the reviews ICER planned for the year. The absence of a similar release in 2017 has been a source of stakeholder frustration.

 

We appreciate this opportunity to provide input on the proposed revisions to the ICER Value Assessment Framework. NPC’s continued engagement with ICER signifies our commitment to the critical dialogue necessary to ensure the development of high-quality, meaningful tools that help patients, physicians, payers, and others make informed decisions.

Respectfully submitted,

Robert W. Dubois, MD, PhD
Chief Science Officer

Kimberly Westrich, MA
Vice President, Health Services Research

[i] ICER Value Assessment Framework. https://icer-review.org/methodology/icers-methods/icer-value-assessment-framework/ Accessed March 7, 2017.

[ii] Guiding Practices for Patient-Centered Value Assessment. National Pharmaceutical Council. http://www.npcnow.org/sites/default/files/npc-guiding-practices-for-patient-centered-value-assessment.pdf. Accessed August 26, 2016.

[iii] NPC Comments on Proposed Improvements to ICER’s Value Assessment Framework. http://www.npcnow.org/sites/default/files/160909npc-comments-ICER-value-framework-final.pdf. Accessed March 7, 2017.

[iv] Sanders GD, Neumann PJ, Basu A, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses. JAMA. 2016;316():1093-1103. doi:10.1001/jama.2016.12195

[v] Pettitt DA, Raza S, Naughton B, et al. The Limitations of QALY: A Literature Review. J Stem Cell Res Ther. 2016; 6(4) 1-7. http://www.omicsonline.org/open-access/the-limitations-of-qaly-a-literature-review-2157-7633-1000334.pdf. Accessed August 26, 2016.

[vi] Pearson SD. Institute for Clinical and Economic Review website. A Framework to Guide Payer Assessment of the Value of Medical Treatments. http://icer-review.org/wp-content/uploads/2014/01/Value-Assessment-Framework-DRAFT-6-13-14.pdf: ICER White Paper. Published June 12, 2014. Accessed August 26, 2016.

[vii] Nadler E, Eckert B, Neumann P. Do Oncologists Believe New Cancer Drugs Offer Good Value? The Oncologist. 2006 Feb;11(2):90-5. doi: 10.1634/theoncologist.11-2-90

[viii] Seabury SA, Goldman DP, Maclean JR, et al. Patients Value Metastatic Cancer Therapy More Highly Than Is Typically Shown Through Traditional Estimates. Health Affairs. 2012 Apr;31(4):691-699.

[ix] Ciarametaro M, Abedi S, Sohn A, Fan Ge C, Odedara N, Dubois RW. Concerns around budget impact thresholds: not all drugs are the same. Value in Health. 2017 Feb;20(2):230-233. doi: 10.1016/j.jval.2016.11.031.