Can Value-of-Information Estimates Help Funding Organizations Prioritize which Cancer Studies to Fund?

• Download Summary

What was the research about?

Organizations that fund cancer research need to decide which studies to fund. Value of information (VOI) is a way to help rank research studies. VOI estimates the value of research by looking at the impacts on health and on healthcare that could result from the research.

SWOG (formerly the Southwest Oncology Group) is a network of cancer researchers funded by the National Cancer Institute. SWOG leaders review and score new study proposals based on the scientific value of the studies and their potential impact. Based on a study’s score, SWOG’s leadership committee decides whether to send the study to the National Cancer Institute for funding review.

In this study, the research team wanted to learn if giving VOI data to the committee affected its scoring of proposals. The team also wanted to see if providing VOI data was helpful in deciding which studies to fund. The study had two parts. The research team created a process to quickly estimate VOI. Then, the team tested the process on nine study proposals that the SWOG committee reviewed.

What were the results?

The research team’s new process to estimate VOI was much quicker than the normal process, taking less than one week instead of several months.

The VOI data caused the committee to change scores for eight of the nine proposals. Most of the time, the committee gave worse scores to proposals after looking at the VOI data. The VOI data didn’t change whether the committee approved these proposals.

Most committee members felt the VOI data were easy to understand (67 percent). Most thought the information was either helpful (50 percent) or neutral (42 percent). A few people (8 percent) felt the VOI data made their review harder. Forty-two percent thought adding VOI data to the review process was a good idea, 41 percent felt neutral, and 17 percent thought it was a bad idea.

What did the research team do?

The research team randomly selected nine proposals the SWOG committee had reviewed from 2008 to 2013. The team used these proposals to create the new process to quickly estimate VOI. The research team improved its process based on feedback from the SWOG committee and other SWOG researchers.

The research team then tested the process using nine new proposals that the SWOG committee received between February 2015 and December 2016. The research team used the new process to estimate VOI for each proposal. The team sent the resulting VOI data to the SWOG committee. The team also sent committee members information about VOI to help them understand the data. Committee members scored each proposal before and after looking at the VOI data.

The research team asked committee members to fill out a survey at the start of the study and again at the end. The survey asked how the committee decides which studies to fund. It also asked how the committee members felt about using VOI data.

What were the limits of the study?

The research team tested the process with a small number of cancer studies within one research network, SWOG. Results may be different for other research networks. The quicker process may not work as well as the longer process to estimate VOI. Members of the committee changed during the study. The changes made it hard to see if committee members’ opinions about VOI changed over time.

Future research could test the new process on more studies. Researchers could see if the process works for research networks other than SWOG.

How can people use the results?

Organizations that fund research may consider using the quicker process to estimate VOI to help choose which research studies to fund.

Objectives

(1) To develop a rapid process for calculating value-of-information (VOI) estimates of clinical and economic returns for research investments; (2) To evaluate the effect of providing VOI estimates to reviewers of cancer research proposals

Study Design

VOI analysis is a health-economics technique that estimates clinical and economic returns for research investments. These estimates can supplement other information in prioritizing research. However, the time it takes to calculate VOI limits its widespread use within proposal-review processes operating under tight deadlines.

The research team worked with researchers and the executive committee at SWOG, a cooperative group in the National Cancer Institute’s National Clinical Trials Network (formerly the Southwest Oncology Group). The research team developed a minimal modeling process for rapidly calculating VOI estimates using a random retrospective sample of nine phase 2 and phase 3 randomized trial proposals that the SWOG executive committee reviewed between 2008 and 2013.

The research team then applied the process to calculate VOI estimates for each of nine additional SWOG proposals for use during a review. The VOI calculations included both per-patient and population-level incremental expected costs and quality-adjusted life years projected to result from the research investments. The team provided both clinical VOI, which estimates only expected clinical impact, and classic VOI, which estimates both clinical and economic impacts.

The research team then evaluated the effect of providing the VOI estimates on committee members’ proposal reviews. Committee members scored each proposal on scientific merit and potential impact before and after viewing the VOI estimates. Committee members also completed surveys before and after the study to assess their decision-making process and attitudes about the use of VOI estimates.

Results

The minimal modeling technique to rapidly calculate VOI estimates was efficient: estimation time per proposal was less than one week. VOI estimates that do not use minimal modeling typically take several months to calculate.

After the executive committee used the VOI estimates in its review, the committee’s proposal ratings changed for eight of the nine proposals (p=0.03), most often in a less favorable direction. Using VOI estimates did not affect the acceptance decision for any proposal.

Committee members reported that the VOI materials were easy to understand and training was sufficient (67%), and they felt confident in interpreting the data (75%). Most respondents felt the VOI estimates aided the evaluation process and their decision making (50%) or were neutral about VOI estimates (42%). A small minority (8%) felt the VOI estimates hindered the evaluation process. Of respondents, 42% supported adding VOI estimates, 41% were neutral, and 17% did not support adding VOI estimates to the evaluation process.

Limitations

The number of proposals that the study evaluated was small. The study included only phase 2 and 3 randomized trials because current VOI methods cannot easily accommodate uncontrolled study designs. Changes in the composition of the executive committee hindered the team’s ability to assess changes in attitudes toward VOI over time. The minimal modeling VOI technique may not be as accurate as VOI frameworks that are more comprehensive.

Conclusions and Relevance

The minimal modeling VOI approach is an efficient process to generate VOI estimates for individual cancer clinical trial proposals. The proposal-evaluation process could incorporate VOI estimates swiftly; however, less than half of the SWOG executive committee members supported incorporating VOI data.

Future Research Needs

Future research could explore patient, clinician, and researcher perspectives on whether and under what conditions VOI analysis helps prioritize research funding.