(1) To develop a rapid process for calculating value-of-information (VOI) estimates of clinical and economic returns for research investments; (2) To evaluate the effect of providing VOI estimates to reviewers of cancer research proposals
|Data Sources and Data Sets
- Retrospective data: 9 phase 2 or phase 3 randomized trial proposals reviewed by the SWOG executive committee between 2008 and 2013
- Prospective data: 9 phase 2 or phase 3 randomized trial proposals received by the SWOG executive committee from February 2015 to December 2016
Development of VOI estimates: minimal modeling techniques and expert opinion
Evaluation of impact of VOI information on proposal ratings: Wilcoxon signed-rank test
Analysis of survey data on committee member opinions of VOI: descriptive statistics and t-tests
- Efficiency of VOI process to support decision making
- Effect of VOI process on committee proposal review scores, committee opinions on VOI process
VOI analysis is a health-economics technique that estimates clinical and economic returns for research investments. These estimates can supplement other information in prioritizing research. However, the time it takes to calculate VOI limits its widespread use within proposal-review processes operating under tight deadlines.
The research team worked with researchers and the executive committee at SWOG, a cooperative group in the National Cancer Institute’s National Clinical Trials Network (formerly the Southwest Oncology Group). The research team developed a minimal modeling process for rapidly calculating VOI estimates using a random retrospective sample of nine phase 2 and phase 3 randomized trial proposals that the SWOG executive committee reviewed between 2008 and 2013.
The research team then applied the process to calculate VOI estimates for each of nine additional SWOG proposals for use during a review. The VOI calculations included both per-patient and population-level incremental expected costs and quality-adjusted life years projected to result from the research investments. The team provided both clinical VOI, which estimates only expected clinical impact, and classic VOI, which estimates both clinical and economic impacts.
The research team then evaluated the effect of providing the VOI estimates on committee members’ proposal reviews. Committee members scored each proposal on scientific merit and potential impact before and after viewing the VOI estimates. Committee members also completed surveys before and after the study to assess their decision-making process and attitudes about the use of VOI estimates.
The minimal modeling technique to rapidly calculate VOI estimates was efficient: estimation time per proposal was less than one week. VOI estimates that do not use minimal modeling typically take several months to calculate.
After the executive committee used the VOI estimates in its review, the committee’s proposal ratings changed for eight of the nine proposals (p=0.03), most often in a less favorable direction. Using VOI estimates did not affect the acceptance decision for any proposal.
Committee members reported that the VOI materials were easy to understand and training was sufficient (67%), and they felt confident in interpreting the data (75%). Most respondents felt the VOI estimates aided the evaluation process and their decision making (50%) or were neutral about VOI estimates (42%). A small minority (8%) felt the VOI estimates hindered the evaluation process. Of respondents, 42% supported adding VOI estimates, 41% were neutral, and 17% did not support adding VOI estimates to the evaluation process.
The number of proposals that the study evaluated was small. The study included only phase 2 and 3 randomized trials because current VOI methods cannot easily accommodate uncontrolled study designs. Changes in the composition of the executive committee hindered the team’s ability to assess changes in attitudes toward VOI over time. The minimal modeling VOI technique may not be as accurate as VOI frameworks that are more comprehensive.
Conclusions and Relevance
The minimal modeling VOI approach is an efficient process to generate VOI estimates for individual cancer clinical trial proposals. The proposal-evaluation process could incorporate VOI estimates swiftly; however, less than half of the SWOG executive committee members supported incorporating VOI data.
Future Research Needs
Future research could explore patient, clinician, and researcher perspectives on whether and under what conditions VOI analysis helps prioritize research funding.