Results Summary
What was the research about?
Organizations that fund cancer research need to decide which studies to fund. Value of information (VOI) is a way to help rank research studies. VOI estimates the value of research by looking at the impacts on health and on healthcare that could result from the research.
SWOG (formerly the Southwest Oncology Group) is a network of cancer researchers funded by the National Cancer Institute. SWOG leaders review and score new study proposals based on the scientific value of the studies and their potential impact. Based on a study’s score, SWOG’s leadership committee decides whether to send the study to the National Cancer Institute for funding review.
In this study, the research team wanted to learn if giving VOI data to the committee affected its scoring of proposals. The team also wanted to see if providing VOI data was helpful in deciding which studies to fund. The study had two parts. The research team created a process to quickly estimate VOI. Then, the team tested the process on nine study proposals that the SWOG committee reviewed.
What were the results?
The research team’s new process to estimate VOI was much quicker than the normal process, taking less than one week instead of several months.
The VOI data caused the committee to change scores for eight of the nine proposals. Most of the time, the committee gave worse scores to proposals after looking at the VOI data. The VOI data didn’t change whether the committee approved these proposals.
Most committee members felt the VOI data were easy to understand (67 percent). Most thought the information was either helpful (50 percent) or neutral (42 percent). A few people (8 percent) felt the VOI data made their review harder. Forty-two percent thought adding VOI data to the review process was a good idea, 41 percent felt neutral, and 17 percent thought it was a bad idea.
What did the research team do?
The research team randomly selected nine proposals the SWOG committee had reviewed from 2008 to 2013. The team used these proposals to create the new process to quickly estimate VOI. The research team improved its process based on feedback from the SWOG committee and other SWOG researchers.
The research team then tested the process using nine new proposals that the SWOG committee received between February 2015 and December 2016. The research team used the new process to estimate VOI for each proposal. The team sent the resulting VOI data to the SWOG committee. The team also sent committee members information about VOI to help them understand the data. Committee members scored each proposal before and after looking at the VOI data.
The research team asked committee members to fill out a survey at the start of the study and again at the end. The survey asked how the committee decides which studies to fund. It also asked how the committee members felt about using VOI data.
What were the limits of the study?
The research team tested the process with a small number of cancer studies within one research network, SWOG. Results may be different for other research networks. The quicker process may not work as well as the longer process to estimate VOI. Members of the committee changed during the study. The changes made it hard to see if committee members’ opinions about VOI changed over time.
Future research could test the new process on more studies. Researchers could see if the process works for research networks other than SWOG.
How can people use the results?
Organizations that fund research may consider using the quicker process to estimate VOI to help choose which research studies to fund.
Professional Abstract
Objectives
(1) To develop a rapid process for calculating value-of-information (VOI) estimates of clinical and economic returns for research investments; (2) To evaluate the effect of providing VOI estimates to reviewers of cancer research proposals
Study Design
Design Elements | Description |
---|---|
Design | Empirical analysis |
Data Sources and Data Sets |
|
Analytic Approach |
Development of VOI estimates: minimal modeling techniques and expert opinion Evaluation of impact of VOI information on proposal ratings: Wilcoxon signed-rank test Analysis of survey data on committee member opinions of VOI: descriptive statistics and t-tests |
Outcomes |
|
VOI analysis is a health-economics technique that estimates clinical and economic returns for research investments. These estimates can supplement other information in prioritizing research. However, the time it takes to calculate VOI limits its widespread use within proposal-review processes operating under tight deadlines.
The research team worked with researchers and the executive committee at SWOG, a cooperative group in the National Cancer Institute’s National Clinical Trials Network (formerly the Southwest Oncology Group). The research team developed a minimal modeling process for rapidly calculating VOI estimates using a random retrospective sample of nine phase 2 and phase 3 randomized trial proposals that the SWOG executive committee reviewed between 2008 and 2013.
The research team then applied the process to calculate VOI estimates for each of nine additional SWOG proposals for use during a review. The VOI calculations included both per-patient and population-level incremental expected costs and quality-adjusted life years projected to result from the research investments. The team provided both clinical VOI, which estimates only expected clinical impact, and classic VOI, which estimates both clinical and economic impacts.
The research team then evaluated the effect of providing the VOI estimates on committee members’ proposal reviews. Committee members scored each proposal on scientific merit and potential impact before and after viewing the VOI estimates. Committee members also completed surveys before and after the study to assess their decision-making process and attitudes about the use of VOI estimates.
Results
The minimal modeling technique to rapidly calculate VOI estimates was efficient: estimation time per proposal was less than one week. VOI estimates that do not use minimal modeling typically take several months to calculate.
After the executive committee used the VOI estimates in its review, the committee’s proposal ratings changed for eight of the nine proposals (p=0.03), most often in a less favorable direction. Using VOI estimates did not affect the acceptance decision for any proposal.
Committee members reported that the VOI materials were easy to understand and training was sufficient (67%), and they felt confident in interpreting the data (75%). Most respondents felt the VOI estimates aided the evaluation process and their decision making (50%) or were neutral about VOI estimates (42%). A small minority (8%) felt the VOI estimates hindered the evaluation process. Of respondents, 42% supported adding VOI estimates, 41% were neutral, and 17% did not support adding VOI estimates to the evaluation process.
Limitations
The number of proposals that the study evaluated was small. The study included only phase 2 and 3 randomized trials because current VOI methods cannot easily accommodate uncontrolled study designs. Changes in the composition of the executive committee hindered the team’s ability to assess changes in attitudes toward VOI over time. The minimal modeling VOI technique may not be as accurate as VOI frameworks that are more comprehensive.
Conclusions and Relevance
The minimal modeling VOI approach is an efficient process to generate VOI estimates for individual cancer clinical trial proposals. The proposal-evaluation process could incorporate VOI estimates swiftly; however, less than half of the SWOG executive committee members supported incorporating VOI data.
Future Research Needs
Future research could explore patient, clinician, and researcher perspectives on whether and under what conditions VOI analysis helps prioritize research funding.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also confirms that the research has followed PCORI’s Methodology Standards. During peer review, experts who were not members of the research team read a draft report of the research. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. Reviewers do not have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve how the research team analyzed its results or reported its conclusions. Learn more about PCORI’s peer-review process here.
In response to peer review, Ramsey made changes including
- Revising the Background section to make more evident what methodological gaps the study addressed, and to provide more justification for the importance of a value of information (VOI) approach to informing research prioritization
- Revising the section on patient and stakeholder participation to describe participation in each of the phases separately
- Adding to the Methods section a description of changes made to the original study protocol, in particular the reduction in study scope due to difficulties completing Aim 1