Results Summary
What was the project about?
Patients may respond differently to the same treatment due to differences in personal traits such as age, gender, or the number and type of health problems they have. Researchers use statistical methods to predict how well a treatment may work for patients based on their personal traits. But current methods may not work well if patients have many health problems or are taking other medicines.
In this project, the research team created new methods to figure out which patient traits are related to treatment benefits to help doctors and patients understand the likely treatment benefits for individual patients.
What did the research team do?
The research team created statistical methods to find out how much individual patients benefit from a treatment based on their traits. The team then used test data to compare the new methods with existing methods to see how they worked.
Then the research team used the new methods to calculate treatment benefit scores for different traits. The scores showed how certain traits affected a patient’s response to treatment. The team then used the scores to rank treatment benefits based on specific patient traits. The team also tested the new methods using real data from a clinical study.
Doctors, other researchers, health system staff, a patient, and a caregiver helped design the project.
What were the results?
The new methods helped find out how treatment benefits varied based on different patient traits. The methods frequently identified patient traits that were related to treatment benefits. The methods worked with different types of data. Some of the methods worked better than others.
What were the limits of the project?
The new methods only work if data that affect both the likelihood of receiving a treatment and the treatment outcomes are not included. Some of the methods may not work well if the analysis includes too many patient traits.
Future studies could test these methods with data from studies looking at different health conditions.
How can people use the results?
Researchers and doctors can use the methods to help know which patients are likely to benefit from a treatment based on a patient’s traits.
Professional Abstract
Background
Heterogeneity of treatment effect (HTE) occurs when patients with different personal or clinical characteristics respond differently to the same treatment. Identifying HTE can help clinicians decide which treatments are best for their patients. Current methods for measuring HTE are limited by the number of patient characteristics that can be accounted for in HTE analyses. These methods are also less accurate when patients have multiple health problems and take many medications. Better methods of identifying characteristics that contribute to HTE in patients with complex conditions can help determine the most beneficial treatments for patients based on their individual characteristics.
Objective
To develop and test methods for estimating HTE for different patient characteristics to rank treatment benefits for patients
Study Design
Design Element | Description |
---|---|
Design | Simulation studies, empirical analysis |
Data Sources and Data Sets | Simulated data, observational data |
Analytic Approach |
|
Outcomes |
In simulation studies, performance of estimators measured by ranking correlation coefficients, receiver operating characteristic curves of benefit scores, and misclassification error In empirical analysis, concordance between ITE and identification of groups of patient characteristics across different patient outcomes |
Methods
To measure HTE, researchers estimated the treatment benefit for individual patients, also known as the individual treatment effect (ITE), based on different patient characteristics.
First, researchers developed a general statistical framework for estimating ITE to identify patient characteristics that moderate treatment outcomes. Using different estimation methods, researchers modeled treatment assignment rather than treatment outcomes. They included outcomes as weights to examine the effect of interactions between treatments and patient characteristics on various treatment assignments.
Using simulation studies, researchers evaluated the framework and tested the performance of different estimation methods. Researchers then used the ITE estimates to create treatment benefit scores. The scores indicated a treatment’s benefit based on the influence of patient characteristics. Researchers used the scores to rank treatment benefits for individual patients based on their specific patient characteristics.
Researchers tested the methods using data from an observational study to identify which patient characteristics were related to benefits from a chronic care management intervention. They adapted the methods for three separate longitudinal outcomes.
Researchers worked with members of a health system management team, a patient, a caregiver, patient advocates, biostatisticians, and clinician researchers.
Results
In simulation studies, the framework performed well with many estimation methods and with continuous and noncontinuous outcomes. The performance was generally undermined when the main effects were larger than interaction effects and patient factors or covariates correlated with treatment effects.
In the observational data analysis, the methods resulted in similar treatment benefit scores across three different outcomes. The groups of patient characteristics identified were largely consistent across all outcomes.
Limitations
The methods apply in situations in which no unmeasured variables affect both treatment assignment and patient outcomes. Some of the methods may not work well when too many patient characteristics and other covariates are in the framework.
Conclusions and Relevance
The new methods can help identify which patients benefit from specific treatments based on patients’ characteristics. Researchers and clinicians can apply the methods to different clinical settings to construct clinical scoring systems for selecting treatments for different patients.
Future Research Needs
Future research could test these methods with different types of data and health conditions.
Final Research Report
View this project's final research report
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers noted that tests are necessary to verify that the methods for treatment selection developed in this study are better than the conventional approach to treatment selection. They indicated that such a hypothesis-testing approach was important in establishing the superiority of the researchers’ approach. The researchers agreed that such an evaluation would be important for establishing their approach but that this work was beyond the scope of the current study. They did propose a resampling procedure in the report to allow for such method comparisons.
- The reviewers asked how loss functions, which help to estimate the potential downside for choosing one treatment over another, should be chosen given that different loss functions would lead to different treatment assignment strategies. The researchers explained that the solution was largely mathematical. In response to this question, the researchers expanded the report’s discussion of alternate mathematical strategies designed to minimize the loss function in order to maximize the value of the treatment assignment.
Conflict of Interest Disclosures
Project Information
Key Dates
Study Registration Information
^This project was previously titled: Developing Statistical Methods to Help Doctors and Patients Decide on the Right Treatment