Results Summary
What was the project about?
Treatment plans for patients with long-term health problems such as diabetes or arthritis often change over time. Such plans are called adaptive treatment plans as doctors adapt treatment based on the patient’s health problem and response to earlier treatments. Adaptive treatment plans are common, but the methods to assess how well a plan works may not always provide accurate results. To know which plans are best for patients, researchers need better methods to compare these adaptive plans.
In this study, the research team developed and tested a new statistical method and looked at whether it could more accurately compare adaptive treatment plans.
What did the research team do?
The research team first developed a new statistical method called GPMatch. Using a computer program, the team created test data. The team used the test data to look at how well GPMatch worked compared with current statistical methods.
Then the research team used GPMatch with real data from patients’ health records. The team compared two types of adaptive treatment plans for children with polyarticular-course juvenile idiopathic arthritis, or pcJIA. The two types of plans were
- Early combination plan. Patients start two types of medicines at the same time after diagnosis.
- Step-up plan. Patients start with one type of medicine and then start the second type later.
Using GPMatch, the research team checked which plan improved children’s health after 6 and 12 months.
Patients, parents of patients with pcJIA, doctors, and patient advocates helped develop and test GPMatch.
What were the results?
With the test data, GPMatch was more accurate than current statistical methods in measuring the effects of adaptive treatment plans.
Using GPMatch, the research team found that the early combination plan was better at improving health for patients with pcJIA than the step-up plan. Both plans improved quality of life.
The research team developed a computer program to help other researchers use GPMatch.
What were the limits of the project?
For studies with data on more than 5,000 patients, GPMatch takes more than one day to run on the computer. The research team tested the new methods using data from patients with pcJIA from one health center. Results may differ with data from other centers or for other health problems.
Future research could work on ways to use the methods with more complex data. Studies could also test these methods using data from other clinics or for other health problems.
How can people use the results?
Researchers can use the new statistical methods to compare treatment plans that change over time.
Professional Abstract
Background
In treating chronic diseases, physicians commonly apply adaptive treatment strategies that adjust treatment plans over time based on disease progression and patients’ responses to previous treatments. Examining the comparative effectiveness of an adaptive treatment strategy requires advanced causal inference methods that correctly assess the effect of changing treatments on patient outcomes while accounting for changes in health status and other confounding variables over time. Existing methods are not designed for evaluating adaptive treatment strategies and may not work in the presence of model misspecification, when the model fails to account for everything it should.
Objective
To develop and evaluate a Bayesian causal inference method called Gaussian process match (GPMatch) to reduce model misspecification and provide accurate treatment effect estimates in studies that compare the effectiveness of adaptive treatment strategies
Study Design
| Design Element | Description |
|---|---|
| Design | Simulations, empirical analyses |
| Data Sources and Data Sets | Simulated data, observational data from electronic health records from a pediatric rheumatology clinic |
| Analytic Approach | GPMatch, BART, propensity score subclassification, augmented inverse probability treatment weighting, regression adjustment, marginal structural model, G-computation, G-estimation |
| Outcomes |
Root mean square error, median absolute error, bias, rate of coverage Clinical Juvenile Arthritis Disease Activity Score at 6 and 12 months, Pediatric Quality of Life Inventory Score at 12 months |
Methods
The research team developed a causal inference method called GPMatch that combines two statistical techniques, Gaussian process covariance function matching and Bayesian nonparametric modeling, in one step.
Using simulated data, the research team compared GPMatch against Bayesian additive regression trees (BART) and other commonly used statistical methods for adaptive and non-adaptive treatment strategies. To test the robustness of the methods in the presence of model misspecification, the team intentionally introduced error into the statistical models.
The research team then applied GPMatch to empirical data from electronic health records. The team compared the effectiveness of two adaptive treatment strategies for children with polyarticular-course juvenile idiopathic arthritis (pcJIA):
- The early combination plan, in which patients start two types of medicine after diagnosis
- The step-up plan, in which patients start with one type of medicine and then add another type later
Patients, researchers, patient advocates, and healthcare providers helped develop and test the methods.
Results
In simulations, GPMatch, followed by BART, performed better and had less bias and less root mean square error than other methods in both adaptive and non-adaptive treatment plans.
With empirical data, GPMatch showed that
- Patients in the early combination plan had greater improvement in disease activity score after 6 and 12 months than patients in the step-up plan.
- Both plans improved quality of life after 12 months.
The research team also developed a web-based graphic user interface to help other researchers use GPMatch and BART for analyzing data from comparative effectiveness studies.
Limitations
GPMatch may require more than a day of computation time for sample sizes larger than 5,000. Data to assess treatment plans for pcJIA came from one medical center and did not include information on dosage or medication adherence. Results may differ in other settings or with additional clinical data.
Conclusions and Relevance
The GPMatch method can compare the effectiveness of adaptive treatment plans while accounting for posttreatment confounding. It addresses the challenges of model misspecification when working with real-world data and time-varying treatments.
Future Research Needs
Future research could further test GPMatch using other data sources and develop GPMatch for measuring heterogeneous treatment effects and treatment effects in multilevel or cluster data.
Final Research Report
View this project's final research report.
Journal Citations
Results of This Project
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers commented that the simulation study is not representative of many electronic medical record (EMR) studies because of the low number of cases and co-occurring variables. They asked that this be noted as a limitation of the study. The researchers explained that they designed their simulation to meet several statistical goals and that most of their simulations came from existing literature. The researchers agreed that their simulation was relatively small in its number of observations and covariates, given the usual scale of comparative effectiveness research (CER) with EMRs. The researchers explained that they did not use larger simulated data sets because of limited time and the small size of their juvenile idiopathic arthritis CER study. However, the researchers noted their GPMatch method ranked second in a comparison of 19 causal inference methods. This comparison used many more cases and covariates than they used in the current project. The researchers felt that this demonstrated the strong performance of GPMatch, even with a larger dataset with more covariates.
- The reviewers asked if the concept of adaptive treatment strategies described in the report was the same as the concept of a dynamic treatment regime. The reviewers asked for clarification on terms and goals. The researchers said the dynamic treatment regime approach tends to focus on trying to find an optimal set of decision rules that leads to a desired final outcome using reinforcement learning, but the approach does not fit well with research that is interested in optimizing results at intermediate time points. The researchers explained they used the term adaptive treatment strategies to distinguish the approach in GPMatch from methods that rely on reinforcement learning, but acknowledged that adaptive treatment strategies and dynamic treatment regime are often used interchangeably in the literature.