Results Summary
What was the research about?
Patients with chronic health problems, such as diabetes, often need to change treatment plans over time to improve their health. To help with this process, doctors can monitor patients’ health through follow-up clinic visits and lab tests. Doctors may also suggest changing a treatment plan in response to visits or lab test results. When a treatment plan changes in this way, it’s called a dynamic treatment plan.
In this study, the research team developed and tested new statistical methods to learn how dynamic treatment plans and choices about follow-up care affect patients’ health. These methods use electronic health records, or EHRs. Using EHRs is helpful because they have data on
- What treatments patients have received over time
- How treatments have affected patients’ health
- Follow-up information such as lab test results
But the data may differ for patients based on when and why they go to the doctor. These differences make it hard for researchers to accurately know the effect of dynamic treatment plans across many patients.
What were the results?
The research team developed the mathematical basis for creating new methods to address these problems in using EHR data. Then they developed and tested the methods. The team showed these methods could accurately measure the effect of dynamic treatment plans on patients’ health. They also created computer programs to help other researchers use the methods.
What did the research team do?
The research team used data created by computer programs to see how the methods worked. The team used the methods to analyze real EHRs from patients with diabetes. The analysis looked at how dynamic treatment plans and choices about follow-up care affect patients’ health.
The research team worked with patients with diabetes, doctors, and patient advocates to develop and test the methods with health topics that are important to patients and doctors.
What were the limits of the study?
The methods may not work if the EHRs don’t have data on certain patient traits, such as smoking or past illness, that affect treatment or follow-up choices and patients’ health.
Future research could develop statistical approaches to check how results from these methods change if data on patients’ traits are missing from EHRs.
How can people use the results?
Researchers could use the methods to give doctors and patients data about how different ways to treat and follow up with patients can affect their health.
Professional Abstract
Objective
To develop and evaluate innovative causal inference methods that are suitable for analyzing electronic health record (EHR) data in comparative effectiveness research (CER) on dynamic treatment regimens
Study Design
Design Elements | Description |
---|---|
Design | Theoretical derivations, simulation studies, empirical analyses, software development |
Data Sources and Data Sets | Simulated data Multicenter observational data that included EHRs for 58,000 patients with type 2 diabetes |
Analytic Approach | Estimation of the effect of dynamic treatment regimens with IPW and TMLE on time-to-event outcomes in studies subject to variation in how often and when patients are observed and clinically evaluated |
Outcomes |
|
In dynamic treatment regimens, clinical management decisions evolve over time based on a patient’s response to treatment and monitoring. CER on dynamic treatment regimens can use EHR data because the data include information about treatment and monitoring outcomes that vary during the course of an illness. However, statistical challenges for estimating causal effects limit wide-scale use of EHR data in CER. For example, the variation in timing and content of EHR data from patient to patient increases concerns about bias and can limit the generalizability of inferences obtained with existing statistical methods. To address these challenges, researchers can develop new methods using EHR variability to better inform treatment and monitoring decisions.
In this study, the research team developed and evaluated innovative causal estimation methods using EHR data to measure the effects of dynamic treatment regimens and monitoring regimens on health outcomes. The team derived theoretical results to construct new estimation methods from two general estimation approaches: inverse probability weighting (IPW) and targeted minimum loss-based estimation (TMLE). The new methods assume that monitoring would influence treatment decisions but would have no direct effect on the health outcome. The team evaluated the various methods developed using simulation studies and empirical analysis of real EHR data from a prior type 2 diabetes study. To simulate data and facilitate use of the methods developed in future CER, the team also developed free, publicly available software.
The research team worked with patients with diabetes, researchers, statisticians, physicians, patient advocates, and pharmacists to develop and test the methods with health topics that are important to patients and doctors.
Results
The research team derived two theorems to use as a basis for constructing new IPW and TMLE methods. The team also developed methods to address potential bias from incomplete outcome data.
Both the simulation and empirical analyses showed that the various methods developed by the research team performed well in estimating the effect of dynamic treatment and monitoring regimens in observational studies with complex and incomplete longitudinal data.
The research team developed a publicly available software package called simcausal to generate simulated EHR data. The team also developed two publicly available software packages called stremr and MSMstructure to analyze data using the new methods.
Limitations
Use of the IPW and TMLE methods developed by the research team may not be suitable in situations in which the effect of monitoring on health outcomes is not entirely mediated by treatment decisions or if there is unmeasured confounding.
Conclusions and Relevance
The new IPW and TMLE methods can improve the validity and applicability of results generated from EHR-based CER on dynamic treatment regimens. The software can streamline CER analyses with data from EHR-based studies with large samples and long duration of treatment follow-up.
Future Research Needs
Future research could develop an analytic toolkit to evaluate the effect of unmeasured confounding on findings from the new IPW and TMLE methods.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented, and the researchers made changes or provided responses. The comments and responses included the following:
- The reviewers suggested moving some details from the results to the methods section. While the researchers agreed that they had described methods in the results section, they maintained that doing so was appropriate because the project aimed to develop analytic methods, and the methods described in the results are products of the project.
- The reviewers indicated that the report did not adequately explain the proof the investigators used for the identifiability result and recommended that the authors explain their results with more detail. The researchers added a more intuitive explanation, as well as a sketch describing the proof that they used to establish the result in the report’s appendix.
- The reviewers asked for additional details on how the researchers constructed simulations and estimated various parameters . Also, they asked for what specific algorithms the researchers used. The researchers said they understood that it would be helpful to have details needed to replicate the study findings, but they felt constrained by the word limit set for the report. They included specifics of simulations and data analyses in the appendix.