Results Summary
What was the project about?
Patients often have more than two treatment options. Researchers can use data from electronic health records, or EHRs, to compare different treatments and to learn about what treatment works best for patients. But current statistical methods don’t work well for comparing more than two treatments. Also, patients using different treatments may have different traits like age or number of health problems, which can bias results.
In this study, the research team created and tested a new method for using EHR data to compare more than two treatments. To develop the new methods, the team used a Bayesian approach. Bayesian approaches include findings from previous studies in the analysis, which can make results more accurate.
What did the research team do?
The research team based their new method on an existing method called Bayesian Additive Regression Trees, or BART. To test the new method, the team used data created by a computer program to look like real patient data. Then they compared the new method with current methods under different scenarios. Each scenario included three treatments. The team changed the total number of patients, the number of patients who took each treatment, and how alike or different the patients were who took each treatment. Across all scenarios, the team predicted the average treatment effect for all patients and for only patients who received a treatment.
Next, the research team used the new method with real data from patients with lung cancer who were receiving care in New York City hospitals. The team compared three types of surgery: open chest, robotic assisted, and video assisted. The team looked at the effects of each type of surgery on four health outcomes: breathing problems; length of hospital stay after surgery; stay in an intensive care unit, or ICU; and the need to return to the hospital.
Patients, doctors, and researchers helped design the study.
What were the results?
With the test data, the new method was more accurate than current methods when looking at the average treatment effect for all patients. The new method worked about the same as current methods when looking at only patients who received a treatment.
When using real patient data, the new method had the most precise predictions compared to other methods. The results showed that one treatment, robotic-assisted surgery, worked better than other surgery types to lower risk of a long hospital stay or an ICU stay. The risk of breathing problems or needing to return to the hospital were similar across types of surgery.
What were the limits of the project?
The research team tested the new method using data from patients who have Medicare. Results may differ for other patients. The team used data from one time point. Results may differ for studies that look at patient outcomes over time.
Future research could test how well the new method works for patient outcomes over time and for patients with other health problems.
How can people use the results?
Researchers can use the methods for studies comparing more than two treatments using EHR data.
Professional Abstract
Background
Patients often have multiple treatment options in clinical settings, but most statistical methods for using electronic health record (EHR) data in comparative effectiveness research are designed to compare only two treatments. Existing methods are also subject to bias due to differences in patient characteristics and unmeasured confounding in EHR data. Researchers lack methods to accurately compare the effectiveness of more than two treatments using EHR data.
Bayesian methods apply previously collected data to improve analysis, making study results more accurate. Bayesian machine learning methods that incorporate patients’ clinical histories from EHR data may help researchers compare the effectiveness of more than two treatments.
Objective
To develop a new Bayesian machine learning method for comparing the effects of more than two treatments on a binary outcome
Study Design
| Design Element | Description |
|---|---|
| Design | Simulation studies; empirical analysis |
| Data Sources and Data Sets | SEER-Medicare database; 11,980 Medicare patients ages 65 and older with stage I–IIIA non-small cell lung cancer |
| Analytic Approach | Bayesian Additive Regression Tree modeling |
| Outcomes | Simulation study: bias, root mean square error Empirical analysis: respiratory complication, prolonged length of hospital stay, intensive care unit stay, and readmission after lung cancer surgery |
Methods
The research team adapted Bayesian Additive Regression Trees to develop a new method for estimating the average treatment effect for the total number of patients (ATE) and the average treatment effect for treated patients (ATT).
In simulation studies, the research team compared the new method with existing methods, estimating ATE and ATT for three treatments options. The team first compared the new method with 10 existing methods in three scenarios that varied the total number of patients and the number of patients receiving each treatment. Then the team compared the new method with two existing methods in four scenarios that varied covariate overlap and confounding.
In the empirical analysis, the research team applied the new method to Surveillance, Epidemiology, and End Results-Medicare (SEER-Medicare) data to compare the effectiveness of three surgical treatments for non-small cell lung cancer on four postsurgical outcomes. The treatments included robotic-assisted surgery, video-assisted thoracic surgery, and open thoracotomy. The team estimated ATT for respiratory complications, prolonged length of hospital stay, intensive care unit stay, and readmission.
Patients, clinicians, and researchers provided input during the study.
Results
In the simulation studies, the new method produced lower bias and root mean square error for ATE estimates across all scenarios, compared with existing methods. The new method performed about the same as existing methods in estimating ATT across most scenarios.
When applying the new method to patient EHR data, the new method produced treatment effects with the smallest confidence intervals compared to other methods. The empirical analysis results suggest that patients who had robotic-assisted surgery would have a lower risk of a prolonged hospital stay and lower risk of an intensive care unit stay than patients who had video-assisted thoracic surgery or open thoracotomy. The risk of respiratory complications and readmission were similar across surgery types.
Limitations
The research team applied the new method to cross-sectional Medicare data. Results may differ for other patients. Results may differ for longitudinal studies that examine patient outcomes over time.
Conclusions and Relevance
The new method accurately estimated treatment effects for comparing the effect of three treatment options on a binary outcome.
Future Research Needs
Future research could test the methods to analyze longitudinal data or treatments for different health conditions.
COVID-19-Related Study
Developing a New Statistical Method to Compare More than Two COVID-19 Treatments over Time Using Electronic Health Records
Results Summary
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
What was this COVID-19 study about?
Different treatment options are available for COVID-19. But questions remain about which treatment works best to treat COVID-19, especially to reduce the risk of death.
Researchers can use electronic health records, or EHRs, to compare COVID-19 treatments. But current statistical methods don’t work well when comparing more than two treatments over time. Current methods also don’t work well when patient traits, like health status, affect treatment choice or when the length of treatment differs for patients.
In this study, the research team created and tested a new method for comparing more than two treatment options for COVID-19 over time.
What did the research team do?
The research team first used a computer program to create test data that looked like real patient data. Then they used the test data to develop the new method. The team compared the new method with a current method for estimating the effects of treatments over time.
Next, the research team used the new method with EHR data from real patients. They studied how well four types of medicines worked to treat COVID-19. For each treatment, the team looked at the risk of death or a stay at the intensive care unit, or ICU, within 28 days of going to the hospital. The four treatments were:
- Remdesivir, a type of medicine that stops the virus from spreading in the body
- Corticosteroids, a type of medicine that reduces inflammation
- Dexamethasone, a type of corticosteroid
- Medicines to reduce inflammation other than corticosteroids
The research team also applied the new method to estimate the effect of remdesivir plus corticosteroids.
The EHR data included 11,286 adult patients with COVID-19. Of these patients, 29 percent were White, 25 percent were Black, 6 percent were Asian, and 40 percent identified as other race; 26 percent were Hispanic. The average age was 65, and 54 percent were men. All received care at the Mount Sinai Health System in New York City.
Patients and doctors helped design the study.
What were the results?
When the research team used the test data, they found that the new method worked better than the current method for estimating the effects of different treatments over time. The new method worked even when patients had fewer follow-up visits and the time between each follow-up was different.
With the real patient data, the new method made more precise predictions compared to other methods. The new method showed that patients who took remdesivir would have a lower risk of death or an ICU stay from COVID-19 than patients who took the other types of medicines. When comparing all treatments, the method predicted that patients who took remdesivir plus corticosteroids would have the lowest risk of death or an ICU stay.
What were the limits of the study?
The study took place at one health system. Results may differ in other settings.
How can people use the results?
Researchers can use these methods to predict the effects of different treatments for COVID-19 over time.
Professional Abstract
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
Background
COVID-19 is a rapidly evolving illness. Multiple options for treating COVID-19 are available, but questions remain about which treatment is best for managing the illness over time. Researchers can use observational data from electronic health records (EHRs) to study treatment outcomes. However, current statistical methods do not account for issues seen in the long-term treatment of COVID-19, including nonrandom treatment assignment and irregular follow-up time points. Also, current methods are limited in their ability to compare the effect of more than two treatment options on longitudinal outcomes.
Objective
To develop and test a new statistical method for comparing more than two COVID-19 treatments on patient outcomes over time
Study Design
| Design Element | Description |
|---|---|
| Design | Simulation studies; empirical analysis |
| Population | 11,286 patients ages 18 and older who were diagnosed with COVID-19 and received care at hospitals within the Mount Sinai Health System in New York City |
| Outcomes | ICU admission or in-hospital death |
| Data Collection Timeframe | February 25, 2020–February 26, 2021 |
The research team developed a new method based on a joint marginal structural survival model in continuous time (JMSSM-CT) that can be used to estimate the effect of longitudinal treatments on patient outcomes. The team conducted simulations to compare the new JMSSM-CT method with an existing discrete-time method (JMSM-DT) for estimating the effects of three different treatments with varying treatment durations.
The research team then applied the JMSSM-CT method to EHR data from patients receiving care at Mount Sinai Health System. The team estimated the effect of four COVID-19 treatment classes on patient intensive care unit (ICU) admissions or in-hospital deaths, whichever occurred first, up to 28 days after hospital admission. The treatment classes included remdesivir, dexamethasone, anti-inflammatory medications other than corticosteroids, and corticosteroids other than dexamethasone. The team also estimated the combined effect of using remdesivir with corticosteroids.
The study sample included 11,286 adult patients who tested positive for COVID-19 in New York City. Of these patients, 29% were White, 25% were Black, 6% were Asian, and 40% identified as other race; 26% were Hispanic. The average age was 65, and 54% were male.
Patients and clinicians provided input throughout the study.
Results
In the simulation analysis, the new JMSSM-CT method produced more accurate and less biased treatment effect estimates compared with the JMSM-DT method. The new method worked even as the number of follow-up events decreased and the time between follow-up events became unevenly spaced.
Of the four COVID-19 treatment classes, the new method estimated that remdesivir would have the lowest risk of an ICU admission or in-hospital death (log hazard ratio (HR)=-0.53; 95% confidence interval (CI): -0.75, -0.31), followed by dexamethasone (log HR=-0.20; 95% CI: -0.35, -0.06), and then anti-inflammatory medications other than corticosteroids and corticosteroids other than dexamethasone.
Among all treatment options, the new method estimated that remdesivir combined with corticosteroids would have the lowest risk of an ICU admission or in-hospital death (log HR=-0.74; 95% CI: -0.95, -0.52).
Limitations
The study took place at a single health system in New York City. Results may differ in other settings.
Conclusions and Relevance
The new JMSSM-CT method can accurately compare the effects of more than two COVID-19 treatment regimens administered over time.
Peer Review Summary
The Peer-Review Summary for this COVID-19 study will be posted here soon.
Final Enhancement Report
This COVID-19 study's final enhancement report is expected to be available by May 2024.
Final Research Report
This project's final research report is expected to be available by May 2024.
Journal Citations
Related Journal Citations
Peer-Review Summary
The Peer-Review Summary for this study will be posted here soon.