Project Summary
One of PCORI’s goals is to improve the methods that researchers use for patient-centered outcomes research. PCORI funds methods projects like this one to better understand and advance the use of research methods that improve the strength and quality of comparative effectiveness research.
This research project is in progress. PCORI will post the research findings on this page within 90 days after the results are final.
What is the project about?
In observational studies, researchers look at what happens after patients and their doctors choose treatments. In these studies, patients with different traits—for example, older or younger patients—may choose different treatments. As a result, researchers may have a hard time knowing if it was the treatment that affected patient health outcomes, or whether patient traits affected the outcomes. Statistical methods can help address this problem.
Propensity score, or PS, methods can help researchers conduct observational studies. When looking at outcomes for different treatments, researchers can use PS methods to balance treatment groups. That is, they can account for differences in traits among patients who get different treatments and improve understanding of whether effects are due to the treatment or patients’ traits. But by creating more balanced, and often smaller, treatment groups, PS methods can decrease precision, making results less certain.
Subgroup analyses show how the same treatment affects groups of patients differently. For example, a treatment might work better for younger patients than older ones. PS methods can be used to study subgroups, but standard PS methods may not work well in subgroups.
In this study, the research team is developing new methods for testing balance and precision when planning subgroup analyses. The team is also developing new PS methods for conducting subgroup analyses.
How can this project help improve research methods?
Results may help researchers get information about what treatments work best for different groups of patients that is more accurate and precise.
What is the research team doing?
First, the research team is developing methods researchers can use to assess balance and precision when designing subgroup analyses. The methods detect and solve potential problems that could arise in subgroup analyses. Second, the team is developing new PS methods that use machine learning to improve balance and precision. In machine learning, computers use data to learn how to perform different tasks with little or no human input. Third, the team is comparing the new methods to existing methods to determine which methods are best for conducting subgroup analyses. Finally, the team is testing the new methods using data on treatments for women with uterine fibroids.
Research methods at a glance
| Design Elements | Description |
|---|---|
| Goal |
|
| Approach |
Simulation, machine learning, propensity scoring, subgroup analyses |
COVID-19-Related Study
Methods to Improve Comparative Effectiveness Analyses for COVID-19 Treatments
Results Summary
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
What was the project about?
Researchers can use data from health records to find out which treatments work better for COVID-19. But it can be hard to know whether changes in a patient’s health are from the treatment or something else, like patient traits such as their age or other health problems.
To address this problem, researchers can use a statistical method known as propensity scoring, or PS. PS helps researchers group patients based on their traits to compare how treatments work. But analyses that use PS may not be accurate. Errors may occur if PS misses some patient traits in creating patient groups. Researchers need better methods for using PS.
In this project, the research team created new methods to improve the use of PS for comparing how well COVID-19 treatments work for different patients.
What did the research team do?
The research team created methods to help researchers find out how likely it is for a PS analysis to have errors. The team designed two tools to help researchers use the methods: a study design assessment and a computer program.
Next, the research team tested the tools in two analyses of how well a medicine called dexamethasone worked to treat COVID-19. They looked at health record data from:
- Patients with COVID-19 who were admitted to the hospital. The team compared the risk of death or discharge to hospice for patients who did and didn't receive dexamethasone.
- Patients with COVID-19 who came to the hospital but weren't admitted. The team compared the risk of future hospital admission for patients who did and didn't receive dexamethasone.
In both analyses, the research team used the new methods and tools to check for errors in how they used PS to create patient groups for analysis.
Doctors helped design the study.
What were the results?
Improving use of PS methods. The study design assessment helped the research team know if errors were likely in the PS analysis. The computer program created graphs that showed whether errors in using PS analysis were likely when comparing patient groups.
Comparing patients with COVID-19 who did and didn’t receive dexamethasone. Results from both analyses suggest that:
- For patients admitted to the hospital, dexamethasone was helpful among those who stayed in the hospital for at least two days.
- For patients who came to the hospital but weren’t sick enough to be admitted, dexamethasone could be harmful.
What were the limits of the project?
The new methods don’t account for errors that happen if health records don’t include data that affect treatment decisions, like patient preferences.
How can people use the results?
Researchers can use the new methods to reduce errors when using data from health records to find out how well COVID-19 treatments work for patients with different traits.
Professional Abstract
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
Background
Propensity score (PS) methods can be useful for researchers conducting observational studies to compare treatments for COVID-19. PS methods create comparable patient subgroups that enable researchers to distinguish treatment effects from confounding factors, such as patient characteristics like age or co-occurring conditions. However, with unmeasured confounding factors, creating comparable subgroups is difficult and can lead to biased findings about treatment effectiveness. Researchers need new methods that improve the accuracy of PS methods for subgroup analyses.
Objective
To develop methods that improve the design of PS-based subgroup analyses for comparing treatments for COVID-19
Study Design
| Design Element | Description |
|---|---|
| Design | Methods development, observational cohort study, empirical (comparative effectiveness) analysis |
| Data Sources and Data Sets |
|
| Analytic Approach |
PS modeling to adjust for confounding factors:
|
| Outcomes |
|
Methods
The research team developed two approaches to improve PS-based subgroup analyses in observational studies. First, they adapted a tool to assess risk of bias in the study design. Second, they developed software to visually compare covariate balance across patient subgroups. Next, the team tested the approaches using multicenter data in two observational studies that used PS methods to evaluate outcomes for patients who received dexamethasone.
The first analysis used Hospital Consortium of America (HCA) electronic health record data from 176 facilities across the United States. The research team compared starting dexamethasone within two days of hospital arrival for COVID-19 versus not starting it within two days on in-hospital mortality or discharge to hospice. They examined whether outcomes varied based on receipt of oxygen support overall and within subgroups based on patient characteristics and confounding factors.
The second analysis used Change Healthcare (CHC) claims data. The research team compared receiving dexamethasone versus not receiving it on subsequent hospitalization in patients with COVID-19 who arrived at the hospital and left the same day.
For both analyses, the research team evaluated the risk of bias and used the software to examine the accuracy of PS methods in comparing treatment effectiveness across different subgroups.
Clinicians helped design the study.
Results
In both analyses, the new methods helped measure the risk of bias in the study design. The software’s visual plots showed the covariate balance across patient subgroups and helped adjust PS weights to improve the accuracy of the analysis.
In the HCA analysis, dexamethasone was associated with lower risk of inpatient mortality or discharge to hospice for patients who did not receive oxygen (odds ratio [OR]=0.90; 95% confidence interval [CI]: 0.78, 1.03) and patients who received oxygen (OR=0.92; 95% CI: 0.86, 0.98). In the CHC analysis, dexamethasone was associated with a 1.2% higher rate of hospitalization in patients who arrived at the hospital and left the same day; this finding was not statistically significant.
Results from both analyses suggest that dexamethasone was moderately helpful for patients who stayed at the hospital for at least two days, but potentially harmful for patients who did not require hospital admission.
Limitations
The new methods do not account for errors due to lack of information in the data sources, such as patient preferences, which may affect treatment decisions.
Conclusions and Relevance
The assessment and software optimize the use of PS methods to accurately compare treatments for COVID-19.
Peer Review Summary
The Peer-Review Summary for this COVID-19 study will be posted here soon.
Final Enhancement Report
This COVID-19 study's final enhancement report is expected to be available by July 2024.