Objective
To compare data-adaptive algorithmic approaches for improving confounding control in comparative effectiveness research that uses electronic healthcare databases
Study Design
| Design Elements |
Description |
| Design |
Empirical analyses and simulation studies |
| Data Sources and Data Sets |
Novel Oral Anticoagulant Prescribing Study data set, Nonsteroidal Anti-inflammatory Drugs Study data set, and Vytorin Study data set |
| Analytic Approach |
Empirical analyses and simulations to examine 4 different algorithmic approaches: hdPS alone, a combination of super learner prediction modeling and the hdPS, a combination of a scalable version of CTMLE and the hdPS, and a combination of penalized regression (lasso) and the hdPS |
| Outcomes |
Outcomes informing variable selection, PS estimation, and causal inferencey
|
Analyzing data from electronic healthcare databases, such as electronic health records, to gain generalizable knowledge of the effectiveness of medical interventions in routine care can improve patient care and outcomes, particularly for populations that are often excluded from randomized trials. However, researchers underuse these data for generating evidence on treatment effects, in part because of concerns about bias. Bias may arise in the data because clinicians selectively base their prescribing decisions on such factors as disease severity and patient prognosis. Current approaches to minimize such bias rely on the investigator to specify all potential confounding factors. New analytic approaches propose using algorithms to maximize control of confounding factors. However, researchers do not know how well these algorithms perform when applied to electronic healthcare data, particularly for special populations and small samples. Further, researchers have lacked readily available software to facilitate use of the algorithms.
This study evaluated the performance of several algorithms for variable selection, propensity score (PS) estimation, and causal inference. Researchers performed simulations using the plasmode framework, which combines simulated and empirical data to more accurately reflect complex relations that typically exist among baseline covariates. The research team then used three healthcare data sets in conjunction with plasmode simulations to evaluate the ability of each algorithm to effectively control for confounding. The team considered different scenarios by varying outcome incidence, treatment prevalence, sample size, and treatment effect. To help researchers use the algorithms, the team developed software and accompanying guidance.
During the study, the research team met with patient representatives, who identified key problems they encounter in healthcare delivery and provided input on what potential research questions are of greatest interest.
Results
- Overall, in settings with many covariates, the high-dimensional propensity score (hdPS) algorithm alone performed as well as or better than other algorithms for automated variable selection. However, the hdPS algorithm can be sensitive to the number of covariates included for adjustment, and severe overfitting of the PS model can negatively affect the properties of effect estimates, particularly for small samples.
- Combining the hdPS algorithm and the scalable version of collaborative targeted maximum likelihood estimation (CTMLE) performed well for many of the scenarios considered, but this combination was sensitive to parameter specifications within the algorithm.
- Combining the hdPS algorithm with super learner prediction modeling performed well across a broad range of settings and conditions and was the most consistent strategy in terms of reducing bias and mean squared error in the effect estimates. This approach seems especially promising for use in early periods of drug approval, where small samples and rare exposures are common.
Limitations
A framework that incorporates empirical data and observed variable relationships with simulated data allowed the research team to evaluate the algorithms in settings that reflect real-world practice. However, simulations that use specific data may have limited generalizability to other settings. In addition, assumptions made in applying the super learner algorithm may have influenced results. Finally, this study relied solely on statistical algorithms to control confounding without investigator input. It is unclear whether investigator input might influence results.
Conclusions and Relevance
Among the data-adaptive algorithms assessed in this study, no single algorithm was optimal across all data sets and scenarios. The widely used hdPS functioned well in most scenarios, although the combination of the hdPS with super learner prediction modeling outperformed hdPS alone under certain conditions. These approaches may be effective in reducing bias due to confounding when estimating treatment effects using healthcare databases.
Future Research Needs
Future studies could determine whether similar findings emerge when investigators identify confounders to include in the model a priori. Additional research could also explore factors that influence the performance of these algorithms.
