Results Summary
What was the project about?
Researchers often use data from patients’ health records to compare treatments. But many things—not just treatments—affect patients’ health. To figure out whether changes in patients’ health result from treatment or something else, researchers can use statistical methods called instrumental variables, or IVs. IV methods account for factors that affect health but aren’t in patients’ health records, such as eating habits. Existing IV methods work well when looking at health outcomes that are measured using certain types of scales, such as blood pressure. But existing methods don’t work as well to measure the time until a health event occurs, particularly when an event, like death, has not occurred for many patients in the study.
In this study, the research team created and tested a new IV method to more accurately estimate how a treatment relates to the time until a health event.
What did the research team do?
The research team created 2SRI-F, a new IV method. The team then created a test data set to mimic patient health records. The test data set included information on patients’ health and treatments. Using the test data, the team compared findings using the new IV method with findings from existing methods.
The research team also tested the new method in three studies using data from patients with blood vessel diseases who had health records in the Vascular Quality Initiative, or VQI. Studies 1 and 2 looked at the effect of treatment type on time to death. Study 3 estimated the effect of treatment on time until having an amputation or until a patient had to start treatment again. In all three studies, the research team compared findings from the 2SRI-F method to those from existing methods.
Patients, physicians, and the VQI director helped the research team design the study.
What were the results?
Compared with existing methods, the 2SRI-F method had more accurate results. The results from the 2SRI-F method were more similar to results found in randomized trials. Researchers consider these trials the best way to compare how well treatments work.
What were the limits of the project?
Current computer programs cannot make estimates for the new methods that are as precise as researchers would like. These programs are not designed to run the 2SRI-F method.
Future research could improve computer programs to analyze data using the 2SRI-F method.
How can people use the results?
Researchers can use the results when designing studies using data from patient health records.
Professional Abstract
Background
Observational studies often use secondary data sources, such as clinical registries, to compare treatments. Without randomization, observational studies cannot account for unmeasured confounders, which affect treatment choices and outcomes but are not available in the data. Unmeasured confounding can lead to biased results.
Use of instrumental variable methods can overcome unmeasured confounding. Instrumental variables (IVs) are related to treatment choice but only affect the outcome through the effect of treatment. IV methods, which mimic randomized treatment assignment, are well developed for linear models, but not for modeling survival time and censored survival data.
Objective
To develop, test, and apply IV methods for survival analysis using the Cox proportional hazard model
Study Design
| Design Element | Description |
|---|---|
| Design | Simulation analysis, empirical analysis |
| Data Sources and Data Sets |
|
| Analytic Approach |
Comparing Cox models that do not address unmeasured confounding and a new IV-based method developed as part of this research in
|
| Outcomes |
|
Methods
Researchers developed the two-stage residual inclusion-frailty (2SRI-F) method to estimate hazard ratios from Cox models. To address unmeasured confounders, 2SRI-F emulates the two-stage residual inclusion (2SRI) procedure by using the residuals, or the differences, between the actual treatment status and the predicted probability of receiving the treatment in addition to the IVs as predictors for estimating treatment effects. However, 2SRI-F also accounts for the fact that the residuals are imperfect surrogates for unmeasured confounders; which in turn better enables the substituted residuals to account for unmeasured confounding effects. To evaluate the performance of these IV methods, researchers conducted Monte Carlo simulations testing different scenarios.
Using patient data from the Vascular Quality Initiative (VQI) registry, researchers conducted three studies to test the performance of 2SRI-F. In study 1, researchers linked Medicare claims data to VQI registry data from patients who underwent carotid endarterectomy or carotid stenting between 2003 and 2016. They compared mortality hazard ratios estimated from the 2SRI-F method with those from existing methods. Study 2 examined the effect of intervention type on time to death from any cause in patients with an abdominal aortic aneurysm. Study 3 estimated the impact of endovascular atherectomy versus traditional peripheral vascular interventions on the time to any type of amputation or any type of reintervention.
A nine-member panel including patients, physicians, and the VQI director provided input and helped design the research plan.
Results
Simulation analysis. Cox models that ignored unmeasured confounding produced greater bias than 2SRI-F in all scenarios. The 2SRI-F method reduced bias the most compared with the 2SRI method and Cox models when unmeasured confounders had strong effects.
Empirical analysis. Compared with other methods that do not address unmeasured confounders, the 2SRI-F method yielded results more similar to benchmark estimates from published randomized clinical trials. This finding suggests that the 2SRI-F method may provide less biased estimates of time-to-event outcomes in observational analysis.
Limitations
Existing statistical software for estimating survival analysis models containing frailties may undermine the performance of the 2SRI-F method. Researchers must identify candidate IVs and cannot fully validate them using empirical data alone. Suitable IVs, including those having a strong association with treatment choice, may be difficult to find.
Conclusions and Relevance
The 2SRI-F method may improve the validity of results for time-to-event outcomes in observational studies.
Future Research Needs
Future research could reduce computational burden and improve the performance of the 2SRI-F method.
Final Research Report
View this project's final research report.
Journal Citations
Results of This Project
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers noted that other methods would be useful to examine via simulations, and that the researchers should present these other methods or discuss them at the end of the report. The researchers responded that their principal comparison approaches focused on the Cox (proportional hazards) regression model because it is heavily used and therefore, the report would be of more interest to readers. The reviewers pointed out that the researchers mentioned less commonly used methods, including the accelerated failure time model. In response to reviewer comments, the researchers added references to this model in their discussion section.
- For the simulation results, the reviewers suggested adding information about root mean square errors (RMSE), which is a way of comparing predicted values against observed values. The researchers acknowledged that they did not report RMSEs for all simulations because they considered bias to be a primary concern, with statistical precision being secondary. The researchers did add appendix tables with results that provided RMSE for the simulations.
- The reviewers asked for additional details to allow readers to replicate simulation results after one reviewer failed to replicate the results. The researchers explained that they believed the simulations the reviewer performed did not work because the reviewer did not run analyses for the Cox model. The researchers noted that their code is freely available on GitHub and use of this code should allow for perfectly reproduced simulations.