Results Summary
What was the research about?
A randomized trial is one of the best ways to learn if one treatment works better than another. Randomized trials assign patients to different treatments by chance. But they are not always affordable, and they take a long time to complete.
When randomized trials aren’t possible, researchers can use observational studies to learn how treatments work. In observational studies, researchers look at what happens when patients and their doctors choose the treatments. Traits such as age or health may affect treatment choices. These traits may also affect patients’ responses to treatment, making it hard to know if the treatment or the traits affected the patients’ responses.
Some study designs and statistical methods may help address this problem and make results from observational studies more useful. These methods can give researchers more data about whether treatments work and how the same treatment can affect groups of patients differently.
The research team conducted three studies to test different methods of designing and analyzing observational studies. They wanted to know if observational studies that used these methods produced results similar to randomized trials.
What were the results?
In study 1, the research team found that the methods for designing and analyzing observational studies had results similar to randomized trials for how well a treatment worked.
In study 2, the research team used statistical methods to divide patients into groups based on their risks of getting an illness. The methods helped show how patients who had high risk responded differently to the same treatment.
In study 3, some methods were better than others in finding out how the same treatment affects patients differently.
What did the research team do?
In the first study, the research team used methods to design observational studies to look like randomized trials. For example, they used data from health records to assess the effectiveness of two medicines for high blood pressure. Using different data from 11 randomized trials, the team then analyzed how effective the medicines were. The team compared the results from the observational study with the randomized trial results.
In the second study, the research team used randomized trial data to figure out patients’ predicted risk of getting an illness. Then the team looked at how groups of patients with different levels of predicted risk responded to the same treatment.
In the third study, the research team used a computer program to create data. They used the data to compare the different methods of finding out how the same treatment can affect patients differently. They also compared the methods using real data.
What were the limits of the study?
The methods used in this study may work only when data include patient traits such as age and other health problems.
Future research could test these methods using data from different data sources on different health problems and treatments.
How can people use the results?
Researchers can consider using these methods to design and analyze studies using observational data when randomized trials aren’t possible.
Professional Abstract
Objective
To evaluate whether methods for estimating treatment effects and heterogeneity of treatment effects (HTE) from observational data produce results consistent with evidence from randomized trials
Study Design
| Design Elements | Description |
|---|---|
| Design | Empirical analyses and simulation studies using randomized trials and observational data |
| Data Sources and Data Sets |
Study 1: randomized trials in public databases or provided by investigators; observational data from an integrated data set of claims, electronic health records, and registries Study 2: randomized trials in public databases or provided by investigators Study 3: simulated data, randomized trial and observational data |
| Analytic Approach | Treatment effectiveness and HTE were estimated from observational studies designed to emulate randomized trials, predictive modeling, risk stratification, outcome modeling followed by standardizations, inverse probability weighting, and doubly robust methods |
|
In this project, researchers conducted three studies to test different methods of designing and analyzing observational studies to estimate treatment effects and HTE. Treatment effects refer to differences in the outcome distributions when comparing two or more treatments applied to the same individuals. HTE means the treatment effect is not the same for all individuals.
In study 1, researchers compared treatment effects from observational studies with treatment effects from randomized trials of the same interventions. Using data from claims and electronic health records, researchers developed four observational studies to emulate one hypothetical randomized trial comparing the effectiveness of antihypertensive medicines. Researchers used multiple methods, including propensity score-based methods, to estimate treatment effects in each emulation. They then estimated treatment effects for antihypertensive medicines in a pooled analysis of 11 trials and compared results from the emulations with the pooled trial analysis.
In study 2, researchers evaluated a novel approach of baseline risk stratification to assess HTE. Researchers used data from 32 randomized trials and modeled the predicted outcome risk in each trial using covariates associated with the outcome. Based on the predicted outcome risk, researchers stratified the trial population into four strata of increasing predicted risk. Researchers then assessed HTE by comparing absolute and relative risk reduction across the four risk strata.
In study 3, researchers compared the performance of different HTE estimation methods using simulated and real data. Researchers simulated different treatment scenarios and patient subgroups. They assessed HTE using outcome regression, inverse probability weighting, and doubly robust methods.
Clinicians, patients, clinical practice guideline developers, regulatory authorities, and funders provided input on this study.
Results
Study 1. Compared with actual randomized trials of similar treatments, trial emulations using observational data produced similar estimates of treatment effects. For example, emulations comparing the effectiveness of antihypertensive medicines on various outcomes for end-stage renal disease produced similar risk differences (ranging from 1.2% to 1.8%) compared with the randomized trial estimate of risk difference (1.7%).
Study 2. Through risk stratification, researchers detected statistically significant HTE over predicted outcome risk. Absolute risk reduction varied substantially over predicted outcome risk and was generally greater in patients with higher risk. Factors affecting HTE detection included rates at which the outcome occurred and whether researchers assessed HTE on the absolute or relative scale.
Study 3. In simulation studies, estimation methods for treatment effects and HTE performed with negligible bias in moderate and large sample sizes. Doubly robust estimation outperformed inverse probability weighting and performed almost as well as outcome regression-based methods.
Limitations
Neither a limited set of empirical analyses or simulation studies can provide a complete understanding of the methods’ strengths and limitations. In addition, this study focused on observational studies that estimated intention-to-treat effects.
Conclusions and Relevance
Observational studies designed to emulate hypothetical randomized trials can produce estimates of treatment effects and HTE similar to results from actual randomized trials and may be preferable when randomized trials are infeasible or do not provide comprehensive covariate data.
Future Research Needs
The methods could be tested in evaluating per-protocol effects and dimensions of treatment effect variation beyond baseline risk. The methods could also be evaluated using different sources of data across additional clinical conditions and treatments.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers brought attention to several past reports that they suggested were relevant to this work and should be cited, particularly related to the validity of using observational studies to make causal inferences. The researchers noted that the idea of using observational data for experimental studies goes as far back as the 1940s at least and has been especially widely discussed in fields where randomizing study participants was not feasible for practical or ethical reasons. They added language to the report acknowledging the potential use of observational studies, particularly when there were ethical or methodological barriers to randomized controlled trials. However, the researchers said that they would not want to frame their views as recommendations because providing recommendations was not within the scope of this project.
- The reviewers questioned how the researchers estimated standard errors in the models they developed given the importance of estimating the standard error correctly when aiming to compare weighted with nonweighted study results. The researchers revised the report to describe the nonparametric bootstrap techniques they used to estimate standard errors. However, in their response to reviewers the researchers also noted that it would be useful to compare different methods to estimate standard errors to test the variance and validity of the estimates. The reviewers chose not to report all of the alternate approaches to standard errors because the number would be fairly large.
- The reviewers suggested the report address additional issues, for example, the implications of varying sample sizes, including using small samples in observational studies. The researchers added some discussion to the report in response to the various suggestions but on the issue of sample size, the researchers said that they believe that inference from observational data generally requires large amounts of data, especially to assess heterogeneity of treatment effects.