Results Summary
What was the project about?
Comparative effectiveness research compares two or more treatments to see which treatment works better for which patients. Such research may include
- Randomized controlled trials, or RCTs. Researchers assign patients to a treatment by chance. Researchers consider RCTs to be the best way to figure out when changes in patients’ health result from the treatment.
- Observational studies. Researchers study what happens when patients and their doctors choose treatments. Patient traits, such as age or health, may affect treatment choices. These traits may also affect patients’ responses to treatments. Determining whether a patient’s traits, the treatment, or a mix of the two affected how well the treatment worked may be difficult.
In observational studies, researchers use statistical methods to help find out whether changes in patients’ health result from treatment or something else. Existing methods work well when studies look at whether treatment affects the risk of a health event, such as a heart attack. In these cases, researchers can compare how often patients had heart attacks before and after patients receive treatment. But existing methods don’t work well when studies look at the risk of a one-time event, such as death.
In this study, the research team tested a new statistical method for observational studies called posttreatment event rate ratio, or PTERR, that helps figure out whether a treatment reduces the risk of death.
What did the research team do?
To see how well PTERR worked, the research team used a computer program to make a test data set. The test data set looked like data from observational studies.
Then the research team tested PTERR using data from three sets of observational studies that compared medicines for different health problems. The team compared PTERR results with previous RCT results to see if the results were similar.
What were the results?
Results for PTERR varied across the set of studies, which means PTERR worked well in certain cases but not in other cases. For one set of studies, PTERR results were similar to RCT results. For the second set, PTERR results differed from RCT results. In the third set, PTERR results were similar to RCT results only for patients ages 50 to 60.
What were the limits of the project?
Patient traits, such as age, differed in some RCTs and observational studies. In these cases, comparing PTERR and RCT results didn’t work.
Future research can continue to look at statistical methods that help to figure out whether the treatment reduces the risk of death in observational studies.
How can people use the results?
Researchers can use PTERR in some observational studies to find out whether the treatment reduces the risk of a one-time event, such as death.
Professional Abstract
Background
Without randomization, observational studies cannot account for unmeasured confounders that affect treatment choices and outcomes but are not available in the data. Unmeasured confounding can lead to biased results.
Use of the prior event rate ratio (PERR) can overcome unmeasured confounding by accounting for the rate of prior episodes of an outcome, such as heart attack, as a proxy for the underlying risk of the outcome. However, PERR does not work for outcomes that are one-time events, such as mortality.
Objective
To examine a new method, the posttreatment event rate ratio (PTERR), to address unmeasured confounding in observational studies with mortality as the study outcome
Study Design
| Design Element | Description |
|---|---|
| Design | Empirical analysis and simulation studies |
| Data Sources and Data Sets | Data from UK Health Improvement Network database and UK General Practice Research Database used in three sets of previous studies |
| Analytic Approach | Cox proportional hazard model, PTERR |
| Outcomes | Bias, comparison of hazard ratios of mortality between PTERR approach and previous RCTs |
Methods
In this study, researchers examined the PTERR to address unmeasured confounding in studies with mortality as the outcome. Using data from the period following the end of treatment, PTERR calculates adjusted hazard ratios by comparing mortality risk for study participants who did and did not receive treatment.
First, researchers showed the assumptions under which the PTERR could alleviate unmeasured confounding. To test the sensitivity of the PTERR estimates to the statistical assumptions, researchers conducted simulation studies using generated data with different baseline mortality rates.
Then researchers used data from three sets of previous observational studies: thiazolidinedione studies, angiotensin-converting enzyme inhibitor (ACEI) studies, and the Women’s Health Initiative (WHI) intact uterus and hysterectomy studies. Researchers examined mortality for patients who did and did not receive treatment during two periods: from the start of the study until patients receiving treatment stopped it, and from the time patients stopped treatment until a predefined end date or death. Researchers compared PTERR mortality hazard ratios from the observational studies with mortality hazard ratios from randomized controlled trials (RCTs) to determine if estimates were similar.
Results
Simulation results. With the PTERR method, the bias remained small (p<0.03) across simulations with varying degrees of baseline mortality. However, bias increased with increasing magnitude of unmeasured confounding and higher baseline mortality rate.
Empirical results. Estimates from PTERR and RCTs varied in the three sets of studies.
- Thiazolidinedione studies found that PTERR hazard ratios were significantly different from the previous RCT estimates.
- ACEI studies indicated no significant differences between PTERR hazard ratios and hazard ratios from previous RCTs.
- The PTERR hazard ratios for both WHI studies were significantly different from hazard ratios from RCTs when restricting the sample to ages 50 to 70 but were not significantly different when restricting the sample to ages 50 to 60.
Limitations
Comparing results from PTERR estimations and RCTs was not feasible when patient characteristics in the samples were different. In addition, when statistical assumptions are not met, the PTERR method can lead to significant bias. For example, PTERR estimates can be biased when unmeasured confounding effects change between the two periods.
Conclusions and Relevance
PTERR can address unmeasured confounding in observational studies with mortality as the study outcome if statistical assumptions are met.
Future Research Needs
Future research could develop methods that are less sensitive to the assumptions of the PTERR.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The original principal investigator of this project did not work on the Final Research Report. Another investigator, Michelle Denburg, MD, MSCE, completed the project and prepared the Final Research Report for peer review. For this reason, peer reviewers focused their comments on aim 3 of the project. The methods and results from aim 1 of this project have been published.
- The reviewers commented that the analyses used in aim 3 of the project did not in fact involve an adaptation of the prior event rate ratio (PERR) as originally described. The statistical methods were in fact conventional methods to account for measured confounders only. The researchers conceded that they did not use an adaptation of PERR for this aim to account for unmeasured confounders in the analyses because the outcome involved a change over time rather than a single event. However, the researchers stated that they applied the key PERR assumption that the ratio between outcome event rates in the treatment-exposed group compared to the treatment-unexposed group prior to the start of the exposure interval should incorporate the effect of all confounders. Similarly, although the researchers did not directly use the posttreatment event rate ratio (PTERR) that they developed in the aim 3 analyses of composite renal outcome, they stated that they applied PTERR assumptions to their analyses. They applied these assumptions in that they contrasted the drug exposure periods with the unexposed periods and use those contrasts to estimate the effect of unmeasured confounders. They revised the description of aim 3 to align with these peer-reviewer observations.
- The reviewers noted that the original PERR adjustment method is biased and asked why the researchers did not use the revised PERR-alt method, which they considered unbiased, in their aim 1 analyses. The researchers explained that the PERR-alt method cannot be directly applied to the outcome of death because death can only happen once, and the PERR-alt method requires that an outcome can occur in both time periods assessed. Instead, the researchers applied the PTERR method in this study.
- The reviewers noted that the study considered event rates before treatment when adjusting for baseline differences between treatment groups. However, many epidemiological studies exclude people with previous occurrences of an outcome so they can focus on new-onset disease and because patients with new-onset disease often have different risk factors than patients who have a subsequent event, rather than a new one. The researchers acknowledged that PERR may not be the best analytic strategy for studying new-onset disease. However, they said that mixing the concepts of new-onset disease and secondary prevention allows results to be generalizable to a broader population.