Results Summary
What was the project about?
Randomized controlled trials, or RCTs, look at how well treatments work. But people who take part in RCTs may differ from patients who receive care in clinics. For instance, patients who take part in RCTs may be less likely to smoke or may have fewer health problems. These differences can affect how well a treatment works. As a result, a treatment may work differently for a patient receiving care in a clinic than it did for patients who took part in the RCT.
Researchers can use statistical methods to account for differences in patient traits and behaviors. In this project, the research team developed and tested new methods to account for these differences. They used the methods to apply RCT results to patients receiving care in clinics.
What did the research team do?
The research team combined data from an RCT with data from patients receiving care in clinics. The team looked at two ways of combining the data:
- Nested design. The RCT data were inserted, or nested, within data from patients receiving care in clinics.
- Non-nested design. The team added data from patients receiving care in clinics to the RCT data.
The research team then created three types of statistical methods to analyze the data from both study designs. The team used the methods to look at how well the treatment would work for the patients receiving care in clinics. The team tested the accuracy of the methods using data created by a computer and data collected from patients.
Patients and healthcare providers helped advise the study team.
What were the results?
All three types of methods worked correctly if enough data were included from the patients receiving care in clinics. If enough data weren’t available, then the findings from the RCT couldn’t be applied to clinic patients.
What were the limits of the project?
The methods need further development if data from patients receiving care in clinics are incomplete or have errors. To use the methods, researchers need to make sure that they have data on the patient traits that may affect how well a treatment works.
Future studies could explore ways to use the methods with other health problems and different data sets.
How can people use the results?
Researchers can use these methods to apply RCT results to patients receiving care in clinics.
Professional Abstract
Background
Patients seen in clinical practice may differ from those represented in randomized controlled trials (RCTs). For example, they may have different distributions of characteristics such as comorbidities. As a result, doctors cannot be sure that treatments tested in RCTs will work the same for patients seen in clinical practice. Statistical methods that generalize inferences from RCTs can help overcome such limitations.
Objective
To develop methods for extending RCT results to patients in clinical practice
Study Design
| Design Element | Description |
|---|---|
| Design | Conceptual framework development; simulation studies; empirical analysis |
| Data Sources and Data Sets |
Data from an RCT, observational data from clinical practice, simulation data |
| Analytic Approach | Causal inference analysis in nested and non-nested study designs using different estimation methods based on outcome modeling, modeling probability of participation with inverse odds weighting, and a combination of outcome modeling and modeling the probability of participation |
| Outcomes | Mean of treatment outcomes such as average treatment effect, probability of trial participation, probability of treatment among randomized individuals, performance of different estimation methods using mean bias and variance |
Methods
Researchers developed and evaluated methods for extending RCT results to patients seen in clinical practice.
Researchers combined baseline covariate, treatment, and outcome data from an RCT with baseline covariate data from a patient population. Based on the RCT’s eligibility and participation criteria, researchers combined the data using two designs:
- Nested trial designs in which the RCT data were embedded in a random sample of data from individuals from the patient population
- Non-nested trial designs in which the RCT data were added to a random sample of data from individuals from the patient population
Under each study design, researchers specified identifiability conditions for extending inferences from RCTs to patients seen in clinical practice. Identifiability conditions are required conditions for determining average potential treatment outcomes in patients who did not participate in the RCT. For example, one condition is that the treatment effect found in the RCT stems from the randomized treatment and was not due to other factors such as trial participation itself.
Researchers then developed three types of statistical methods, called estimators, for estimating treatment outcomes and average treatment effects based on outcome modeling, modeling the probability of participation in the trial, and a combination of outcome modeling and modeling the probability of participation.
To evaluate the estimators’ performance, researchers conducted studies using simulations and empirical data. Researchers used sensitivity analyses to assess how the methods worked when the identifiability conditions did not hold.
Patients and healthcare providers provided input to the study team.
Results
All estimators were unbiased when the model was correctly specified and included sufficient covariate data from the patient population. The estimators that were based on outcome modeling had the lowest variance followed by the estimators that combined the outcome model and the probability of participation model. If the estimator was biased, inferences from the RCT could not be extended to patient populations.
Limitations
The methods do not work when identifiability conditions are violated. In such cases, researchers can use sensitivity analyses to examine the impact of violations of the conditions. The methods do not address missing covariate data in the patient population, measurement error, or clustering. Applying the methods requires collecting baseline covariate data from patient populations of interest.
Conclusions and Relevance
Researchers can use these methods for extending causal inferences from RCTs to patients seen in clinical practice.
Future Research Needs
Future research could test the methods with other health outcomes and diverse data sources.
COVID-19-Related Study
Summary
This study received additional funding in 2020 to quickly initiate new research related to COVID-19. The additional research is in progress. PCORI will post the research findings on this page once the results are final.
Nursing homes have seen high numbers of COVID-19 infections among residents. The enhancement to this project will use electronic medical records and claims data from a select population to build predictive models of individual infection and nursing home-level outbreaks that can be then applied to a nationally representative sample of nursing homes and nursing home residents. The resulting data can be used to select nursing homes for intervention, such as increased preventative measure.
Enhancement Award Amount: $193,653
Final Research Report
View this project's final research report.
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers questioned the applicability of the methods tested in this study. They noted that if a target population includes patients who would not have been eligible for the randomized trial, then the assumptions resulting from the randomized trial would not be generalizable to that target population. The researchers disagreed, stating that as long as the factors that made that target population ineligible were not factors that affected the trial assumptions, investigators could make inferences from the randomized trial to the target population.
- The reviewers noted that the report often refers to a target population in a way that readers might think that each study can only have one target population. The researchers agreed that this would be an incorrect assumption and revised the report to more clearly explain that the methods can be applied to any population, and that any one analysis would require the investigator to choose a specific group or population to test.
- The reviewers suggested that the researchers eventually provide an annotated version of the statistical codes from the appendix of this report on a public platform to make their methods more accessible. The researchers agreed and explained that they had already posted annotated statistical codes from this study on the GitHub.com platform.