Results Summary
What was the project about?
In randomized controlled trials, or RCTs, researchers assign patients by chance to different treatments to see how well they work. But patients who take part in RCTs may differ from patients receiving care at clinics. For example, they may have fewer health problems. Also, outside of RCTs, the treatment dose or the length of treatment patients receive may differ. These differences can affect how well a treatment works. As a result, researchers can’t be sure that treatments tested in RCTs will work the same for patients at clinics and hospitals.
Hybrid studies help researchers apply RCT results to patients outside a study. Hybrid studies combine data from RCTs with data from patients’ health records. They use statistical methods to calculate how RCT results apply to other patient groups.
In this project, the research team created new methods to design and analyze hybrid studies to apply RCT results to patients receiving care at clinics.
What did the research team do?
The research team first designed a hybrid study to compare two colon cancer treatments. They wanted to estimate the effect of treatment on the risk of neuropathy, a type of nerve damage, and on the risk of dying. The study combined RCT data and health record data from cancer clinics. To identify which traits were important to include in analyses, the team developed four data visualization methods. The methods created graphs that showed the differences in traits between the two groups of patients.
Next, the research team created a new method for analyzing hybrid studies. To test the method, the team used a computer program to create different treatment scenarios. These scenarios changed the number and dose of the two cancer treatments over time. The team applied the data from the graphs to look at how these changes affected risk of neuropathy and death.
Patients, doctors, health system leaders, and health insurers helped plan the study.
What were the results?
The graphs showed that patients in the RCT and at clinics were similar in age and sex. Compared with patients in the RCT, more patients at clinics were a race other than White (14 percent versus 3 percent). They also had an earlier stage of cancer (16 percent versus 7 percent) and a higher body mass index (33 percent versus 12 percent).
The new method helped apply RCT results to patient groups at clinics. The method showed that patients who received more treatment doses would have a lower risk of death within five years but a higher risk of neuropathy.
What were the limits of the project?
The research team tested the method with two treatments for colon cancer. The method may need to be adapted for other health problems.
Future studies could test these methods with other health problems.
How can people use the results?
Researchers can use these methods to apply RCT results to patient groups at clinics.
Professional Abstract
Background
Patients in clinical practice may differ from those who participate in randomized controlled trials (RCTs). For example, they may have different characteristics, such as their age or comorbidities, or they may have different treatment dose or duration than patients in an RCT. As a result, researchers cannot be sure that treatments tested in RCTs will work the same for patients in clinical practice.
Hybrid studies can help address these issues by combining data from RCTs and health records so that the results from RCTs better apply to patients in clinical practice. However, researchers lack practical guidance and analytic methods for conducting hybrid studies that account for variations in treatment dose and duration seen in clinical practice.
Objective
To develop methods to improve hybrid study design and analysis that can account for treatment differences in clinical practice
Study Design
| Design Element | Description |
|---|---|
| Design | Empirical analysis |
| Data Sources and Data Sets |
|
| Analytic Approach | Data visualizations; parametric g-formula; inverse odds of sampling weights |
| Outcomes | Risk of 5-year mortality, treatment-related risk of neuropathy |
Methods
First, the research team designed a hybrid study to compare the safety and effectiveness of two colon cancer therapies, FOLFOX and 5FU. The study combined patient data from the Multicenter International Study of Oxaliplatin/5-Fluorouracil/Leucovorin in the Adjuvant Treatment of Colon Cancer (MOSAIC) RCT, and electronic health record (EHR) data from patients receiving care at oncology clinics.
The research team used data visualization methods to identify a set of patient characteristics for conducting hybrid study analyses. Each visualization evaluated statistical assumptions required for extending the RCT results to patients at clinics. Using patient characteristics from the visualizations, the team calculated inverse odds of sampling weights. They developed a new analytic method that combined the inverse odds of sampling weights with a statistical approach called the parametric g-formula. The method accounted for time-varying confounders, such as treatment discontinuation and adverse events.
The research team then applied the method to analyze the effect of different FOLFOX and 5FU treatment strategies on patients’ risk of five-year mortality and neuropathy. They tested whether the method could apply RCT treatment effects to clinic patients under five scenarios, each describing a different treatment strategy.
Patients, doctors, health system administrators, and health insurers helped design the study.
Results
The data visualizations showed that, compared with patients in the RCT, patients at oncology clinics were similar in age and sex. But compared with patients in the RCT, more patients at oncology clinics were a race other than White (14% versus 3%), had an earlier stage of cancer (16% versus 7%), and had a higher body mass index (33% versus 12%).
The new method helped extend treatment effects from the RCT to patients at oncology clinics. Based on patients’ characteristics, the method estimated an increased risk of neuropathy for these patients with more treatment cycles of FOLFOX, but the risk of five-year mortality decreased.
Limitations
The research team tested the method with two treatments for one condition. The method may require refinement for use with other conditions.
Conclusions and Relevance
Researchers can use these methods to design hybrid studies to extend results from RCTs to clinical practice.
Future Research Needs
Future research could test the methods with other conditions.
COVID-19-Related Study
Using Visual Methods to Apply Results from COVID-19-Related Randomized Controlled Trials to Patients Receiving Care at Clinics and Hospitals
Results Summary
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
What was this COVID-19 study about?
In randomized controlled trials, or RCTs, researchers assign patients by chance to different ways to prevent or treat illness, such as COVID-19. But patients with COVID-19 who take part in RCTs may have different traits than patients with COVID-19 in clinics or hospitals. For example, patients may differ in age, severity of COVID-19, or whether they have other health problems. It can be hard for doctors to know if the results from an RCT apply to these patients.
In this study, the research team used new methods to see if traits for patients in RCTs differed from the traits of patients in clinics and hospitals. Using the new methods, the team created graphs to show the differences in traits.
What were the results?
The research team identified 149 RCTs related to COVID-19. The team looked at which patient traits researchers reported as part of the RCT results.
Of the 149 RCTs:
- All reported patients’ age.
- 99 percent reported patients’ sex.
- 34 percent reported patients’ race.
- 74 percent reported if patients had other health problems, but only 3 percent reported problems that affected the lungs and breathing.
The graphs showed how traits differed between patients in the RCTs and patients in clinics and hospitals. Compared with patients in clinics and hospitals:
- Fewer women and Black patients took part in the RCTs.
- Patients who took part in the RCTs were older.
Because few RCTs reported patients’ race or whether patients had problems affecting the lungs, it may be hard to know whether RCT results apply to patients receiving care in clinics and hospitals.
What did the research team do?
The research team searched for articles about RCTs related to COVID-19 between October 2020 and June 2021. They found 137 articles that had results from 149 RCTs. The team summarized the patient traits reported in the RCTs, including patients’ age, sex, race, and other health problems.
Next, the research team used the graphs to see how the traits of patients in the RCTs differed from the traits of patients receiving care in clinics and hospitals. To look at patients in clinics or hospitals, the team used data from the North Carolina Department of Health and Human Services for patients diagnosed or hospitalized with COVID-19 through June 2021.
Doctors helped plan the study.
What were the limits of the study?
The research team used data from patients diagnosed with COVID-19 in one state. Results may differ in other states.
How can people use the results?
Researchers can use the methods to see how patients in RCTs related to COVID-19 differ from patients with COVID-19 in clinics and hospitals.
Professional Abstract
In response to the COVID-19 public health crisis in 2020, PCORI launched an initiative to enhance existing research projects so that they could offer findings related to COVID-19. The initiative funded this study and others.
Background
Patients diagnosed with COVID-19 in clinical practice may differ from those participating in randomized controlled trials (RCTs). For example, they may have different characteristics, such as age, severity of COVID-19, or other health conditions. As a result, doctors cannot be sure if approaches to prevent or treat COVID-19 that are tested in RCTs will work the same for patients receiving care in clinical practice. New ways of identifying differences in patients in RCTs and patients in clinical practice could help researchers understand how the results of COVID-19-related RCTs apply to other patient populations.
Objective
To use data visualization methods for applying COVID-19 RCT results to patients diagnosed with COVID-19 in clinical practice.
Study Design
| Design Element | Description |
|---|---|
| Design | Systematic literature review; descriptive analysis; data visualizations |
| Population | Patients with COVID-19 from the NCDHHS |
| Outcomes | Patient data from RCTs compared with data from patients in the NCDHHS |
| Data Collection Timeframe | Systematic review: October 2020–June 2021 NCDHHS: Data as of June 23, 2021 |
The research team adapted data visualization methods they developed in a previous study for use with aggregate data. They expanded the methods to help identify overlapping characteristics for patients in RCTs and patients in clinical practice, and to help evaluate statistical assumptions for generalizing results from RCTs to clinical practice.
The research team conducted a systematic review of 137 articles presenting results from 149 COVID-19-related RCTs from October 2020 to June 2021. They identified characteristics of patients taking part in these RCTs and conducted a descriptive analysis of patients’ age, sex, race, and health conditions.
Next, the research team used the data visualization methods to compare patients in the COVID-19-related RCTs with patients who were diagnosed with COVID-19 in clinical settings. The team used aggregated data reported as of June 2021 from the North Carolina Department of Health and Human Services (NCDHHS) to identify characteristics of patients who were diagnosed or hospitalized with COVID-19 in North Carolina. They then compared aggregate patient data from the RCTs with the NCDHHS data.
Infectious disease and critical care clinicians helped design the study.
Results
Systematic review. Of the 149 RCTs included in the systematic review:
- 100% reported patients’ age.
- 99% reported patients’ sex.
- 34% reported patients’ race.
- 74% reported whether patients had health comorbidities, but only 3% reported on respiratory comorbid conditions.
Data visualization. Data visualizations showed that compared with patients in the NCDHHS data set, fewer women and Black patients were in RCTs. Also, patients in RCTs were older.
Limitations
The methods compared patient data from North Carolina with aggregate data from 149 RCTs. The differences found may not be generalizable to patients diagnosed with COVID-19 in clinical settings across the United States.
Conclusions and Relevance
Data visualization methods can help researchers see differences between patients with COVID-19 in RCTs and patients in clinical practice. However, limited reporting on race and respiratory-related comorbid conditions in RCTs poses a challenge for applying results from RCTs to patients at risk for severe illness from COVID-19.
Peer Review Summary
The Peer-Review Summary for this COVID-19 study will be posted here soon.
Final Enhancement Report
View this COVID-19 study's final enhancement report.
DOI - Digital Object Identifier: 10.25302/08.2023.ME.2017C39337
Final Research Report
This project's final research report is expected to be available by February 2024.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers were concerned that the researchers understated the scope of the real-world assumptions required to generalize trial results to broader populations. The researchers expanded their discussion of the relevant assumptions, stating first that the assumptions needed to generalize results required high-quality data and that the trial and real-world data used the same measurement tools.
- The reviewers asked for more information on the stakeholder engagement experiences in developing this project. The researchers expanded their description of patient and stakeholder engagement in this study, describing findings from interviews they conducted with their patient partner as well as oncologist, payer and regulator stakeholders. The researchers explained how the information they learned from these discussions impacted the study conduct and dissemination.
- The reviewers asked the researchers to provide some summary points throughout the report that would be more readily understandable by non-statisticians who might be interested in learning more about the methods presented in this project. The researchers added summary boxes and other ways of encapsulating their results to be more understandable to clinical researchers without the methodological expertise.
- The reviewers noted that there was insufficient attention in the report to the topic of race and ethnicity despite the fact that homogeneity in the participants for a randomized controlled trial is a motivator for the methodological approach used in the current project. The researchers added a discussion of the potential for race and ethnicity to be effect modifiers in these types of trials, explaining that these constructs could not be measured in the current study because the recruitment sites were in 20 different countries, so race and ethnicity would not be defined the same way everywhere. The researchers acknowledged that racism and discrimination could affect the study results, but do so differently in each country.