What was the project about?
In randomized controlled trials, or RCTs, researchers assign patients by chance to different treatments to compare the benefits and harms. In RCTs, researchers have a high level of control over how patients receive treatment. RCTs often take place in research clinics with staff who monitor how patients follow treatment plans.
Pragmatic RCTs, or pRCTs, take place where patients typically receive treatment, such as a regular clinic. pRCTs can help capture the real-world effects of treatment but determining whether a treatment works can be hard in pRCTs. Also, no clear guidelines exist about how to collect and analyze data from pRCTs. Some kinds of analysis are better for helping researchers focus on what’s important to patients.
In this study, the research team created guidelines for collecting and analyzing data in pRCTs so that results reflect what matters to patients and researchers.
What did the research team do?
First, the research team asked patients and researchers what results from pRCTs matter to them. The team had three group discussions with adult patients. The team also did five interviews and an online survey with pRCT researchers. Then the team reviewed results from published pRCTs. They looked at the gaps between results that matter to patients and results generally reported in pRCTs.
From this information, the research team developed four recommendations about how to design pRCTs. One focused on reporting per-protocol effects. Per-protocol effects are what would happen in a trial if all patients closely followed the treatment plan. Based on this recommendation, the research team looked closely at per-protocol effects. The team used data from six previous RCTs to show how three statistical methods can estimate per-protocol effects. Then the team applied the new methods to data created using a computer program. The test data mimicked data in pRCTs.
The research team used what they learned to create guidelines for pRCTs. Patients, doctors, and researchers provided feedback about the guidelines.
What were the results?
The research team created 14 guidelines for collecting and analyzing data from pRCTs, including
- How to choose results that matter to patients and researchers
- How to collect data to predict if someone will drop out of a pRCT
- How to predict if a patient will follow the treatment plan
What were the limits of the project?
The study included patients from the Boston area and researchers in the United States. Results may differ for patients and researchers in other places. The research team looked at only certain types of pRCTs, such as those with yes-or-no outcomes like death. The guidelines may not apply to other types of pRCTs.
Future research could test and refine the guidelines for other types of pRCTs.
How can people use the results?
Researchers can use the results when planning pRCTs.
Pragmatic randomized controlled trials (pRCTs) can help answer questions that patients, caregivers, and clinicians have about the effectiveness of treatment options in real-world settings. However, in pRCTs, researchers have less control over whether patients follow their treatment compared with standard RCTs, making it difficult to assess the benefits and harms of treatments.
Currently, no standardized causal inference methodology exists for the design or analysis of pRCTs. Also, most pRCTs exclusively use intention‐to‐treat (ITT) analyses. ITT analyses compare outcomes between treatment and control groups based on initial participant randomization. However, in pRCTs, ITT analyses may lead to biased results because of post-randomization differences in treatment or control groups resulting from study attrition or participant treatment adherence. Other statistical techniques that infer causal relationships may lead to results that better reflect patient and stakeholder preferences.
To develop guidelines for designing pRCTs so that results reflect what matters to patients and investigators
|Design||Qualitative analysis, empirical analysis, simulation analysis|
|Data Sources and Data Sets||
Empirical analysis in 6 case studies using data from previous RCTs:
Adjustment for adherence in calculation of per-protocol effects in simulation and empirical analysis using
Recommendations, guidelines for design and analysis of pRCTs
Methods and Results
To understand what pRCT results matter to patients and researchers, the research team conducted focus groups with patients, semistructured interviews with pRCT statisticians, and a survey of pRCT principal investigators. The team then conducted a systematic literature review to determine how reporting of pRCTs compared with patient and investigator preferences. From this analysis, the research team developed four recommendations for causal inference in pRCTs:
- Focus on superiority in effectiveness or safety, rather than noninferiority.
- Involve patients in specifying subgroups during the study design phase.
- Report absolute measures of risk.
- Complement ITT effect estimates with per-protocol effect estimates. Per-protocol effects are effects that would have been observed if the analysis included only participants who had adhered to the original trial protocol.
Using six example studies, the research team demonstrated the use of three statistical methods to estimate per-protocol effects. The team also conducted simulation studies to further demonstrate the methods’ statistical properties and inform guidelines for the estimation of per-protocol effects.
Finally, the research team developed 14 guidelines describing how to choose a causal effect, clarify the appropriate methods to adjust for loss to follow-up and nonadherence, and define the effect of interest in the presence of competing events.
The research team solicited feedback on the proposed guidelines from stakeholders including patients, clinicians, and researchers.
Qualitative data collection included patients in the Boston area and investigators located primarily in the United States, which may limit the generalizability of these findings. The example studies focused on binary outcomes—with mortality as the primary outcome in five out of six case studies—and included pRCTs with individual randomization. The guidelines may not apply to continuous outcomes such as symptom severity or to crossover or cluster pRCTs.
Conclusions and Relevance
This study developed guidelines for the design and analysis of pRCTs that reflect causal effects of interest to patients and investigators and the use of ITT and per-protocol analyses.
Future Research Needs
Future research could extend the existing methods and guidelines for complex trials such as cluster pRCTs.
Final Research Report
View this project's final research report.
Related Journal Citations
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers said it was difficult to identify which recommendations and guidelines resulted from the study’s case studies and simulations and which were previously known. The researchers said the guidelines document that they were finalizing provides a detailed explanation of the basis for recommendations along with citations to relevant literature. The final report now refers extensively to the guidelines, which were in the finalization process during the completion of peer review.
- The reviewers asked the researchers to discuss the potential for bias in the results their focus group participants produced because those participants are unlikely to represent a random sample of the population. The researchers stated that while it is never possible to guarantee that study participants are a random sample of the population, the researchers could state positively that the patients who participated in the study focus groups came from a wide cross section of an urban population.
- The reviewers recommended considering quality-of-life outcomes, not only safety and efficacy, and different subgroups. The researchers said their guidelines are agnostic about types of outcomes and subgroups, so investigators are free to choose different types as they see fit.
- The reviewers commented that the software developed for this study is coded only in SAS and not available to all researchers. The researchers responded that they are trying to secure funding to release software in SAS, R, and Python.
Conflict of Interest Disclosures
Study Registration Information
^This project was previously titled: Developing Guidelines to Choose Ways to Analyze Results from Real-World Studies
Final Research Report
View this project's final research report.
- Has Results