• Download Summary

What was the project about?

In randomized controlled trials, or RCTs, researchers assign patients by chance to different treatments to compare the benefits and harms. In RCTs, researchers have a high level of control over how patients receive treatment. RCTs often take place in research clinics with staff who monitor how patients follow treatment plans.

Pragmatic RCTs, or pRCTs, take place where patients typically receive treatment, such as a regular clinic. pRCTs can help capture the real-world effects of treatment but determining whether a treatment works can be hard in pRCTs. Also, no clear guidelines exist about how to collect and analyze data from pRCTs. Some kinds of analysis are better for helping researchers focus on what’s important to patients.

In this study, the research team created guidelines for collecting and analyzing data in pRCTs so that results reflect what matters to patients and researchers.

What did the research team do?

First, the research team asked patients and researchers what results from pRCTs matter to them. The team had three group discussions with adult patients. The team also did five interviews and an online survey with pRCT researchers. Then the team reviewed results from published pRCTs. They looked at the gaps between results that matter to patients and results generally reported in pRCTs.

From this information, the research team developed four recommendations about how to design pRCTs. One focused on reporting per-protocol effects. Per-protocol effects are what would happen in a trial if all patients closely followed the treatment plan. Based on this recommendation, the research team looked closely at per-protocol effects. The team used data from six previous RCTs to show how three statistical methods can estimate per-protocol effects. Then the team applied the new methods to data created using a computer program. The test data mimicked data in pRCTs.

The research team used what they learned to create guidelines for pRCTs. Patients, doctors, and researchers provided feedback about the guidelines.

What were the results?

The research team created 14 guidelines for collecting and analyzing data from pRCTs, including

• How to choose results that matter to patients and researchers
• How to collect data to predict if someone will drop out of a pRCT
• How to predict if a patient will follow the treatment plan

What were the limits of the project?

The study included patients from the Boston area and researchers in the United States. Results may differ for patients and researchers in other places. The research team looked at only certain types of pRCTs, such as those with yes-or-no outcomes like death. The guidelines may not apply to other types of pRCTs.

Future research could test and refine the guidelines for other types of pRCTs.

How can people use the results?

Researchers can use the results when planning pRCTs.

Background

Pragmatic randomized controlled trials (pRCTs) can help answer questions that patients, caregivers, and clinicians have about the effectiveness of treatment options in real-world settings. However, in pRCTs, researchers have less control over whether patients follow their treatment compared with standard RCTs, making it difficult to assess the benefits and harms of treatments.

Currently, no standardized causal inference methodology exists for the design or analysis of pRCTs. Also, most pRCTs exclusively use intention‐to‐treat (ITT) analyses. ITT analyses compare outcomes between treatment and control groups based on initial participant randomization. However, in pRCTs, ITT analyses may lead to biased results because of post-randomization differences in treatment or control groups resulting from study attrition or participant treatment adherence. Other statistical techniques that infer causal relationships may lead to results that better reflect patient and stakeholder preferences.

Objective

To develop guidelines for designing pRCTs so that results reflect what matters to patients and investigators

Study Design

Methods and Results

To understand what pRCT results matter to patients and researchers, the research team conducted focus groups with patients, semistructured interviews with pRCT statisticians, and a survey of pRCT principal investigators. The team then conducted a systematic literature review to determine how reporting of pRCTs compared with patient and investigator preferences. From this analysis, the research team developed four recommendations for causal inference in pRCTs:

• Focus on superiority in effectiveness or safety, rather than noninferiority.
• Involve patients in specifying subgroups during the study design phase.
• Report absolute measures of risk.
• Complement ITT effect estimates with per-protocol effect estimates. Per-protocol effects are effects that would have been observed if the analysis included only participants who had adhered to the original trial protocol.

Using six example studies, the research team demonstrated the use of three statistical methods to estimate per-protocol effects. The team also conducted simulation studies to further demonstrate the methods’ statistical properties and inform guidelines for the estimation of per-protocol effects.

Finally, the research team developed 14 guidelines describing how to choose a causal effect, clarify the appropriate methods to adjust for loss to follow-up and nonadherence, and define the effect of interest in the presence of competing events.

The research team solicited feedback on the proposed guidelines from stakeholders including patients, clinicians, and researchers.

Limitations

Qualitative data collection included patients in the Boston area and investigators located primarily in the United States, which may limit the generalizability of these findings. The example studies focused on binary outcomes—with mortality as the primary outcome in five out of six case studies—and included pRCTs with individual randomization. The guidelines may not apply to continuous outcomes such as symptom severity or to crossover or cluster pRCTs.

Conclusions and Relevance

This study developed guidelines for the design and analysis of pRCTs that reflect causal effects of interest to patients and investigators and the use of ITT and per-protocol analyses.

Future Research Needs

Future research could extend the existing methods and guidelines for complex trials such as cluster pRCTs.