Results Summary

What was the project about?

In randomized controlled trials, or RCTs, researchers assign patients by chance to different treatments to compare the benefits and harms. In RCTs, researchers have a high level of control over how patients receive treatment. RCTs often take place in research clinics with staff who monitor how patients follow treatment plans.

Pragmatic RCTs, or pRCTs, take place where patients typically receive treatment, such as a regular clinic. pRCTs can help capture the real-world effects of treatment but determining whether a treatment works can be hard in pRCTs. Also, no clear guidelines exist about how to collect and analyze data from pRCTs. Some kinds of analysis are better for helping researchers focus on what’s important to patients.

In this study, the research team created guidelines for collecting and analyzing data in pRCTs so that results reflect what matters to patients and researchers.

What did the research team do?

First, the research team asked patients and researchers what results from pRCTs matter to them. The team had three group discussions with adult patients. The team also did five interviews and an online survey with pRCT researchers. Then the team reviewed results from published pRCTs. They looked at the gaps between results that matter to patients and results generally reported in pRCTs.

From this information, the research team developed four recommendations about how to design pRCTs. One focused on reporting per-protocol effects. Per-protocol effects are what would happen in a trial if all patients closely followed the treatment plan. Based on this recommendation, the research team looked closely at per-protocol effects. The team used data from six previous RCTs to show how three statistical methods can estimate per-protocol effects. Then the team applied the new methods to data created using a computer program. The test data mimicked data in pRCTs.

The research team used what they learned to create guidelines for pRCTs. Patients, doctors, and researchers provided feedback about the guidelines.

What were the results?

The research team created 14 guidelines for collecting and analyzing data from pRCTs, including

  • How to choose results that matter to patients and researchers
  • How to collect data to predict if someone will drop out of a pRCT
  • How to predict if a patient will follow the treatment plan

What were the limits of the project?

The study included patients from the Boston area and researchers in the United States. Results may differ for patients and researchers in other places. The research team looked at only certain types of pRCTs, such as those with yes-or-no outcomes like death. The guidelines may not apply to other types of pRCTs.

Future research could test and refine the guidelines for other types of pRCTs.

How can people use the results?

Researchers can use the results when planning pRCTs.

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers said it was difficult to identify which recommendations and guidelines resulted from the study’s case studies and simulations and which were previously known. The researchers said the guidelines document that they were finalizing provides a detailed explanation of the basis for recommendations along with citations to relevant literature. The final report now refers extensively to the guidelines, which were in the finalization process during the completion of peer review.
  • The reviewers asked the researchers to discuss the potential for bias in the results their focus group participants produced because those participants are unlikely to represent a random sample of the population. The researchers stated that while it is never possible to guarantee that study participants are a random sample of the population, the researchers could state positively that the patients who participated in the study focus groups came from a wide cross section of an urban population.
  • The reviewers recommended considering quality-of-life outcomes, not only safety and efficacy, and different subgroups. The researchers said their guidelines are agnostic about types of outcomes and subgroups, so investigators are free to choose different types as they see fit.
  • The reviewers commented that the software developed for this study is coded only in SAS and not available to all researchers. The researchers responded that they are trying to secure funding to release software in SAS, R, and Python.

Conflict of Interest Disclosures

Project Information

Miguel Hernán, MD, DrPH
Harvard University School of Public Health
$1,014,573 *
Causal Inference Guidelines for Pragmatic Clinical Trials^

Key Dates

September 2015
September 2020

Study Registration Information

^This project was previously titled: Developing Guidelines to Choose Ways to Analyze Results from Real-World Studies

Final Research Report

View this project's final research report.

Journal Articles


Has Results
Award Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 20, 2023