Results Summary
What was the project about?
Cluster randomized trials, or CRTs, are research studies that compare treatments among different groups of patients, or clusters. An example of a cluster is a group of people who receive care at a single clinic.
One type of CRT is a stepped-wedge CRT. These CRTs compare patients’ health before and after a new treatment. In stepped-wedge CRTs, all groups start with the standard treatment. Then, each group switches to the new treatment at a specific time during the study. By the end of the study, all groups are receiving the new treatment.
In stepped-wedge CRTs, group characteristics, such as how clinics follow up with patients, can affect how well a treatment works. It’s hard to figure out if changes in a patient’s health are due to the treatment or group characteristics.
In this study, the research team wanted to improve how to plan and analyze stepped-wedge CRTs for studying the effect of treatments.
What did the research team do?
The study had two parts. In the first part, the research team looked at ways to measure how well treatments work in stepped-wedge CRTs. The team also looked at how to analyze data from stepped-wedge CRTs in ways that account for group characteristics.
In the second part, the research team looked at which statistical methods got accurate results when using data from stepped-wedge CRTs. The team first used a computer program to create test data that looked like data from a stepped-wedge CRT. The team created the test data using nine scenarios; each scenario had a different set of conditions. For example, the number of patient groups varied across each scenario. Using the test data, the team compared six statistical methods for analyzing data from stepped-wedge CRTs.
The research team also created a statistical program to help plan and analyze stepped-wedge CRTs.
What were the results?
The research team found that all six statistical methods worked for some scenarios; none worked for all nine scenarios.
The new statistical program helps calculate the number of patients that should be in a study.
What were the limits of the project?
The study focused on stepped-wedge CRTs that group patients at a single level, such as by clinic.
Future research could extend the methods to stepped-wedge CRTs that group patients at multiple levels, such as patients receiving care at clinics within a region.
How can people use the results?
Researchers may be able to use these results to improve the design and analysis of stepped-wedge CRTs.
Professional Abstract
Background
In cluster randomized trials (CRTs), researchers, examine the effectiveness of interventions that are offered to groups of patients, such as patients receiving care in a particular clinic.
One type of CRT is a stepped-wedge CRT in which each group starts in the control condition and switches to the treatment at pre-selected time points. This way, all groups receive the treatment, and all groups serve as a control group at some point during the study.
However, these groups often differ by characteristics such as a clinic’s patient volume, standards of care, or patient risk factors. These differences make it hard to determine if changes in a patient’s health are due to the treatment or to group characteristics. Current analysis methods and statistical software for do not adequately address the effect of group characteristics and the longitudinal design of stepped-wedge CRTs in determining patients’ health outcomes.
Objective
To develop and test statistical methods and create a software package for planning and analyzing stepped-wedge CRTs
Study Design
Design Element | Description |
---|---|
Design | Theoretical development, simulation studies |
Data Sources and Data Sets |
Simulated data for 9 scenarios with varying
|
Analytic Approach |
|
Outcomes | Bias, Type I error |
Methods
The researchers first developed a new causal framework for analyzing stepped-wedge CRTs, which included five ways to measure causal effects that account for the role of clinic practices in determining health outcomes. For example, the team defined average causal effect for a provider as the average outcome for treatment and control conditions as if all study participants were treated at a single clinic.
Then, using simulated data, the researchers evaluated two semi-parametric generalized estimating equation (GEE) methods and four linear mixed model (LMM) methods for analyzing stepped-wedge CRTs. The simulated data mimicked stepped-wedge CRTs for nine scenarios, varying by conditions such as the number of groups. The team compared the six analytic methods using two approaches. The model-based approach relied on the assumptions of each method. The design-based approach used the assumptions of the stepped-wedge CRT design.
Finally, the researchers developed a computer program, swCRTdesign, to facilitate the design and analysis of stepped-wedge CRTs.
Results
Each method worked under some scenarios, but none worked for all nine scenarios.
- Model-based approach. For each of the nine scenarios, both LMM and GEE methods gave unbiased estimates of treatment effects. LMM methods produced a low Type I error rate, while the Type I error rate for GEE methods decreased as the number of groups increased from 20 to 40.
- Design-based approach. LMM and GEE methods had varying levels of bias across the nine scenarios. Type I error rates for LMM methods were inflated for some scenarios. GEE methods produced similar Type I errors for all nine scenarios.
Limitations
The study focused on methods for stepped-wedge CRTs with single-level grouping. In the design-based approach, the researchers assumed equal sample size across groups, which may not reflect real-world conditions.
Conclusions and Relevance
Researchers can use the measures of causal effects, analytic methods, and the computer program to improve the design and analysis of stepped-wedge CRTs.
Future Research Needs
Researchers could extend the methods to stepped-wedge CRTs with multiple levels of grouping such as by clinic and geographic region.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers stated that it was unclear how the causal framework, with various causal effect definitions, was connected to the study’s proposed models and simulation results. The researchers added more information on causal methods in the background section of the report, including sections on causal identification along with an example. The researchers noted that they added material on identification of causal estimates for cluster randomized trials, but that the extension of this work to stepped wedge trial designs would occur in future research.
- The reviewers noted a few instances where the models and equations used different notations, with no explanation for these differences. The researchers explained that they combined information from eight different papers in this final research report, creating challenges for continuity. The researchers revised the report by attempting to orient the reader to notational changes that could create confusion.
Conflict of Interest Disclosures
Project Information
Patient / Caregiver Partners
No information provided by awardee
Other Stakeholder Partners
- Jerry Jarvik, MD, MPH
Key Dates
Study Registration Information
This project's final research report is expected to be available by March 2022. |