12. Standards on Research Designs Using Clusters

RC-1: Specify whether the study objectives, the interventions, and the primary outcomes pertain to the cluster level or individual level

1. Describe the target population of clusters and individuals to which the study findings will be generalizable.
2. Describe the clusters to be randomized and the subjects to be enrolled in the trial.

Public comments

As a whole Standard 12, and in particular RC-1 and RC-2, is restrictive in its binary classification of the cluster and individual level of study objectives, interventions, and primary outcomes. This hierarchical manner of thinking is restrictive for research designs.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:50pm

RC-2: Justify the choice of cluster randomization

Describe the benefits and disadvantages of cluster randomization versus individual-level randomization for the proposed research. Cluster randomization should be substantiated by a sound theoretical and conceptual framework that describes the hypothesized causal pathway. Cluster randomization generally is applicable when*:

1. An intervention is delivered at the cluster level
2. An intervention changes the physical or social environment
3. An intervention involves group processes, or
4. An intervention cannot be delivered without a serious risk of contamination

*Logistical considerations can also justify cluster randomization, for example, to reduce costs or to improve participation, adherence, or administrative feasibility.

Public comments

No comments.

RC-3: Power and sample size estimates must use appropriate methods to account for the dependence of observations within clusters, and the degrees of freedom available at the cluster level

The methods used to reflect dependence should be clearly described. Sources should be provided for the methods and for the data used to estimate the degree of dependence. Sensitivity analyses incorporating different degrees of dependence must be reported.

1. For simpler designs, the dependence in the data can be reflected in the intraclass correlation.
2. Dependence can also be reflected in variance components
3. Other factors that affect the power calculation include: the design of the study, the magnitude of the hypothesized intervention effect, the pre-specified primary analysis, and the desired Type I error rate.

Public comments

We applaud the addition of standards for research designs using clusters, but do suggest some supplementary language to the proposed standard. Suggested language: Consider simulation methodology to examine power under scenarios that reflect conditions that are not pre-specified, such as varying sample sizes within clusters if such sample sizes are not pre-specified

Eli Lilly and Company, Industry, 03/30/2016 - 2:29pm

RC-4: Data analyses must account for the dependence of observations within clusters regardless of its magnitude

Data analyses must also reflect the degrees of freedom available at the cluster level. Investigators must propose appropriate methods for data analyses with citations and sufficient detail to reproduce the analyses.

Public comments

No comments.

RC-5: Because cluster randomization trials often involve a limited number of groups or clusters, stratified randomization is recommended

Non-randomized intervention trials often involve a limited number of groups or clusters, and efforts should be made to balance treatment or study conditions on potential confounders.

1. The recommended stratification factors are those that are expected to be strongly correlated with the outcome or with the implementation of the intervention, such as:
   1. Baseline value of the outcome variable
   2. Cluster size
   3. Geographic area

Public comments

The PCORI Methodology Standards overall and Standard 12 (in particular Standard RC-5), would be strengthened by mentioning the importance of assessing and documenting context (which may change over time) in evaluating and comparing interventions, including the internal and external contexts. Research may be improved upon by documenting and learning from heterogeneity of results rather than simply seeking to adjust away such variation. Furthermore, measurement of implementation factors, such as fidelity, adaptation, implementation procedures, and deviations from the planned approach, is critical in order to learn what works best for whom and in what context. Attention should be paid to how the investigator will explore the potential reasons surrounding why a seemingly good intervention fails (should that be the finding) or why some programs sites are more successful than others. Researchers should describe their approach to gathering this information—both quantitative and qualitative—on implementation and how they will integrate it with their analysis of program effects. This is also an area where qualitative and mixed methods approaches are critical to understanding the implications and sustainability of program effects.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:50pm

I agree with the principle of achieving balance when CRT's are "small" (number of clusters). However, I disagree with the focus on stratification as the only solution. It may work for moderately-sized trials, but in very small CRT's, stratification may not be feasible. Alternatives such as restricted randomization may be highlighted as an alternative approach (Hayes and Moulton: Cluster Randomized Trials, 2009, p. 86-103).

Matthew Gurka, University of Florida, Health Researcher, 01/28/2016 - 3:07pm

General feedback on the Standards on Research Designs Using Clusters

Public comments

AcademyHealth appreciates the addition of this new Standard, which will be increasingly important as the use of cluster design increases. This Standard is unique in that it’s limited to a design-specific set of standards, while the others are somewhat design agnostic. Nevertheless, while we agree these Standards are important to include, Standard 12 includes information at a comparatively granular level. Furthermore, and notably, AcademyHealth implores PCORI to remember that cluster design is just one approach being used in the growing number of comparative studies of complex interventions. We urge PCORI to include other designs for evaluating complex interventions—including designs from implementation science—in a future iteration of the Methodology Standards.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:50pm