HT-1: State the goals of HTE analyses, including hypotheses and the supporting evidence base

State the inferential goal of each HTE analysis, and explain how it is related to the topic of the research. Specify whether the HTE analysis is hypothesis-driven (sometimes denoted as confirmatory), or hypothesis-generating (sometimes denoted as exploratory). Hypothesis-driven HTE analyses should be pre-specified, based on prior evidence (described clearly in the study protocol and published paper), and supported by a clear statement of the hypotheses the study will evaluate, including how subgroups will be defined (e.g., by multivariate score or stratification), outcome measures, and the direction of the expected treatment effects.

Public comments

No comments.

HT-2: For all HTE analyses, provide an analysis plan, including the use of appropriate statistical methods

The study protocol should unambiguously pre-specify planned HTE analyses. Appropriate methods include, but are not limited to, interaction tests, differences in treatment effect estimates with standard errors, or a variety of approaches to adjusting the estimated subgroup effect, such as Bayesian shrinkage estimates. Appropriate methods should be used to account for the consequences of multiple comparisons; these methods include, but are not limited to, p-value adjustment, false discovery rates, Bayesian shrinkage estimates, adjusted confidence intervals, or validation methods (internal or external). A common error in HTE analyses is to claim differences in treatment effect when one subgroup shows a statistically significant treatment effect and another does not.

Public comments

AcademyHealth first recommends addressing the topic of type II errors, which is absent from the Methodology Standards altogether. To this point, researchers who focus on or care about type II errors, might not consider p-value adjustment a credible approach.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:37pm

HT-3: Report all pre-specified HTE analyses and, at minimum, the number of post-hoc HTE analyses, including all subgroups and outcomes analyzed

Protocols and study reports must report the exact procedures used to explore HTE, including data mining or any automatic regression approaches. HTE analyses should clearly report the procedures by which subgroups were defined, and the effective number of subgroups and outcomes examined. Within each subgroup level, studies should present the treatment effect estimates and measures of variability. Pre-specified HTE analyses (hypothesis-driven) should be clearly distinguished from post-hoc HTE analyses (hypothesis-generating). Statistical power for all analyses should be reported.

Public comments

The last sentence reads that "statistical power for all analyses should be reported." This is true if it is a test of hypothesis, but it does not apply if the purpose of the exercise is estimation of effects. {EBort}

{EBort}Merck & Co Inc, Industry, 02/23/2016 - 2:37pm

I think it is an "overkill" to require reporting of statistical power for all analyses. This should only be required for hypothesis-driven analyses.

Ravi Varadhan, Johns Hopkins University, Health Researcher, 01/26/2016 - 2:45pm

General feedback on the Standards for Heterogeneity of Treatment Effects

Public comments

In general, the standards for HTEs would benefit from listing a few examples of variables that could be considered, such as gender, age, co-morbidity, lifestyle attributes, race, ethnicity, and/or regional factors. In addition, the standard should state that it is important to include disease state-specific variables.

Eli Lilly and Company, Industry, 03/30/2016 - 2:18pm

These standards do not cover "predictive" HTE or individualized treatment-effect estimation using complex machine learning models. I think this is an emerging area of great interest and standards might be required to ensure that the predictive learning is done according to rigorous principles.

Ravi Varadhan, Johns Hopkins University, Health Researcher, 01/26/2016 - 2:45pm