CI-1: CI-Model: Specify the causal model underlying the research question

Researchers should describe the causal model relevant to the research question, which should be informed by the PICOTS framework: populations, interventions, comparators, outcomes, timing, and settings. The causal model represents the key variables; the known or hypothesized relationships among them, including the potential mechanisms of effect; and the conditions under which the hypotheses are to be tested. Researchers should use the causal model to determine whether and how the study can handle bias and confounding and the extent to which valid estimates of the effects of an intervention can be generated based on the particular data source, hypothesis, and study design.

Public comments

Agreed, important step

Robert Lubwama; Merck & Co Inc., Industry, 02/23/2016 - 2:41pm

This is a nice addition. A simple model should help investigators think clearly about causation and potential confounding, and then select an appropriate causal inference strategy.

Steven D. Pizer, Assoc. Prof. of Health Economics, Northeastern University, Health Researcher, 01/27/2016 - 12:36pm

How about asking the researchers to `define the causal estimand?' This is the most fundamental step in causal inference.

Ravi Varadhan, Johns Hopkins University , Health Researcher, 01/26/2016 - 2:52pm

---

CI-2: Define and appropriately characterize analysis population used to generate effect estimates

Decisions about whether patients are included in an analysis should be based on information available at each patient’s time of study entry in prospective studies or on information from a defined time period prior to the exposure in retrospective studies. For time-varying treatment or exposure regimes, specific time points should be clearly specified and relevant variables measured at baseline and up to, but not beyond, those time points should be used as population descriptors. When conducting analyses that in some way exclude patients from the original study population, researchers should describe the final analysis population that gave rise to the effect estimate(s).

Public comments

Since the Methodology Standards revolve around patient-centered studies, the first sentence of CI-2 reads as peculiar. AcademyHealth assumes PCORI is referring to decisions about including ‘specific patients,’ concerning their inclusion or exclusion in studies, but PCORI may wish to stipulate its meaning more plainly.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:45pm

Add to the end something like "Researchers should determine whether and how the study can handle selection bias and the extent to which valid estimates of the effects of an intervention can be generated based on the final analysis population."

Robert Lubwama; Merck & Co Inc., Industry, 02/23/2016 - 2:41pm

---

CI-3: Define with the appropriate precision the timing of the outcome assessment relative to the initiation and duration of exposure

To ensure that an estimate of an exposure or intervention effect corresponds to the question that researchers seek to answer, the researchers must precisely define, to the extent possible, the timing of the outcome assessment relative to the initiation and duration of the exposure.

Public comments

No comments.

---

CI-4: Measure potential confounders before start of exposure and report data on potential confounders with study results

In general, variables for use in confounding adjustment (either in the design or analysis) should be ascertained and measured prior to the first exposure to the interventions (or intervention) under study. If confounders are time varying, specific time points for the analysis of the exposure effect should be clearly specified and the confounder history up to, and not beyond, those time points should be used in that analysis.

Public comments

No comments.

---

CI-5: Report the assumptions underlying the construction of propensity scores and the comparability of the resulting groups in terms of the balance of covariates and overlap

When conducting analyses that use propensity scores to adjust for measured confounding, researchers should assess the overlap and balance achieved across compared groups with respect to potential confounding variables.

Public comments

Also researchers need to consider how the PS will be implemented (e.g. matching, weighting), what is the desired approach and its implications on inference. Further, after PS analysis, what is the potential for unmeasured/residual confounding?

Robert Lubwama; Merck & Co Inc., Industry, 02/23/2016 - 2:41pm

---

CI-6: Assess the validity of the instrumental variable (i.e., how the assumptions are met) and report the balance of covariates in the groups created by the instrumental variable

When an instrumental variable (IV) approach is used to address potential unmeasured confounding, empirical evidence should be presented describing how the variable chosen as an IV satisfies the three key properties of a valid instrument:

1. the IV influences choice of the intervention or is associated with a particular intervention because both have a common cause;
2. the IV is unrelated to patient characteristics that are associated with the outcome; and
3. the IV is not otherwise related to the outcome under study (i.e., it does not have a direct effect on the outcome apart from its effect through exposure).

Public comments

The revised version adds the phrase “unmeasured confounding” when referring to the instrumental variable analysis. This is potentially confusing for several reasons. First, the three assumptions that follow apply regardless of whether the IV approach is used for “unmeasured confounding” or used simply as an analysis tool without specifically targeting unmeasured confounding. Second, as this is the only mention of a method for unmeasured confounding within the standards, it can appear as an endorsement of this approach. The need for sensitivity analysis for unmeasured confounding is clear and should be emphasized more, though the specific method that is best will depend on study-specific factors (Schneeweiss S. Pharmacoepidemiology and drug safety 2006). In addition, several new advances have appeared in the recent literature allowing quantitative assessments of the potential impact of unmeasured confounding (Faries D et al. Value in health 2013; Ryan P et al. Statistics in medicine 2012; Yu et al. Pharmacoepidemiology and drug safety 2012). These references should be considered for inclusion within this standard.

Eli Lilly and Company , Industry, 03/30/2016 - 2:24pm

This has improved as well. Reporting the balance of covariates between groups created by the IV is a nice way to test assumption 2. You may want to consider suggesting falsification testing as a means to evaluate assumption 3.

Steven D. Pizer, Assoc. Prof. of Health Economics, Northeastern University, Health Researcher, 01/27/2016 - 12:36pm

---

General feedback on the Standards for Causal Inference Methods

Public comments

AcademyHealth applauds PCORI’s efforts to cover a wide range of methodologies. However, after reviewing the Standards—and Standard 8 in particular—one is left with the false impression that methods employing instrumental variables and propensity scores are the primary observational data methods. AcademyHealth recommends that PCORI describe other methods such as difference-in-differences, regression discontinuity, factorial experiments and partial factorial experiments, interrupted time series, and sample selection models to give the reader a flavor for the variety of methods that are now available and are likely to be expanded in the future.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:45pm

Good to address these issues. I'm puzzled by Cl-6: "(i.e., how the assumptions are met)". We can never know. So I suggest: "(i.e., support for assumptions)" Similarly: "describing how the variable chosen as an IV satisfies the three key properties" --> "describing to what extent the variable chosen as an IV satisfies the three key properties"

E.W. Steyerberg, Erasmus MC, Rotterdam, Netherlands, Health Researcher, 03/25/2016 - 4:43am

General feedback: time-varying exposures and confounders are mentioned but there's no discussion on analytic techniques to study/account for these.

Robert Lubwama; Merck & Co Inc., Industry, 02/23/2016 - 2:41pm

The standards do not address generalizability or "transportability" of findings. A separate standard may be needed.

Ravi Varadhan, Johns Hopkins University , Health Researcher, 01/26/2016 - 2:52pm