The PCORI Board of Governors approved this proposed category of Methodology Standards for a public comment period in October 2017. The public comment period took place between October 30 and December 29, 2017, and the final standards were approved by the Board in April 2018.

View the Final Standards

View the Draft Proposed Standards for the Public Comment Period

 

Submitted Public Comments

Submitted Time

11/9/17 11:32

SCI-4: Describe planned data collection and analysis

There is an opportunity here to recommend the use of statistical process control charts for analysis of complex interventions. SCI-4 rightly describes matching the analysis to the questions, but with complex interventions the emerge and change over time, SPC is the best method to monitor the effects on the system. I recognize the likely hesitancy to recommend any specific type of analysis, but perhaps a statement such as "methods used to draw inferences from the data on efficacy and understand the variation of outcomes over time"

Name and Organization

Greg Ogrinc, Geisel School of Medicine at Dartmouth

Community

Stakeholder

Stakeholder Community

Training Institution

---

Submitted Time

11/9/17 22:56

SCI-1: Fully describe the intervention and comparator and define their core functions

This definition is missing direction to describe the complexity itself, which is separate from the causal pathways in SCI-2. Simply stating the mode of delivery, providers, materials, dose, frequency, and target of the intervention not sufficient to help reviewers understand the complexity that should guide data collection and analyses. this description should further indicate the interaction/interdependency of any/all of the mentioned components on outcomes.

SCI-4: Describe planned data collection and analysis

other methods such as Qualitative Comparative Analysis (QCA) and Configurational Comparative Methods (CCM) are valid and more robust than quantitative methods for determining causal pathways in complex interventions and systems. Because they are complex, standard quantitative methods alone are not sufficient for understanding and evaluating complex interventions. The third paragraph above should state instead in the second sentence that plans should include appropriate quantitative, qualitative, and mixed methods analyses. In complex interventions, qualitative and mixed methods analyses are NOT supplemental to quantitative methods - they are critical to understanding and evaluation in these interventions. Quantitative analyses alone are insufficient for evaluating and understanding complex interventions, yet the third paragraph as currently written is heavily weighted that direction.

General feedback on the Standards for Studies of Complex Interventions (SCI)

Thank you for providing updated methodology standards specific to complex interventions and recognizing the important differences.

Name and Organization

Alanna Kulchak Rahm, Geisinger

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/12/17 14:45

General feedback on the Standards for Studies of Complex Interventions (SCI)

These are important improvements in the standards for PCORI methods. They still stop one step short of enabling vigorous learning organizations. PCORI methodology committee should undertake a serious study of Shewhart statistics and its approach to claims of changes worth understanding, which are grounded in quite reputable statistics. In addition, when getting into the effects of context, PCORI methodologists should understand and be willing to use the Context-Mechanism-Outcome structure now widely used in Britain and Europe and initially spelled out by Pawson and Tilley. CMO combinations that are highly context-dependent will challenge our wisdom as to generalizability, but Shewhart statistics can provide solid guidance as to whether the effects upon outcomes are important to understand.

Name and Organization

Joanne Lynn, Altarum Institute

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/17/17 5:40

SCI-1: Fully describe the intervention and comparator and define their core functions

Nice distinction between form and function

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

Notes could suggest that it is not necessary, or even desirable, to base interventions on one theory. Simply, the rational for the presumed causal pathways should be described.

SCI-4: Describe planned data collection and analysis

It should be recognised that formal measurement of mediating, intermediate, outcomes is not always possible. Measures are not always available, and if they are available participant burden can often be far too high. I have experience of working in very deprived areas in the UK and in low literacy settings in Low and Middle Income Countries. In these settings it is imperative to minimise respondent burden. So we need to recognise that it is not always practically possible to gather sufficient data to full test the causal pathways.

General feedback on the Standards for Studies of Complex Interventions (SCI)

Useful, and rigorous.

Name and Organization

Sally Wyke, University of Glasgow

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/17/17 9:26

SCI-1: Fully describe the intervention and comparator and define their core functions

In addition to describing the interventions, it is equally important to justify the choice for the intervention and control. Such choices should be based on several factors including (but not limited to): 1) Acceptability 2) Feasibility 3) Stringency 4) Uniformity 5) Relevance 6) Resemblance

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

Consider specifying the direction of the hypothesized effects and rationale with appropriate support

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

I think another issue here is the intended degree of pragmatism of the trial. Perhaps, using the PRECIS model would be helpful.

SCI-4: Describe planned data collection and analysis

I think gender/sex interactions with interventions should be explored whenever relevant and possible. Also, the guidance can be more specific about how to approach subgroup analyses to minimize false positives and false negatives

Name and Organization

Lehana Thabane, McMaster University

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/19/17 14:19

SCI-1: Fully describe the intervention and comparator and define their core functions

The description of the intervention should address key feasibility issues including likelihood of future implementation by stakeholders and effective use by patients.

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

 

SCI-4: Describe planned data collection and analysis

Process evaluations should be required. Often this evaluation involves use of mixed methods to inform why or why not the complex intervention succeeded (or not) by examining causal pathways often in the form of logic model that addresses key steps in the process. Given that complex interventions often fail, such data are vital to informing next research steps and enhance the value of "negative studies." Process evaluation is analogous to but not synonymous with testing of basic hypothesized mechanisms. In the case of process evaluation of complex interventions, the goal is to inform to determine whether the steps in the process occurred as anticipated in this particular study. The latter (testing of hypothesized basic mechanisms) is designed to generate generalizable knowledge regarding fundamental physiological or behavioral mechanisms.

Name and Organization

Kevin Fiscella, Dept of Family Medicine, University of Rochester Medical Center

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/20/17 9:44

Preamble

Very useful to identify this issue. Multiple active ingredients has been dealt with by behavioral interventionists for decades.

SCI-1: Fully describe the intervention and comparator and define their core functions

Agree and very important to describe all intervention arms in detail. You don't indicate if this could be in supplemental materials or not, but most failures to describe interventions in detail are the result of inadequate space to do so in publications. Also, our terminology for describing intervention components is not standardized. I'd like to see some tip of the hat to efforts to standardize better these intervention components so we describe them similarly in publications.

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

Excellent to require a casual pathway. That said, what often happens is that the investigator cites a theory or model, then fails to describe in detail how the various intervention component target causal mediators of that theory or model. Greater specificity is critical to this standard.

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

This is also an excellent standard that encourages planned adaptations.

---

Submitted Time

 

Preamble

This looks like what should be routine protocol specification. This standard seems broad enough that most investigators will be able to say they already do this. Not sure if there is something specific here that the standard is trying to achieve that is not already commonly achieved. Greater specificity seems needed for this standard.

SCI-1: Fully describe the intervention and comparator and define their core functions

These are excellent standards that are great to see PCORI considering.

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

Bill Riley

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

Stakeholder

General feedback on the Standards for Studies of Complex Interventions (SCI)

Policymaker

Name and Organization

Approved

---

Submitted Time

11/21/17 18:13

General feedback on the Standards for Studies of Complex Interventions (SCI)

Non-adherence to treatment can be a particularly serious concern in complex interventions. How will this be addressed, e.g., ITT analysis or per-protocol analysis, or something else? This needs to be clearly stated and justified. If this is already addressed in a different standard, one could point to that standard. Another issue is that blinding is typically impossible in complex interventions. What would be the impact of lack of blinding (e.g., placebo effect?)

Name and Organization

Ravi Varadhan, Johns Hopkins University

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/26/17 8:54

Preamble

It is also important to point out that in different health care settings, the people actually fulfilling the roles required to implement and intervention may be different from setting to setting. Thus, a medical assistant may be performing actions in one setting where in another setting that same work is being performed by an registered nurse, a physician, or even a community stakeholder. This is why specific training is so important, but training that does not expect that all those involved are necessarily starting from the same place. As well activities performed by one person in one setting may also be done by more than one person in another. It would be good to consider how to describe and engage in complex interventions such that some activities do not need to be performed with fidelity while others do. Thus, have adaptable and non-adaptable components. Investigators could set a priori which interventions or actions need to be carried out with fidelity vs. which components could be more adaptable. For instance in a practice or health system level intervention that is dependent on continually identifying the cohort of subjects to recruit, the practice or health system needs to commit to having dedicated staff members serve roles in pulling patient cohorts in a standard and consistent query throughout the trial. As well to mitigate measurement bias, investigators could describe why fidelity to certain measurements, like accurate assessment of blood pressure of participants in a hypertension trial, is critical to the scientific integrity of the trial. Also there likely needs to be some thought put into the issue of how Vanguard site experiences are used to inform trials. Are there standards for this? In many pragmatic trials, the teams start with Vanguard sites to work out many details, but in some cases the Vanguard site continues to enroll subjects, implement protocols while the intervention sites (non-vanguard) are activated. How are lessons from the Vanguard woven into protocols/decisions made by teams? How is this issue of temporality handled when there is overlap between Vanguard and intervention phase sites? Is there some kind of in analytical approach needed in such cases?

SCI-1: Fully describe the intervention and comparator and define their core functions

NO comment other than what is already listed in the Preamble section: specifically how different people with different roles, lived experiences, and training may be implementing the same actions in a complex intervention study.

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

No comment from JRH, but suspect this is a place for our methodologists.

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

Agree! My only thought here is that efficient and easily understandable data systems must be built that can handle this type of longitudinal information collection strategy. And folks with biostatistical know-how need to be on these teams such that such changes can be handled in the analysis. Research teams have to lead these types of efforts, clinical staff cannot be relied upon to try to think of and document staff changes, role changes and other changes in the context of research. As a general strategy, we need to minimized data collection and data entry by our health system clinical partners. It takes them too much time, will be placed in a low priority level, can add greatly to clinical team stress which can have negative effects on data quality, data completeness, and site retention.

SCI-4: Describe planned data collection and analysis

Agree, but I also think it behooves the funding agencies to share models that they feel are particularly valuable at this point in our evolution of patient centered research. At least some examples of thoughts on constructing conceptual models, but indeed with a keen eye on driving research teams to consider if their choices truly fit within a larger conceptual framework vs. just coming up with things that may be interesting, but ultimately unconnected and over burdensome to all from a volume perspective. Research teams should explain their plans for how they will message about their studies to clinical staff (be sense makers) and how they will continue to be available to clinical staff to keep them informed and on task.

General feedback on the Standards for Studies of Complex Interventions (SCI)

This newly proposed content is logical, but it would be great for PCORI to publish a template for investigators to see that demonstrates how to craft this language. The challenge is often that the funding agency wants more description of actions that research team will take, but then still wants to limit the number of pages for proposals. So please be careful about how much more you are asking for if you are keeping the page limits the same.

Name and Organization

Jacqueline R Halladay

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

11/29/17 11:51

Preamble

 

General feedback on the Standards for Studies of Complex Interventions (SCI)

You still are missing the boat by not including qualitative research standards in your methodology report. Please strongly consider this. I am an expert in qualitative research, and would be happy to work with you as a consultant to help develop rigorous, patient-centered standards.

Name and Organization

Heather Stuckey, Penn State Hershey College of Medicine

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

12/5/17 6:28

Preamble

not just healthcare staff

SCI-1: Fully describe the intervention and comparator and define their core functions

Should the popn be under a different heading, and what about the setting?

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

no comment

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

no comment

SCI-4: Describe planned data collection and analysis

the process evaluation will not always be able (nor will it be appropriate to measure all of these things - this should be made claer)

General feedback on the Standards for Studies of Complex Interventions (SCI)

The standards are comprehensive - but the data collection and analysis standards might prove un-achievable as written

Name and Organization

CIndy Gray University of Glasgow

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

12/8/17 13:07

SCI-1: Fully describe the intervention and comparator and define their core functions

It is also important to describe how participants receive the components of the intervention. Are participants treated individually? In groups? By a common therapist or other change agent? Do they interact in groups or online? To the extent that participants interact with one another post randomization, whether face-to-face or electronically, we can expect some correlation in their data, and that would need to be addressed in the analysis. But if we don't track which participants are seen by which therapists, or receive components of the intervention in which groups, we cannot address it in the analysis.

SCI-4: Describe planned data collection and analysis

Complex interventions are often multi-level, and variables are often measured at multiple levels. It is common to evaluate multi-level interventions with group- or cluster-randomized designs, stepped wedge designs, or with individually randomized group-treatment or partially clustered designs. Such designs pose special sample size and analytic issues, and these would need to be addressed in the application. The important point is to anticipate whether observations will be correlated, and to address that correlation when the study is being planned and analyzed. A useful resource is available from NIH at https://researchmethodsresources.nih.gov.

Name and Organization

David M. Murray, Associate Director for Prevention, NIH

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

12/13/17 8:47

SCI-1: Fully describe the intervention and comparator and define their core functions

Perhaps add whether it is standardized or tailored, and how (although I do see that this is one of the characteristics above & may be what is referred to below as adaptations)

SCI-4: Describe planned data collection and analysis

The phrase "nature of the functions defined by the causal pathways" was not clear to me.

Name and Organization

Nancy Feeley, McGill University

Community

Stakeholder

Stakeholder Community

Health Researcher

---

Submitted Time

12/29/17 15:39

Preamble

AcademyHealth recommends that PCORI be more specific about what “levels” are being referred to in “multiple entities or levels targeted by the intervention”. Is this referring to the six levels of community, patient, provider, microsystem, mesosystem, and macrosystem. A clear definition should be provided.

SCI-1: Fully describe the intervention and comparator and define their core functions

AcademyHealth recommends that just as researchers should fully describe the intervention(s) and comparator(s) being studied, they should also describe the intervention implementation strategy, or at least the planned implementation approach. When the comparator is the standard of care, this standard should be fully specified. We also recommend that this standard include guidance for researchers to explain key contextual factors that may affect implementation outcomes, effectiveness, fidelity and variation across sites as well as a definition of the meaning of “levels”. Further we recommend that researchers describe the extent of freedom that exists for implementers to vary core functions and forms of the intervention, that is, how much variation in functions and forms is allowed within the study context (see SC-3 for more detail). In some instances (e.g. adaptive designs), researchers may be further refining the intervention mode of delivery through iterative phases and if so, this should be specified. Minor edit on the last sentence in the standard: the examples in parentheses are just that – examples and this should be edited to be an e.g. not an i.e.

SCI-2: Specify the hypothesized causal pathways and their theoretical basis

AcademyHealth agrees that defining, in advance, the causal pathway and theoretical basis for change is necessary. To that end, the logic model should be supported by an established conceptual framework and appropriate citations provided. Logic models are linear and may fail to display the hypothesized interactions that are driven by context and complexity. Lack of a supporting conceptual model makes it difficult to see where the planned intervention fits in the larger context, as well as to visualize interactions. Accounting and planning for context is critical as context inevitably has an impact on the dose of an intervention that is actually received, which could be seen as a mediator of outcomes. Thus, the context influences the actual intervention (forced modification of the intervention due to the context), the dose of the planned intervention that actually is received, the success of the planned activities in achieving planned outputs, change in process measures/behaviors, and outcomes. These dimensions should be mapped to the components of the logic model for clarity and not be limited to “prespecified patient outcomes(s)”. In addition, to recognize the potential to ensure maximum learning from the implementation of the intervention, we recommend requiring both the documentation of the causal pathway in advance (prior to seeing the data) as well as any modifications made to the model after data analysis and the rationale for such changes. Retrospective analysis, and publication, of such discovered pathways and the rationale for the changes in the logic model will further contribute to the body of knowledge. We further note a theory explaining how and why the proposed interventions will affect the outcomes is insufficient. Researchers should provide an explicit quantitative prediction of the attributable effect, along with the expected precision of this estimate (in the form of degree of belief, prior probability, or confidence intervals). Too often failure to specify a target outcome is due to lack of good evidence regarding the likely attributable effect, failure to consider the low reliability of health systems in implementing core changes, and/or adequate consideration of bias and confounding. The hypothetical causal pathway and the implied attributable effect should be weighted in the light of the Bradford Hill or other epidemiological criteria/standards. Minor edit to the third sentence: it would be unrealistic to ask for “any” contextual factors. This should refer to “key” contextual factors.

SCI-3: Specify how adaptations to the form of the intervention and comparator will be allowed and recorded

Recognizing that an intervention will not be delivered 100 percent of the time to 100 percent of the patients or community members—and often for good reasons— AcademyHealth recommends that this standard not only detail specification of adaptations, but also the documentation of unplanned, observed adaptations that were not pre-specified, and the rationale, setting, and frequency of those adaptations. The need to describe clearly the rationale for any adaptation goes beyond the desirability of comprehensiveness or completeness. In cases where an adaptation to an intervention may be chosen because the chooser knows or suspects that this particular adaptation will work better than any other in the specific setting, outcomes of the particular adaptation may be uniquely good in the setting in question, but not generalizable to other settings. Allowing for endogenous adaptations may itself be a characteristic of an intervention, but this needs to be appreciated and documented. Qualitative data is especially helpful in understanding whether an adaptation was chosen because it was known or suspected to be especially effective in a specific setting.

SCI-4: Describe planned data collection and analysis

The wording of this standard raises a number of concerns. First, effectiveness can be measured by both process and outcomes. The term “process outcome” is confusing, however we recognize that in some instances, especially when true outcomes are hard to obtain, intermediate outcomes are used, and these can in fact be processes. We suggest editing the first sentence to: “draw inferences about the impact of the intervention on processes of care and patient outcomes”. Second, we strongly support the use of valid and reliable patient outcome measures but only when they are appropriate to the patient, population, intervention and context. The standard should not encourage use of measures for a different patient population/setting that the researchers think are inappropriate for their study. This statement also should discuss the balance between process and outcome measures. When outcomes are rare, hard to capture, or in the distant future, an explicit case must be made for why process measures are reasonable proxies. Third, the statement also appears to favor quantitative methods over qualitative ones. AcademyHealth suggests that for complex interventions qualitative and mixed methods should be more strongly suggested as quantitative methods alone are likely to be insufficient. In fact, some questions may only be analyzed with rigorous qualitative methods. Fourth, specifying contextual factors at all levels of the targets of the intervention is certainly the ideal; however, depending upon the scope and budget of the project, it may be reasonable to target measures to selected levels or selected aspects of the process. Fifth, Researchers should describe in detail how the subject/settings for study were identified and how intervention status was assigned. In many health services research studies of complex interventions the subjects may be a convenience sample and intervention assignment may be based on voluntary participation. In others, the data will be observational with or without a true natural experiment. In either instance, researchers should identify and describe potential sources of bias and, if possible, determine the likely direction of the bias. Researchers should also describe the methods used to minimize bias and to quantify its likely magnitude and direction. Additional points include: • It would be helpful to define the expected duration of the intervention as well as the expected timeline for effects to appear for various outcomes and processes. This may well differ for different settings, patients, and populations and researchers should describe how this will be determined. Effects may appear at different times for different outcomes and processes. In complex interventions, it may take a considerable amount of time for the program to become fully effective—researchers should specify that for each outcome or group of outcomes and the basis for their assumptions. • The definition of data collection tools and sources should be documented and additional implementation outcomes (acceptability, reach) and strategy outcomes (speed, quality, reach) should be considered. An assessment of the strength of evidence for the anticipated impact on outcomes is also necessary. • It would be wise to consider collecting data on the potential costs and budget impact of the intervention and its implementation, including ongoing maintenance and opportunity costs of the intervention. Expending significant effort to develop and test a complex intervention that has little to no likelihood of adoption because of its cost is not ideal.

General feedback on the Standards for Studies of Complex Interventions (SCI)

AcademyHealth believes the pre-definition and documentation of planned analyses, data sources, and data collection tools as outlined in these standards is highly desirable, and will support the overall integrity of the research. We also acknowledge that doing so requires a significant investment of time and budget. Successful adherence to these standards may be difficult or unrealistic for a study with a limited budget, and considerations for this work should be made in the grant process. To the degree possible, we also recommend simplification of the language and word choice in each standard to ensure both specificity and understanding across settings and disciplines. Finally, it is not clear whether this guidance has been cross-walked with the published guidance in SQUIRE and STaRI and a crosswalk would be a helpful table, and if there are gaps in these guidelines or in SQUIRE and STaRI, they should be addressed.

Name and Organization

Lisa Simpson, MB, BCh, MPH, President & CEO, AcademyHealth

Community

Stakeholder

Stakeholder Community

Health Researcher

---