RQ-1: Identify gaps in evidence

Analysis of gaps in the evidence based on systematic reviews should be used to support the need for a proposed study. If a systematic review is not available, a systematic review should be performed using accepted standards in the field (see standard SR-1), or a strong rationale should be presented for proceeding without a systematic review. In the case where a systematic review is not possible, the methods used to review the literature should be explained and justified.

Public comments

As proposed, the call to use ‘analysis of gaps in the evidence based on systematic reviews’ is much more complex than the requirement in the prior version of the Standard. Given the increased emphasis on using systematic reviews to identify evidence gaps as well as to ensure that PCORI’s documents are as useful and accessible as possible, PCORI should consider making available on its website the best sources of systematic reviews, such as Cochrane, formally known as the Cochrane Collaboration (http://www.cochrane.org/) and Canada’s Health Systems Evidence (https://www.healthsystemsevidence.org/), so researchers have a more solid foundation from which to begin their study. Furthermore, when conducting systematic reviews, the central issue is not necessarily about what is possible, but what is useful and worthwhile. Therefore, we propose that the last sentence be: “In cases where incremental systematic reviews may not be useful over existing literature or in which the effort needed is not worthwhile, the methods used to review the literature should be explained and justified.”

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:23pm

For rare diseases a systematic review is seldom available. This puts undo demand on investigators in a under investigated area.

Kate Lorig, Patient, 03/05/2016 - 10:58am

The last sentence now reads: "In the case where a systematic review is not possible.......". Is a systematic review ever not possible? Even is the evidence for something is sparse, it can still be systematically reviewed. Consider replacing "possible" with "done". {EBort}

{EBort} Merck & Co Inc, Industry, 02/23/2016 - 2:31pm

Identification of gaps in the evidence is imperative, specially those who have excluded women, children and the elderly. However, ideally evidence-base also should include socio-economic and psychological aspects of the population.

anonymous, Caregiver/Family member of patient, 01/31/2016 - 10:17am

Specifically regarding the statement "in the case where a systematic review is not possible, the methods used to review the literature should be explained and justified": it would be interesting to suggest that these methods should follow as many as possible among the components specified under the Methods section of the PRISMA statement http://goo.gl/BijVhu

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

RQ-2: Develop a formal study protocol

Studies should include a formal protocol specifying at least one purpose for which the data were collected (e.g., effectiveness, safety, natural history of disease, quality improvement); data sources and linkage plans, if any; data feasibility and quality, measure(s) of effect; and use of any standardized data dictionaries (nationally or internationally accepted).

Public comments

Given that the EQUATOR NETWORK http://www.equator-network.org/ currently has a number of design-specific standards, it might be interesting to recommend a few of those when study protocols are formulated. Examples could include CONSORT, STROBE, PRISMA, COREQ, STARD, SQUIRE AND CHEERS

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

RQ-3: Identify specific populations and health decision(s) affected by the research

To produce information that is meaningful and useful to people when making specific health decisions, research proposals and protocols should describe:

1. the specific health decision the research is intended to inform;
2. the specific population for whom the health decision is pertinent;
3. and
4. how study results will inform the health decision.

Public comments

The PICOTS typology to which PCORI refers later in its Methodology Standards is used in systematic reviews for precisely this purpose of identifying and characterizing evidence gaps from systematic reviews. Therefore, PCORI should cite the framework in this Standard and advise researchers to cite it when producing the requested information. Additionally, AcademyHealth recommends changing the word ‘population’ in the second bullet to ‘populations’ (plural) both to maintain consistency with the Standard’s title and to encompass as wide a group as necessary for a given study.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:23pm

Sometimes in behavioral studies the end point is not a decision but a behavior. I suppose you could say the decision at the end is which behavioral intervention if effective but sometimes you are only looking at effectiveness as opposed to standard care which is the norm

Kate Lorig, Patient, 03/05/2016 - 10:58am

Given the abundance of epidemiological data on a variety of conditions, it might be interesting to suggest that citations be based on primary rather than secondary sources. Examples of primary sources might include information extracted from raw data, Web sites where datasets can be directly queried (see http://cancerstatisticscenter.cancer.org/#/ for an example) or publications where epidemiological information was part of the study Results rather than just a comment. Focusing on primary sources would improve accuracy, ultimately providing readers with a better understanding of the real impact of a given project.

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

RQ-4: Identify and assess participant subgroups

In designing studies, researchers should identify participant subgroups and explain why they are of interest, preferably based on prior data. Where feasible, the study should have adequate precision and power to reach conclusions specific to these subgroups.

Public comments

Within this Standard, AcademyHealth felt it was unclear whether PCORI was articulating whether researchers should not do subgroup analysis if there was not adequate precision and power to reach conclusions or whether that information should simply be reported. Clarification from PCORI on this point would be helpful. The clarification could distinguish when subgroup analyses are aimed at providing definitive advice for the effectiveness of particular interventions or when such analyses may be useful for learning more about subgroup issues and generating hypotheses for future research.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:23pm

Of importance, the method used to reach that subgroup should be specified, so that readers can distinguish subgroup analyses where findings were derived from a well-thought clinical or policy hypothesis, versus data fishing.

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

RQ-5: Select appropriate interventions and comparators

The interventions and comparators should correspond to the actual healthcare options for patients who would face the clinical decision. The clinical decision should be vital and consequential, and one in which current evidence is insufficient to inform the trade-offs of benefits versus harms associated with the different options. Researchers should make explicit what the comparators are and why they were selected, focusing on clearly describing how the chosen comparator(s) define the causal question, reduce the potential for biases, and allow direct comparisons. Generally, non-use (or no specific treatment) comparator groups should be avoided unless no specific treatment is a likely option in standard care.

Public comments

Within the new phrasing of this Standard, AcademyHealth appreciated the language on interventions and comparators, specifically the mention of health care options, feeling it was sufficiently broad. However, we would suggest altering the first sentence of RQ-5—both to include additional stakeholders and to emphasize the real-world component—to read, “Interventions and comparators included in the study should correspond to the options available to patients, providers, and caregivers.” In addition, we are concerned that the many references to "the clinical decision" exclusively results in this standard being far too limiting in its framing. Not only does PCORI use the term "health decision" in RQ-3, straying from consistent terminology, but also, particularly at the health care organization and system levels, decisions are not solely “clinical” decisions, a term which implies a narrow focus on the actions of just the clinical provider. PCORI’s priorities are far broader, and this Standard should appropriately reflect that breadth for researchers; many PCORI investigations are not directly about clinical decisions but are about particular interventions with non-clinical or clinical staff engaging with patients or families. In these cases, it often makes sense to compare this to outcomes in the absence of such engaging interventions.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:23pm

The researcher should make considerations regarding the availability (present or future), cost, and feasibility of its implementation under different healthcare contexts, e.g., rural areas. This point is essential in that while some interventions might potentially lead to improvement in health quality and safety, there might be a number of impediments in their implementation. Requiring researchers to reflect on those aspects will allow them to focus on potential interventions with a greater likelihood of being useful if proven effective.

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

RQ-6: Measure outcomes that people representing the population of interest notice and care about

Identify and include outcomes the population of interest notices and cares about (e.g., survival, functioning, symptoms, health-related quality of life) and that inform an identified health decision. Define outcomes clearly, especially for complex conditions or outcomes that may not have established clinical criteria. Provide information that supports the selection of outcomes as meeting the criteria of “patient-centered” and “relevant to decision makers,” such as patient and decision-maker input from meetings, surveys, or published studies. Select outcomes that reflect both beneficial and harmful effects, based on input from patient informants and people representative of the population of interest.

Public comments

I believe it is important to emphasize the connection between this recommendation and PC-4: "Support dissemination and implementation of study results." In other words, provided that measures represent something that the population of interest cares about, it needs to be clear how each and every one of these metrics will be reported back to this population. Another important connection might be 9. Standards for Adaptive Trial Designs, emphasizing that designs where analysis is conducted in parallel with data collection are encouraged, as they allow for a more immediate incorporation of study findings into healthcare and policy practice.

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am

---

General feedback on the Standards for Formulating Research Questions

Public comments

Outstanding initiative, very well-formulated.

Ricardo Pietrobon [email protected], Health Researcher, 01/27/2016 - 10:24am