IR-1: A priori, specify plans for quantitative data analysis that correspond to major aims

Researchers should describe the analytic approaches that will be used to address the major research aims before analysis is undertaken. These include definitions of key exposures, endpoints, and covariates. Also identify patient subgroups of interest, plans (if any) for how new subgroups of interest will be identified or how analysis plans may be adapted based on changing needs and scientific advances, and plans for how missing data will be handled.

Public comments

AcademyHealth appreciates Standard IR-1 and its attempt to prompt researchers to think through analytic approach issues as well as to define these issues prior to conducting a base analysis. However, we feel that this Standard is missing specified plans for robustness tests. Although PCORI mentions these later in the Standards, specifically when referring to missing data methods, robustness tests are not limited to dealing only with missing data issues. To rectify this omission, PCORI could add a simple sentence, such as, “Researchers should specify their plans for robustness and sensitivity tests in advance of doing these analyses.”

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

---

IR-2: Assess data source adequacy

In selecting data sources, researchers should ensure robust capture of exposure, outcome, and relevant covariates. When statistically adjusting for covariates or confounding factors, measurement properties of the important covariates must be considered to allow adequate adjustment.

Public comments

PCORI’s Standards should reflect a wide range of data and methodologies, and the language used in IR-2 very much implies that all data are quantitative, when qualitative data are equally and often of even more importance. Investigators should specify how their analyses will incorporate and allow for the inclusion of qualitative data or mixed methods, incorporating both qualitative and quantitative data.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

Given the wording of this standard, it appears to be mainly focused on and applicable to secondary use of data, but it does not state this explicitly. We recommend PCORI considers including language in the standard that it is distinctly about the usage of existing data sources. In addition, we suggest adding a new section/standard describing the best approaches to assure data integrity for primary data collection studies. For example, the standard on data registries outlines the unique considerations for primary data collection. We recommend taking that portion of the Data Registries standard and putting it under the suggested new section in the Standards for Data Integrity and Rigorous Analysis, or suggest a new standard focused on primary data collection, and then referencing that standard within the Data Registry Standard.

Eli Lilly and Company, Industry, 03/30/2016 - 2:13pm

---

IR-3: Describe data linkage plans, if applicable

For studies involving linkage of patient data from two or more sources (including registries, data networks, and others), describe

1. each data source and its appropriateness, value, and limitations for addressing specific research aims;
2. any additional requirements that may influence successful linkage, such as information needed to match patients, selection of data elements, and definitions used; and
3. the procedures and algorithm(s) employed in matching patients, including the success, limitations, and any validation of the matching algorithm.

Public comments

When referring to data linkage plans, it’s truly the combination of data sources that matters. To this point, in bullet one, where PCORI says “each data source,” AcademyHealth would recommend instead altering the language to reflect the utility and fitness of the linked dataset as a whole, such as focusing on “the appropriateness and limitations of the data linkage plan,” or language to this extent.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

---

IR-4: Document validated scales and tests

Studies should include documentation of the names of the scales and tests selected, reference(s), characteristics of the scale, and psychometric properties.

Public comments

No comments.

---

IR-5: Provide sufficient information in reports to allow for assessments of the study’s internal and external validity

Reporting guidelines for specific designs can be found at the relevant stakeholders EQUATOR Network website. This website has brought together all reporting guidelines that have been developed using formal approaches, many of which have been adopted by journals, such as CONSORT (for randomized clinical trials), STARD (for diagnostic tests), STROBE (for observational studies), and SRQR and/or COREQ (studies using qualitative research). Researchers should register their studies with the appropriate registry (e.g., clinicaltrials.gov for clinical studies or observational outcomes studies) and provide complete and accurate responses to the information requested (e.g., enter the required and optional data elements for clinicaltrials.gov).

Public comments

There appears to be a discrepancy between Standard IR-5 and other PCORI guidance on reporting guidelines. For example, in much of PCORI’s dissemination and communication work, including in its 2015 document, “PCORI’s Process for Peer Review of Primary Research and Public Release of Research Findings,” PCORI includes mention of both the Registry of Patient Registries (RoPR) as a repository in which “[p]atient registries must be registered” and Health Services Research Projects in Progress (HSRProj) as the database which researchers should use to register “[m]ethodological projects and others that are not appropriate for ClinicalTrials.gov or RoPR.” Yet, RoPR and HSRProj are absent from the Methodology Standards entirely. For consistency with earlier guidance PCORI should account for these sites within the Methodology Standards as well. Furthermore, given that HSRProj houses the largest collection of patient-centered outcomes research projects, AcademyHealth recommends that HSRProj be included as a primary source for observational outcomes studies. Moreover, for easy reference and convenience for users, PCORI may wish to consider providing the URLs to the websites mentioned in this Standard.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

We applaud the Committee on clarifying how patients and stakeholders should be involved in the prioritization of research, conduct of research, and the dissemination of research findings. As outlined in the standards, a broad set of stakeholders, including the biopharmaceutical industry, contribute to a more robust research process. Purchasers, payers, and industry communities seek to ensure that relevant questions are addressed, research findings are usable, and results are translated and implemented in practice. This engagement is an important aspect towards improving the relevance of CER questions and ultimately the impact of CER on health care decision-making. Standards for Data Integrity and Rigorous Analysis: We commend the Committee on the recognition of the growing body of research standards and the need to increase research plan and analytic transparency. These are important steps forward. However, we believe there are other opportunities for the PCORI Methodology Committee to play a pivotal role in the nation’s research enterprise. We commend the broader recognition of standards such as STROBE (for observational research) and SRQR and COREQ (for qualitative research). As outlined in the Affordable Care Act, the methodology standards “shall build on existing work on methodological standards for defined categories of health interventions and for each of the major categories of comparative clinical effectiveness research methods.” Good practices identified by professional societies and consortia such as RECORD (for observational studies), the GRACE Checklist (for observational, the CER Collaborative (for observational studies, indirect treatment comparisons, and modeling studies), CENT (for N-of-1 trials), CONSORT Extension for pragmatic trials, and others are not referenced in this version of standards. , , , , , , Because stakeholders are increasingly looking to PCORI as a leader in the establishment of standards for CER, broader inclusion and recognition of good research practices is needed. 2 Recommendation: We recommend that additional methods or standards developed by other groups be considered and at a minimum referenced to increase stakeholder awareness or adopted in the PCORI methods. Over the past decade, standards for research have proliferated among different research disciplines (e.g., biostatistics, pharmacovigilance, econometrics). This proliferation can increase the reach of standards across the types of research. However, it can also have unintended consequences. For example, a recent study compared and contrasted nine existing sets of standards or guidelines (based on 23 elements) for conducting observational studies. While most guidelines standards agreed on what elements were important (e.g., the need for a study protocol), there was disagreement 52% of the time on how the standards should be acted upon and addressed. This disagreement can contribute to variation in study quality, create discrepancies adopted into care decisions. There is a need to identify common and agreed upon methods for research through consensus-based approaches. An ongoing process enables agreement where quicker consensus is feasible and an iterative process for new, novel, or controversial methods. Recommendation: Stakeholders perceive PCORI to be a leader in establishing CER standards. Few other organizations have the access to research experts from a variety of research disciplines and communities or experience in facilitating multi-stakeholder processes for prioritization as PCORI. We commend the Committee on the advancement of the standards. We believe the Committee and PCORI can offer their leadership to advance a set of common and agreed upon methods through a consensus-based process that will form the foundation for research that the public can use and trust. Finally, we commend the Methodology Committee on the recognition of the importance of research transparency. Without insight into which outcomes were pre-specified and the research analysis process used, the validity of study results may be questioned. The efforts to register studies before study start at clinicaltrials.gov and other sites seek to increase research transparency. Recommendation: Some sites such as clinicaltrials.gov were developed for clinical trials. Although used for observational studies as well, many modifications are needed to for these sites to account for features relevant for observational studies (e.g., propensity methods to balance confounding, analytic definitions to identify exposures). PCORI has an opportunity to collaborate with other entities such as the National Institutes of Health to enhance existing research infrastructure which encourages transparent, reproducible, and credible research.

Jennifer Graff, National Pharmaceutical Council, Industry, 03/28/2016 - 9:44am

---

New IR-6: Masking should be used when feasible

Masking of evaluation staff should be implemented, especially in situations for which study participant and investigator masking are not feasible. When masking is not feasible, the impact of lack of masking on the results should be discussed.

Public comments

First, as written, IR-6 assumes that ‘evaluation staff’ do not write up the results, but this is often the case and should be addressed. Additionally, for clarification purposes within this Standard, AcademyHealth recommends that PCORI change “evaluation staff” to “data collection staff.”

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

---

General feedback on the Standards for Data Integrity and Rigorous Analysis

Public comments

AcademyHealth would like to reiterate from our past comments that research projects funded by PCORI should reflect a wide range of data and methodologies – both traditional and innovative – that support robust, practical, and timely evidence generation. This set of standards could be improved upon by including a preamble stating that PCORI is referring to all kinds of data from all kinds of methods, especially including those of qualitative and mixed methods research, but is also utilizing new thinking in causal methods, including step-wedge and factorial designs and interrupted time series and regression discontinuity statistical approaches.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:34pm

• At a minimum, distinctions should be made between primary and secondary data sources as well as prospective and retrospective observational research within these standards. • In general, there appears to be gaps in the consideration of the unique properties of retrospective database analysis. We recommend reviewing the document A Checklist for Retrospective Database Studies--Report of the ISPOR Task Force on Retrospective Databases to gain further insight on some of the issues and adjust the standards as appropriate. The checklist can be found at: www.ispor.org/workpaper/healthscience/finalreportretror.pdf

Eli Lilly and Company, Industry, 03/30/2016 - 2:13pm