PCORI

The new proposed Methodology Standards continue to prove a valuable contribution to the field of health services research and to researchers wrestling with how to conduct high quality and relevant patient-centered outcomes research (PCOR). AcademyHealth appreciates the Methodology Committee’s updating of these Standards to guide the field at a time of many changes in the research enterprise. PCOR—like all health services research—has great potential to improve health, but only when it focuses on relevant questions, is produced rigorously, and is disseminated and used widely, rapidly, and by patients, caregivers, and other stakeholders. It is essential that the best scientific practices be applied in order to generate trustworthy evidence. The proposed revisions to the Methodology Standards are useful, but reflect a paradigm that is unduly limited to research on discrete clinical services and interactions. Given that PCORI’s established priorities include assessing systems and eliminating disparities, PCORI needs to consider broadening the paradigm under which the standards are developed. Below are some general thoughts on how AcademyHealth feels the Standards could be improved upon: The Methodology Report would be improved by discussing in more detail the opportunities and rigor of delivery system science, also known as improvement, implementation, or health care delivery science and of embedded research considerations as well as of methodologies beyond traditional trial methodologies (including statistical process control, step-wedge and factorial designs, and new efforts to understand rapid-cycle evaluation, such as the CMS Innovation Center has been using). Additionally, as echoed within many comments throughout the various Standards, AcademyHealth would like to reiterate that research should reflect a wide range of data and methodologies – both traditional and innovative – that support robust, practical, and timely evidence generation. As our previous recommendations state, the Methodology Standards, which are important and complex, could be improved not only by including a descriptive paragraph per category of Standards, but also by reminding researchers that the data and methods to which it is referring are both qualitative and quantitative in nature.

Actionable and Measurable Standards: The current version of the PCORI Methodology Standards identifies a minimum set of practices for conducting comparative effectiveness research (CER). However, it can be difficult if some standards are not “actionable” or “measurable”. For example, IR-5 specifies that researchers “provide sufficient information in reports to allow for assessments of the study’s internal and external validity”, but the criteria for “sufficient” remains in the eye of the researcher and the reader. In contrast, MD-2, which addresses statistical methods for dealing with missing data, outlines which methods are considered valid and which methods are discouraged. Actionable and measurable standards provide a target for decision-makers to judge the usability of the evidence produced and for researchers to aim. Recommendation: We recommend the Methodology Committee engage in a review of the current methods standards to outline the criteria required to meet each standard. For standards in which there are not clear requirements, examples of ways in which the standards would be fulfilled would benefit all stakeholders.

I would like to see methodology standards for qualitative research. Stakeholder engagement often uses qualitative methods, and given the fact that engagement is required in all projects, most PCORI applicants will propose to use them.