Failure to follow good research practices can result in studies that don’t provide findings useful for informing clinical decisions. To avoid such wasted effort, PCORI’s authorizing law required that our research program be based on rigorous scientific methods and that we develop a set of standards to improve the quality and value of patient-centered outcomes research, or PCOR.
As we’ve said for years, methods matter; methodological standards can improve the ways in which research questions are selected, studies are designed and analyzed to address those questions, and findings are reported. Standards also help prevent the use of flawed methods and provide a common set of expectations about high-quality research.
Our Board of Governors adopted our initial PCORI Methodology Standards, which were developed by our Methodology Committee, in November 2012. Because we know that methods innovations evolve quickly, we started working almost immediately on refining those standards. That way, the projects we fund will be even more likely to generate the evidence patients and clinicians need to address questions about what works best for whom and under what circumstances. We recently released an update that refines those original standards and adds a new category for cluster-randomized studies. Now, we’ve posted an updated PCORI Methodology Report that explains the context and rationale for the standards.
Using the Standards
The Methodology Standards establish baseline requirements for PCOR studies. Here at PCORI, we use the standards as we review applications for research funding, monitor our funded research, and review awardees’ final research reports. The report is viewed on our website many times each year, and it is often cited in journal articles.
The standards form the foundation for education about PCOR. The Perelman School of Medicine at the University of Pennsylvania offers a course based on these standards. PCORI also has created a continuing education course to help a wide range of clinicians learn about PCOR standards; more than 200 certificates have been awarded so far. We also created an academic curriculum to help researchers gain knowledge and skills to apply the standards to the design and conduct of research. These instructional materials are appropriate for a diverse audience, including graduate students and healthcare decision makers.
As we’ve said for years, methods matter; methodological standards can improve the ways in which research questions are selected, studies are designed and analyzed to address those questions, and findings are reported.
Preparing the Update
In 2014, the PCORI Methodology Committee began to identify areas of emerging methodological advances and gaps in the current standards. The intention was to prioritize potential new standards that could be helpful to PCORI’s evolving research portfolio. As part of this process, we convened a panel of methodological experts to help us develop a new category of methodology standards on research designs using cluster randomization.
The Methodology Committee also undertook a systematic process to review, revise, and update the original 47 standards in 11 categories. The goals were to streamline and clarify language, align the standards, and reflect advances in PCOR methods. In several cases, standards were merged.
Important changes in the original standards dealt with explaining key components of a study protocol, making study results available in lay-language summaries, specifying the causal model underlying a research question, appropriately using data networks for research, and adhering to best practices for systematic reviews. We also broadened the scope of the original category of standards covering studies of diagnostic tests, and this category was renamed “Standards for the Studies of Medical Tests.”
The new and revised standards were posted for public comment in the first half of 2016. After the comment period, the committee made further revisions. Following the committee’s recommendation, the Board adopted the new set of 48 individual standards in 12 categories.
Randomizing by Cluster
The new section of the updated Methodology Report focuses on cluster randomization, a strategy for evaluating complex approaches to health care in real-world settings. In cluster randomization, patients are grouped into units of care delivery—eg, all patients who receive care from a specific clinician or at a certain practice location. The researchers randomly allocate these care delivery units—rather than individual patients—to two or more groups that then experience different models of care.
This type of design, which requires a more complex analytical approach than designs using individual randomization, has grown in popularity. However, it should be used only when justified by the circumstances of the clinical problem being addressed. Several studies, such as this one, have shown that researchers often fail to present such justification.
Examples of Cluster Randomization in Current Projects Funded by PCORI
|Project Name||Units of Care Delivery|
|Helping Adults with Serious Mental Illness Improve Their Health and Wellness||Rural community mental health centers|
|A Patient-Centered Intervention to Increase Screening of Hepatitis B and C among Asian Americans||Primary care physicians’ practices|
|Improving Communication for Chemotherapy: Addressing Concerns of Older Cancer Patients and Caregivers||Community oncology sites|
|A Community-Based Executive Function Intervention for Low-Income Children with ADHD and ASD||Schools|
|On the Move: Optimizing Participation in Group Exercise to Prevent Walking Difficulty in At-Risk Older Adults||Independent living facilities|
PCORI is committed to evaluating and updating the scientific methods guidance we provide to the research community, including broadening the scope of the standards to include additional areas in which baseline standards can strengthen PCOR studies. The Methodology Committee is now working to develop standards in areas including complex interventions, individual participant-level data meta-analysis, data quality and management, and qualitative and mixed methods. The next update of the PCORI Methodology Standards and Methodology Report is anticipated in 2018.
We are interested in hearing whether you find our methodology standards and report useful and what additional topics you would like them to cover.