In February 2015, following a public comment period, our Board of Governors adopted PCORI's Process for Peer Review of Primary Research and Public Release of Research Findings. By law, all PCORI-funded primary research (i.e., research asking which of two or more approaches is better) must undergo peer review, and all results must be made available to the public.
The goal of peer review is to ensure that the primary research studies funded by PCORI are held to the highest standards of scientific integrity, methodological rigor, and relevance and usefulness to patients, caregivers, clinicians, and other healthcare stakeholders. Our process includes review of study findings by content experts, methodologists, patients, and other healthcare stakeholders with experience related to the study.
How do we assess scientific integrity? Scientific integrity refers to transparency, completeness, and candor as applied to presentations of the scientific work. From experience with journal peer review, we know that some authors communicate only the findings that present their research in the most positive light and underplay the importance of shortcomings in the conduct of the research. Selective reporting of study results and study limitations might lead a reader to have more confidence in the conclusions than the results warrant. Consumers of research need to know the whole story. We ask research teams to provide a full report on the research, including a full account of any cautions about interpreting the study results.
How do we assess methodological rigor? PCORI has a comprehensive list of methodological standards that our Methodology Committee developed in 2013 and subsequently expanded to cover additional topics. These include both broad standards—those that should be followed by all patient-centered outcomes research—and specific standards, which apply only to certain types of studies. Our peer-review process will evaluate how well the investigators met our methodology standards.
How do we assess relevance and usefulness? Scientific integrity and methodological rigor are specified as goals in our authorizing legislation. Our Board added a third goal when it approved our peer-review process in 2015. The Board wanted to ensure that the final research reports address real problems faced by patients, caregivers, clinicians, and others, and provide information that they can use to make better healthcare decisions. In our peer-review process, we ask patients and other stakeholders serving as reviewers whether the research findings will help people like them make healthcare decisions.
What are the Investigator’s Responsibilities in Peer Review?
The principal investigator, who heads the research team, is responsible for meeting the peer-review milestones laid out in the research contract. These obligations include submitting results tables to ClinicalTrials.gov or another required registry, submitting the draft final research report for peer review, and revising in response to comments from peer reviewers and peer-review staff.
The draft final research report will be due four to six months after the research project ends. If the project is required to submit results for posting on ClinicalTrials.gov or another registry, it must do so at least 30 days before submitting the draft final research report.
The instructions for completing the draft final research report provide a detailed guide to submitting results to a registry and writing the report. Principal investigators should contact the PCORI Peer Review Office if they think that the format prescribed by the instructions does not fit with their study.
Who Are Our Peer Reviewers?
PCORI peer reviewers come from a variety of communities under the healthcare umbrella. Reviewers include:
- Subject matter experts who are familiar with the topic of the research
- Biostatisticians and other methodologists who can assess the quality of the study methods
- Clinicians and other healthcare stakeholders, such as managers of clinics, manufacturers of healthcare devices, and policy makers
- Patients, caregivers, or others with a personal stake in the research topic
Peer review has an important role in ensuring transparency about the quality and usefulness of our research findings. If the research funded by PCORI is to have value for the diverse communities within the healthcare system, we need input from a wide variety of individuals to provide our peer-review process with the knowledge and perspectives they have acquired through their own experiences.
How Does Peer Review Work?
PCORI's Process for Peer Review of Primary Research and Public Release of Research Findings provides a roadmap that reflects the requirements of our authorizing law. Peer review is part of PCORI's program to put knowledge gained from high-quality patient-centered outcomes research (PCOR) into practice.
Our roadmap begins with our peer-review contractor inviting the principal investigator to submit an abstract and other information to our electronic peer-review system, which many medical journals use for their peer review. This allows for the contractor to begin identifying and inviting reviewers for the report up to three months before the principal investigator submits the report.
On or around the due date, the principal investigator submits a draft final research report that includes background about the project and the project’s aims, methods, results, discussion of all study aims, and conclusions. The principal investigator submits the draft report to our electronic peer-review system. At PCORI, an experienced editor reviews the draft report to make sure that the writing is clear and includes all of the necessary components for peer review. After this pre-review, the PI will have two weeks to make changes before the report can move forward to external peer review.
Once the report is deemed ready for peer review, the already-invited peer reviewers receive access to the report in the peer-review system. Once the external reviews are received, the associate editor, with input from PCORI staff, assembles their key concerns and suggestions into a letter to the principal investigator that also includes the unedited reviewer comments. The PI then has 45 working days (about nine weeks) to respond to the points raised and make changes in the draft report.
The associate editor then determines whether the report meets the requirements of peer review or the principal investigator needs to make more revisions. If the revision fails to address major concerns, a second round of peer review may be necessary. Once the report has completed all requirements of peer review, the Program Director for Peer Review and Scientific Publications determines whether it is ready to be accepted as final.
Once PCORI accepts the final research report, PCORI's Patient-Centered Outcomes Research Translation Center completes two summaries of the project—one for the public and one for health professionals—in accordance with our authorizing law. PCORI posts the peer-review summary and later the final research report and anonymized comments from the peer reviewers, along with the principal investigator’s responses, on the project’s web page on the PCORI website.
Draft Final Research Report - Tips for Awardees
Please remember that the draft final research report should be a detailed report of all work completed as part of the PCORI-funded contract. The report should:
- Include a detailed discussion of the study’s adherence to PCORI’s methodology standards
- Describe in detail how the researchers engaged patients and stakeholders in the development, execution, and interpretation of the study
- Include the study protocol as an appendix
- Incorporate all study aims, methods, and results that were planned and executed
- Include the statistical analyses and results, even if inconclusive, for all analyses planned in the study protocol
- Present a complete list of study limitations to put study results in appropriate context
- Draft Final Research Report: Instructions for Awardees
- Step-by-Step Instructions for Awardees: Peer Review of Draft Final Research Report
- Instructions for Submitting Materials to Editorial Manager
- Ancillary Information Conflicts of Interest Disclosure Form Relating to PCORI-Funded Research Project
- PCORI Peer Review Webinar for Awardees (Thursday, May 4, 2017)
Posted: October 10, 2016; Updated: September 29, 2017