PCORI-funded researcher Mike Jones, PhD shares the value of PCORI’s peer-review process, what distinguishes it from journal review, and his advice to others getting ready to go through the process.
How do we assess scientific integrity? Scientific integrity refers to transparency, completeness, and candor as applied to presentations of the scientific work. From experience with journal peer review, we know that some authors communicate only the findings that present their research in the most positive light and underplay the importance of shortcomings in the conduct of the research. Selective reporting of study results and study limitations might lead a reader to have more confidence in the conclusions than the results warrant. Consumers of research need to know the whole story. We ask research teams to provide a full report on the research, including a full account of any cautions about interpreting the study results.
How do we assess methodological rigor? A comprehensive list of The PCORI Methodology Standards that our Methodology Committee developed in 2013 and subsequently expanded to cover additional topics. These include both broad standards—those that should be followed by all patient-centered outcomes research studies—and specific standards, which apply only to certain types of studies. Our peer-review process will evaluate how well the investigators met our methodology standards.
How do we assess relevance and usefulness? Scientific integrity and methodological rigor are specific goals in our authorizing legislation. Our Board added a third goal when it approved our peer-review process in 2015. The Board wanted to ensure that the final research reports address real problems faced by patients, caregivers, clinicians, and others. In our peer-review process, we ask patients and other stakeholders serving as reviewers whether the research findings will help people like them make healthcare decisions.
What are the Investigator’s Responsibilities in Peer Review?
The principal investigator, who heads the research team, is responsible for meeting the peer-review milestones laid out in the research contract. These obligations include submitting results tables to ClinicalTrials.gov or another approved study registry, submitting the draft final research report for peer review, and revising the report in response to comments from peer reviewers and peer-review editors.
The draft final research report is due about three months after data analysis for the research project is complete. PCORI awardees must submit their results to ClinicalTrials.gov or another approved study registry at least one month before the draft final research report due date.
The instructions for completing the draft final research report provide a detailed guide to writing the report. The instructions are quite specific about the structure of the draft final research report, but principal investigators may contact the PCORI Peer Review Office if they think that the format prescribed in the instructions does not fit with their study.
How Does Peer Review Work?
Peer review is part of PCORI's program to put knowledge gained from high-quality patient-centered outcomes research into practice.
Our peer-review process begins with our peer-review contractor, the PCORI Editorial Office, inviting the principal investigator to submit an abstract and other information to our electronic peer-review system, which many medical journals use similar tracking systems for peer review. This allows for the contractor to begin identifying and inviting reviewers for the report up to three months before the principal investigator submits the report.
On or around the due date, the principal investigator submits a draft final research report that includes background about the project and the project’s aims, methods, results, discussion of all study aims, and conclusions. The principal investigator submits the draft final research report to our electronic peer-review system. At PCORI, an experienced member of our peer-review staff reviews the draft final research report to make sure that the writing is clear and follows the prescribed structure outlined in report instructions for awardees. PCORI science staff also make sure that the final report includes all of the funded study activities. After this pre-review, the principal investigator will have two weeks to make any changes before the report can move forward to external peer review.
Once the report is deemed ready for peer review, the already invited peer reviewers receive access to the report in the peer-review system. The associate editor reads through the completed reviewer comments, assembles their key concerns and suggestions into a letter to the principal investigator that also includes those unedited reviewer comments. The principal investigator then has 45 working days, about nine weeks, to respond to the points raised and to make changes in the draft report.
The associate editor then determines whether the revised report has addressed reviewers’ comments adequately or if the principal investigator needs to make more revisions. If the revision fails to address major concerns, a second round of peer review may be necessary. Once the report has completed all requirements of peer review, the PCORI Program Director for Peer Review and Scientific Publications determines whether it is ready to be accepted as final. The director may require another round of revisions.
Once PCORI accepts the final research report, the Patient-Centered Outcomes Research Translation Center completes two summaries of the project—a lay summary for the public and a professional abstracted aimed at health professionals—in accordance with our authorizing law. PCORI will also post the peer-review summary (short summary of the main points brought up during peer review and how they were handled) and later the final research report on the project’s web page on the PCORI website.
Additional Resources
- Webinar: How to Prepare for PCORI Review
With Eric Johnson, PhD; Associate Director & Associate Editor, Editorial Office
December 9, 2019 - PCORI Peer Review: Awardee Preparation for Peer Review
2018 PCORI Annual Meeting Workshop - Instructions for Submitting Materials to Editorial Manager™
- PCORI Policy for Data Management and Data Sharing
- The PCORI Methodology Standards
Posted: March 8, 2019; Updated: December 12, 2019