The proposed method does not guarantee that the final report will be reviewed by someone with the proper content expertise to judge the research. In addition, there will be only a single methodological reviewer. Medical journals maintain a list of content experts and request these individuals to review articles consistent with their expertise. In this manner, journals work to ensure that articles get the best review possible. Journals also have manuscripts reviewed by multiple reviewers. Every journal editor will tell you it is typical to have significant disagreement between reviewers for a paper, and the editor must use judgment to decide which of the concerns are substantive versus individual opinion. I strongly believe this peer-review process improves the quality of research and the final product.
It is therefore likely there would often be discordance between the recommendations from the reviewers at a journal and the PCORI methods expert. How would these be reconciled? Imagine that a PCORI report is approved, but then a journal requires changes in analyses or additional analyses, and possibly changes in the conclusions and discussions. PCORI's proposed method could result in a published manuscript that differs from the PCORI report, a highly problematic state of affairs.
I suggest you consider an alternative approach that I think would meet statutory requirements, optimize review quality, and meet your goals of rapidly disseminating findings:
1) Methodological review - Doing the methodological review after work has been completed is like checking for defects at the end of the assembly line. Everyone in manufacturing and healthcare quality and safety agree this is an outdated approach. Instead of doing this as your propose, I suggest you require a methods report BEFORE work begins (i.e., patient enrollment, or the start of data synthesis) and require certification after the work has been completed that the investigators adhered to the methods plan. This would obviate the need for the methods review after work has been fully completed, speeding time to dissemination.
2) Manuscript submission - Grantees should be required to submit a manuscript with their main results within 3 months of completion of their work. This manuscript should also be sent to PCORI. The comments from the journal reviewers should be sent to the PCORI methods expert who reviewed the methods report (see 1 above) to make PCORI aware of any other problems in methods that were not identified on the PCORI methods review. In essence, PCORI is doing the required peer-review with supporting information from multiple independent reviewers. This would give the needed content expertise and review by multiple experts. Hopefully, it would also result in reconciling discordant opinions between journal and PCORI methods experts.
3) Final PCORI report - content from the submitted or published manuscripts should be used as the foundation for the final report. In this way, the two are fully consistent. Manuscripts often do not have room to include details of the methods needs for clinicians to understand the generalizability of the results or to implement similar interventions. So, the final report should be an expanded version of the manuscripts or have additional sections required. For most of my final reports for grants, I have taken this approach of using material from manuscripts. The problem is that some researchers are very slow to produce manuscripts. So, PCORI would need to ensure that grantees write manuscripts promptly for this to work.
I think this should be the responsibility of the grantees rather than PCORI staff. You should consider having training sessions for grantees to learn how to do this well, and then have PCORI staff work to improve the initial draft. In my experience with many press releases, I almost always end up re-writing these.
Your proposed timeline is significantly longer than the current state-of-the-art at many journals, in which reviews are completed within four weeks of submission, revisions are required to be made within a brief period of time, an editor expeditiously makes a decision to adjudicate discordant reviews and accept or reject an article, and the article is published online soon thereafter. It is quite possible that the proposed process could slow time to publication in both scientific journals and publication/distribution to patients and stakeholders.
Somewhat Well
I think the prompt dissemination of results is not guaranteed by this process and may actually be slower than the best case scenario of current processes of dissemination in journals
Very Clear
I think this could create a burdensome and duplicative process for grantees who have to reconcile discordant comments from PCORI experts and journal reviewers
It is therefore likely there would often be discordance between the recommendations from the reviewers at a journal and the PCORI methods expert. How would these be reconciled? Imagine that a PCORI report is approved, but then a journal requires changes in analyses or additional analyses, and possibly changes in the conclusions and discussions. PCORI's proposed method could result in a published manuscript that differs from the PCORI report, a highly problematic state of affairs.
I suggest you consider an alternative approach that I think would meet statutory requirements, optimize review quality, and meet your goals of rapidly disseminating findings:
1) Methodological review - Doing the methodological review after work has been completed is like checking for defects at the end of the assembly line. Everyone in manufacturing and healthcare quality and safety agree this is an outdated approach. Instead of doing this as your propose, I suggest you require a methods report BEFORE work begins (i.e., patient enrollment, or the start of data synthesis) and require certification after the work has been completed that the investigators adhered to the methods plan. This would obviate the need for the methods review after work has been fully completed, speeding time to dissemination.
2) Manuscript submission - Grantees should be required to submit a manuscript with their main results within 3 months of completion of their work. This manuscript should also be sent to PCORI. The comments from the journal reviewers should be sent to the PCORI methods expert who reviewed the methods report (see 1 above) to make PCORI aware of any other problems in methods that were not identified on the PCORI methods review. In essence, PCORI is doing the required peer-review with supporting information from multiple independent reviewers. This would give the needed content expertise and review by multiple experts. Hopefully, it would also result in reconciling discordant opinions between journal and PCORI methods experts.
3) Final PCORI report - content from the submitted or published manuscripts should be used as the foundation for the final report. In this way, the two are fully consistent. Manuscripts often do not have room to include details of the methods needs for clinicians to understand the generalizability of the results or to implement similar interventions. So, the final report should be an expanded version of the manuscripts or have additional sections required. For most of my final reports for grants, I have taken this approach of using material from manuscripts. The problem is that some researchers are very slow to produce manuscripts. So, PCORI would need to ensure that grantees write manuscripts promptly for this to work.