Agree, this is consistent with current NIH and other funders policies. Some of these sites want publication of a protocol, but the format is variable, and the completeness of the presentation is variable. There may be risk that the protocol submitted to PCORI may vary somewhat from the protocol submitted to clintrials.gov. Or that protocol modifications may not be in sync. Can the clintrial.gov (or other) serve as 'protocol post' and PCORI point to it? That way, researchers modify one site.
Agree, although there are lots of issues regarding timing. The final report should coincide with the end of the project (and funding). Main results would need to be complete > 3 months prior to end-date since PCORI does not allow no-cost extensions, programmer and statistician time will be needed if there are revisions needed.
This will be a significant educational issue for investigators, since the final report to NIH does not go through such a peer review process. Posting the abstract, but keeping the full report confidential for a year seems appropriate. There will need to be substantial collaboration with journal editors on this process to avoid accusations of prior publication. I assume PCORI is in touch with the appropriate editors and boards. Otherwise each investigator will be lobbying xyz journal regarding timely publication.
Investigators will hopefully publish manuscripts prior to the final report/end of the study and the document encourages this. These may be methods reports, baseline data or ancillary analyses. They will already have been peer-reviewed, and since they are published they can't be changed without going back to the journal for a correction. This is of course complicated and only done when its a serious error. It would be a real problem for a published article and the posted final report to have non-consistent results. Expectations for what should be in the final report need to be reasonable: main study results only and not all of the subanalyses. Otherwise the document will become very long and potentially discourage submission of ancillary publications.
I like the idea of the 500 word abstract as the major posted product.
Recommend that PCORI talk with journals and investigators regarding time to publication and discuss on-line publication options in major journals. An investigator submits to journal A, goes through 3 months of peer review and is rejected. Then goes to journal B, two rounds of review and is accepted. But, the article won't be published for several months. Even in this rosy scenario, one bumps up against one year. Extensions will undoubtedly be asked for, get ready for it. Journals do need to publish more quickly, and hopefully some pressure from PCORI will goad them to do so. However, even the very focused EPC program from AHRQ has challenges in this area.
Journals can tell us their lag from submission to publication, but many articles are submitted to more than one journal in an iterative manner. Rejection by journal A is not always due to major flaws, often its audience and 'fit'.
There was some ambiguity in the document as to whether some of the 'peer' review would be done by PCORI staff. Peer review should be conducted by scientific peers. Program officers have an important role in the research and production process, but its not the same as peer review.
The potential number of rounds of review discussed is daunting. Although reviewers will be instructed to focus on the PCORI methods standards, they are highly likely to come up with lots of possibly good ideas for sub-analyses, control for selection bias, changes in presentation and interpretation, etc. The clinical reviewer may focus more on interpretation, but additional ideas will arise. They are reviewers, its what they do- they will essentially always have helpful suggestions. My anecdote is from the EPC program, where editors conduct a pre-review read with instructions to bring up only major methods problems. But, once they start reading it, they bring up everything, with several pages of comments. All useful at some level, but many minor, editorial, or suggestions regarding emphasis.
Will PI's be instructed to respond to all comments, or just those the program officer views as major issues? The program officer could have an important role here to avoid the process becoming very time consuming without a commensurate increase in scientific quality. When review comments are returned to the PI, instructions to 'focus on comments 1, 7 and 9' could be very helpful. Staff should grade the reviewers (journals often do this), and replace reviewers who are not helpful.
I could easily imagine a 6 month period of multiple iterative reviews, with re-or sub-analyses recommended after the end of funding, which is of course problematic.
Does PCORI plan on a broad range of reviewers or a smaller group who develop expertise and consistency regarding focusing on major methods problems? I would favor the latter, although that would likely require some compensation of the reviewers.
This sounds analogous to the function of the Eisenberg Center for AHRQ, great idea. The same mechanism could be used. Some PCORI awards will be doing their own dissemination materials, and the messaging needs to be consistent.
Agree. Post it once with links.
See comment above regarding timing and journal publication. Because of peer review process by journals, some interpretation and sub analyses may vary between the two publications. Its going to happen, and journals are highly unlikely to defer the peer review process to PCORI. I don't think there is a solution here, just be aware it'll happen.
Linkage to HSRPROJ for publications (planned by NLM) would be helpful. PCORI awards are on the database, and publication links are planned for the future, but not available at present. The more the process can be automated, the better.
I understand PCORI's desire for notification regarding submissions, so as to know what's in the pipeline. However, this is an additional task with more response burden for investigators, as well as large burden for PCORI staff to track all of these materials.
See comments above.
Neither Well nor Poorly
I understand the need to respond to the legislation, but implementation of this policy is going to be challanging. I think the intent of congress was that fatally flawed studies be identified. Focus on that task. PCORI cannot prevent publication, but I assume they could require a disclaimer on the funding acknowledgement in the article if a study is indeed fatally flawed? As written, I worry that this process could become yet another prolonged round of review with only minimal scientific gain for many of the reviews.
Somewhat Unclear
Clear on release, I think you need to work out the details on peer review. Negotiations with journal editors will be important regarding timely publication.
Somewhat Agree
Agree if the peer review process is efficient and focused on major flaws in methods.
This will be a significant educational issue for investigators, since the final report to NIH does not go through such a peer review process. Posting the abstract, but keeping the full report confidential for a year seems appropriate. There will need to be substantial collaboration with journal editors on this process to avoid accusations of prior publication. I assume PCORI is in touch with the appropriate editors and boards. Otherwise each investigator will be lobbying xyz journal regarding timely publication.
Investigators will hopefully publish manuscripts prior to the final report/end of the study and the document encourages this. These may be methods reports, baseline data or ancillary analyses. They will already have been peer-reviewed, and since they are published they can't be changed without going back to the journal for a correction. This is of course complicated and only done when its a serious error. It would be a real problem for a published article and the posted final report to have non-consistent results. Expectations for what should be in the final report need to be reasonable: main study results only and not all of the subanalyses. Otherwise the document will become very long and potentially discourage submission of ancillary publications.
I like the idea of the 500 word abstract as the major posted product.
Recommend that PCORI talk with journals and investigators regarding time to publication and discuss on-line publication options in major journals. An investigator submits to journal A, goes through 3 months of peer review and is rejected. Then goes to journal B, two rounds of review and is accepted. But, the article won't be published for several months. Even in this rosy scenario, one bumps up against one year. Extensions will undoubtedly be asked for, get ready for it. Journals do need to publish more quickly, and hopefully some pressure from PCORI will goad them to do so. However, even the very focused EPC program from AHRQ has challenges in this area.
Journals can tell us their lag from submission to publication, but many articles are submitted to more than one journal in an iterative manner. Rejection by journal A is not always due to major flaws, often its audience and 'fit'.
The potential number of rounds of review discussed is daunting. Although reviewers will be instructed to focus on the PCORI methods standards, they are highly likely to come up with lots of possibly good ideas for sub-analyses, control for selection bias, changes in presentation and interpretation, etc. The clinical reviewer may focus more on interpretation, but additional ideas will arise. They are reviewers, its what they do- they will essentially always have helpful suggestions. My anecdote is from the EPC program, where editors conduct a pre-review read with instructions to bring up only major methods problems. But, once they start reading it, they bring up everything, with several pages of comments. All useful at some level, but many minor, editorial, or suggestions regarding emphasis.
Will PI's be instructed to respond to all comments, or just those the program officer views as major issues? The program officer could have an important role here to avoid the process becoming very time consuming without a commensurate increase in scientific quality. When review comments are returned to the PI, instructions to 'focus on comments 1, 7 and 9' could be very helpful. Staff should grade the reviewers (journals often do this), and replace reviewers who are not helpful.
I could easily imagine a 6 month period of multiple iterative reviews, with re-or sub-analyses recommended after the end of funding, which is of course problematic.
Does PCORI plan on a broad range of reviewers or a smaller group who develop expertise and consistency regarding focusing on major methods problems? I would favor the latter, although that would likely require some compensation of the reviewers.
I understand PCORI's desire for notification regarding submissions, so as to know what's in the pipeline. However, this is an additional task with more response burden for investigators, as well as large burden for PCORI staff to track all of these materials.