Peer-Review Process Comments (2014)
Public Comment Submissions
Submitted
9/16/14 13:46
Name of person or organization submitting comments:
Arthur Weinstein, MD
Please identify which stakeholder group you represent:
Researcher
Public posting of full final reports on PCORI’s website
I attended yesterday's Board of Governor's meeting and I am very pleased that you now have added a section that posting the full final report will await notice of acceptance of publication by a peer-reviewed journal. It would be comforting to know that the journals, especially New England Journal of Medicine, will accept the posting of a 500 word abstract and a standalone table, prior to publication.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please add any comment about your rating
Since there is no guarantee that the first journal to which the manuscript is submitted will accept it, adherence to flexible timelines rather than rigid timelines should be considered.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
9/18/14 7:07
Name of person or organization submitting comments:
Tim Carey MD MPH
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Agree, this is consistent with current NIH and other funders policies. Some of these sites want publication of a protocol, but the format is variable, and the completeness of the presentation is variable. There may be risk that the protocol submitted to PCORI may vary somewhat from the protocol submitted to clintrials.gov. Or that protocol modifications may not be in sync. Can the clintrial.gov (or other) serve as 'protocol post' and PCORI point to it? That way, researchers modify one site.
Production of a draft final report, scientific abstract, and standalone results table
Agree, although there are lots of issues regarding timing. The final report should coincide with the end of the project (and funding). Main results would need to be complete > 3 months prior to end-date since PCORI does not allow no-cost extensions, programmer and statistician time will be needed if there are revisions needed. This will be a significant educational issue for investigators, since the final report to NIH does not go through such a peer review process. Posting the abstract, but keeping the full report confidential for a year seems appropriate. There will need to be substantial collaboration with journal editors on this process to avoid accusations of prior publication. I assume PCORI is in touch with the appropriate editors and boards. Otherwise each investigator will be lobbying xyz journal regarding timely publication. Investigators will hopefully publish manuscripts prior to the final report/end of the study and the document encourages this. These may be methods reports, baseline data or ancillary analyses. They will already have been peer-reviewed, and since they are published they can't be changed without going back to the journal for a correction. This is of course complicated and only done when its a serious error. It would be a real problem for a published article and the posted final report to have non-consistent results. Expectations for what should be in the final report need to be reasonable: main study results only and not all of the subanalyses. Otherwise the document will become very long and potentially discourage submission of ancillary publications. I like the idea of the 500 word abstract as the major posted product. Recommend that PCORI talk with journals and investigators regarding time to publication and discuss on-line publication options in major journals. An investigator submits to journal A, goes through 3 months of peer review and is rejected. Then goes to journal B, two rounds of review and is accepted. But, the article won't be published for several months. Even in this rosy scenario, one bumps up against one year. Extensions will undoubtedly be asked for, get ready for it. Journals do need to publish more quickly, and hopefully some pressure from PCORI will goad them to do so. However, even the very focused EPC program from AHRQ has challenges in this area. Journals can tell us their lag from submission to publication, but many articles are submitted to more than one journal in an iterative manner. Rejection by journal A is not always due to major flaws, often its audience and 'fit'.
PCORI peer review process of the draft final report
There was some ambiguity in the document as to whether some of the 'peer' review would be done by PCORI staff. Peer review should be conducted by scientific peers. Program officers have an important role in the research and production process, but its not the same as peer review. The potential number of rounds of review discussed is daunting. Although reviewers will be instructed to focus on the PCORI methods standards, they are highly likely to come up with lots of possibly good ideas for sub-analyses, control for selection bias, changes in presentation and interpretation, etc. The clinical reviewer may focus more on interpretation, but additional ideas will arise. They are reviewers, its what they do- they will essentially always have helpful suggestions. My anecdote is from the EPC program, where editors conduct a pre-review read with instructions to bring up only major methods problems. But, once they start reading it, they bring up everything, with several pages of comments. All useful at some level, but many minor, editorial, or suggestions regarding emphasis. Will PI's be instructed to respond to all comments, or just those the program officer views as major issues? The program officer could have an important role here to avoid the process becoming very time consuming without a commensurate increase in scientific quality. When review comments are returned to the PI, instructions to 'focus on comments 1, 7 and 9' could be very helpful. Staff should grade the reviewers (journals often do this), and replace reviewers who are not helpful. I could easily imagine a 6 month period of multiple iterative reviews, with re-or sub-analyses recommended after the end of funding, which is of course problematic. Does PCORI plan on a broad range of reviewers or a smaller group who develop expertise and consistency regarding focusing on major methods problems? I would favor the latter, although that would likely require some compensation of the reviewers.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This sounds analogous to the function of the Eisenberg Center for AHRQ, great idea. The same mechanism could be used. Some PCORI awards will be doing their own dissemination materials, and the messaging needs to be consistent.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Agree. Post it once with links.
Public posting of full final reports on PCORI’s website
See comment above regarding timing and journal publication. Because of peer review process by journals, some interpretation and sub analyses may vary between the two publications. Its going to happen, and journals are highly unlikely to defer the peer review process to PCORI. I don't think there is a solution here, just be aware it'll happen.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Linkage to HSRPROJ for publications (planned by NLM) would be helpful. PCORI awards are on the database, and publication links are planned for the future, but not available at present. The more the process can be automated, the better. I understand PCORI's desire for notification regarding submissions, so as to know what's in the pipeline. However, this is an additional task with more response burden for investigators, as well as large burden for PCORI staff to track all of these materials.
Timeline of the proposed peer review and public release process
See comments above.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
I understand the need to respond to the legislation, but implementation of this policy is going to be challanging. I think the intent of congress was that fatally flawed studies be identified. Focus on that task. PCORI cannot prevent publication, but I assume they could require a disclaimer on the funding acknowledgement in the article if a study is indeed fatally flawed? As written, I worry that this process could become yet another prolonged round of review with only minimal scientific gain for many of the reviews.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Unclear
Please add any comment about your rating
Clear on release, I think you need to work out the details on peer review. Negotiations with journal editors will be important regarding timely publication.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
Agree if the peer review process is efficient and focused on major flaws in methods.
Submitted
9/24/14 14:14
Name of person or organization submitting comments:
Deane K. Felter
Please identify which stakeholder group you represent:
Patient
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Based on my experience as a Patient Reviewer for Cycles 2 & 3 in 2013 and most recently as a Consumer Reviewer for the DoD Discovery Award program,. I would suggest submitting a copy of the Lay Abstract and the report to a one or more Patients for a review to insure clarity as the report is geared to an 8th grade reading level. My experience with Lay Abstracts is that they lean towards being too technical or scientific.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Peer review should come first to insure that the final report is accurate and complete.
Submitted
9/25/14 20:29
Name of person or organization submitting comments:
Janine Chiappa McKenna
Please identify which stakeholder group you represent:
Patient
PCORI peer review process of the draft final report
1. Will there be an IRB requirement for the awarding institution? 2. Would PCORI allow PIs to "take" reviews to a publication, thereby reducing the work of the journal editor in finding reviewers, waiting for the review, etc.? 3. Will PCORI impose a limit on the number of re-analyses of research? 4. What about faulty data? Will the reviewers do anything beyond requesting a revison of the final draft report?
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
10/5/14 18:37
Name of person or organization submitting comments:
David Becker
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Youre recommendations are insufficiently state specific. In addition, I don't agree that just experts should be involved in peer review. Experts have a bandwagon bias and engage in group think. IS it no wonder why the cholesterol myth and dangerous substances like VIOXX have been allowed into te public consciousness? Experts are too focused on their own well being and the well being of their peers. They have lost credibility in the eyes of many Americans. The public good would be better served if the peer review process was not left to the designs of experts.
PCORI peer review process of the draft final report
Once again, experts have their own biased agenda and their politics have done some damage to the public good. An example is opioids- they promote opioids despite the lack of evidence. Why should we trust experts in the peer review process?
Submitted
10/10/14 17:22
Name of person or organization submitting comments:
Carole Price
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
PCORI peer review process of the draft final report
Why is PCORI developing the lay abstracts? The Principle Investigator should be required to work with an advocate. That advocate would help them develop their lay abstract. Then PCORI would offer revisions.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Poorly
Please add any comment about your rating
There needs to be more connection between the public and the researcher through out their project. That includes abstracts and publications.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Disagree
Please add any comment about your rating:
This delays the results getting to the public. Then you require the medical abstract, you should also require the lay abstract. That would cut down on the time delay.
Submitted
10/14/14 14:58
Name of person or organization submitting comments:
Association of Rehabilitation Nurses
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
As outlined in the authorizing legislation, PCORI shall, “convey the findings of research in a manner that is comprehensible and useful to patients and providers in making health care decisions.” Specifically, PCORI is charged with reviewing and editing the medical professionals’ abstract, results table, and ancillary information for patients and general public with readability at the 8th-grade level. ARN is very supportive of ensuring patients and the public have access to research findings and are therefore concerned with the appropriateness of materials prepared at the 8th- grade level. While the average American reads at an 8th-grade level, it is recommended that patient education materials and instructions are written at a 3rd-5th grade reading level.
Public posting of full final reports on PCORI’s website
We would recommend additional strategies and mechanisms for dissemination other than the PCORI web site. Professional organizations and patient advocacy groups could be utilized as catalysts to disseminate research findings to patients and healthcare professionals. Other means including videos, social media, and text messaging could be additional mechanisms for distributing research findings.
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//PCORI Peer Review Comments.pdf
Submitted
10/22/14 15:37
Name of person or organization submitting comments:
Wendy Demark
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
OK - no comments
Production of a draft final report, scientific abstract, and standalone results table
OK - no comments
PCORI peer review process of the draft final report
I was very disappointed in the review. Few experts and mismatched experts to review the science. Seems that if so much money is being allocated, the opinions of experts should be sought.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
ok - no comments
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
cannot comment
Public posting of full final reports on PCORI’s website
cannot comment
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
cannot comment
Timeline of the proposed peer review and public release process
adequate
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Poorly
Please add any comment about your rating
I really had high hopes for this initiative, instead I find that it is overly proscriptive and bureaucratic and it does not seek the input of experts. It is the "no child left behind" of the grant world. Why all the resources were put into this organization instead of just putting them into the NIH and creating an institute for comparative effectiveness is beyond me. As a researcher who has served on both sides of the table, as a submitter and a reviewer, as well as a tax payer, I am gravely disappointed.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Neither Clear nor Unclear
Please add any comment about your rating
see above
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
This is a minor concern
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Of course - this is a stupid question
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Again, please see comments above - the fact that the redundancy of PCORI exists in an age where NIH is nothing less than criminal. As a researcher who is trying to make strides in this era of such austerity, it is a shame that so many resources have been wasted in bringing to bear a mechanism that shifts in its requirements continually, that makes documents and requirements so proscriptive as to be absurd and which then calls reviewers to review the science who have little expertise in the area is criminal. Obviously, I have strong opinions. I will not waste my time in either submitting or reviewing another PCORI grant unless things change drastically.
Submitted
10/23/14 16:09
Name of person or organization submitting comments:
The BMJ
Please identify which stakeholder group you represent:
Journal/Publisher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
We think this is a positive step to address bias in research reporting and encourage PCORI to go further and require prospective registration for all the studies it funds.
Production of a draft final report, scientific abstract, and standalone results table
See uploaded document
PCORI peer review process of the draft final report
See uploaded document
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
See uploaded document
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
See uploaded document
Public posting of full final reports on PCORI’s website
see document
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
see document
Timeline of the proposed peer review and public release process
see document
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
see document
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//Pcori editorial.pdf
Submitted
10/24/14 11:08
Name of person or organization submitting comments:
Mark Creekmore, Ph.D., President NAMI Washtenaw County
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
PCORI peer review process of the draft final report
Do plans exist to assess this peer review process and its effect on the research? It seems that part of PCORI's goal is to change the way research is done and presented. (The PCORI process seems to create a parallel process to the journal publication process.) An assessment of the ability to move the standard process (toward access and use by non-elite groups), to improve research and to speed its dissemination should be done.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
I did not find anything to suggest that research results would be distributed and reviewed by its direct participants, sometimes referred to as "subjects." Lacking access to the actual participants, I would like to see some review by people who might qualify as their representatives. The review conference seems to be an appropriate spot for this to occur. I suspect research might be written differently were researchers to know that this would occur and that those reactions were considered to be valuable.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
It is not clear to me who has access to the ClinicalTrials.gov site where research results (i.e. "the 500-word (lay) abstract, the 500-word medical abstract, a standalone table, and ancillary information") might be reviewed and what controls might exist on its dissemination prior to journal publication.
Public posting of full final reports on PCORI’s website
It is not clear to me how uniform is the restriction by journals about prior publication. The process here does not seem to suggest that exceptions to the timelines might occur for earlier release when submissions have been made to journals which do not restrict prior publication so strictly.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please provide any additional comments that you have not already provided about any aspect of this proposal.
I presume that other, more direct attempts to speed the peer review and publication process exist. There is no question that the peer review process is more successful - better and speedier - in some journals rather than others.
Submitted
10/24/14 15:50
Name of person or organization submitting comments:
Lorraine Johnson
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Absolutely essential.
Production of a draft final report, scientific abstract, and standalone results table
I am concerned that the limitations imposed by the time line of 90 days may be impractical. I have conducted large scale surveys of the patient population and have found that the analysis alone—figuring out what is true and relevant—can take a considerable amount of time and needs to percolate. Hence, I have concerns about the tight timeline creating a “rush to judgment”. Maybe a generous reading of “conduct or receipt” of findings could help here.
PCORI peer review process of the draft final report
I would want patients involved in the peer review. My involvement as a patient in appraisal of research tells me that there can be a considerable gap between the research design and what is of importance to patients. There also can be a considerable gap between the findings and the conclusions of a study, which may simply reflect intellectual bias of the researchers. I would select patients that have experience and knowledge to hold their own in the assessment. Frequently, unempowered, unprepared patients are just by-standers and window dressing in peer review. Get patients with experience to avoid tokenism. I would want patient concerns placed front and center. Each time, someone should ask was the patient perspective considered in terms of adequately expressing uncertainty and the room for patient values and preferences provided in conclusions, was the question important to patients, were the outcomes important to patients, do the findings ring true to patients. I am concerned about the narrowness of the peer review here, which appears to be limited to one methodologist. Doesn’t this create a risk for bias? I am also concerned that there may be an appearance of suppression or content driven modifications required that are perceived as “policing of the science”. This is not what PCOR stands for, but all appearances should be addressed. Perhaps an open peer review with all comments published would address this concern.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Absolutely essential.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Appropriate.
Public posting of full final reports on PCORI’s website
This is appropriate.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
This is appropriate.
Timeline of the proposed peer review and public release process
I believe that the anonymity of peer review is damaging to the integrity of science. Transparency is key. I suggest using an open peer review process where the reviewers know the authors and the authors know who the reviewers are, and, most importantly, the whole thing, comments and all, is published. This lifts the veil of secrecy and allows public scrutiny of all. BMJ has recently adopted this and added patient peer reviews, all published along- side. One unique option to consider would be post-publication peer review like that offered by F-1000. In this system, the entire peer review process is open, public, transparent, and occurs real time. This would address the problem of having a PCORI peer review that conflicts with the “real publication” peer review.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
I see it as a work in progress. It is right where it should be at this stage, but should be modified to address significant public comments.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Neither Clear nor Unclear
Please add any comment about your rating
There is a need for further development here as to both "who" peer reviews and "how" the peer review process is done, for example post-publication.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
The time frame seems tight to do the job right.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Neither Agree nor Disagree
Please add any comment about your rating:
I think you need some quality control, but really need to think through the process, how it might be perceived, and the goals. Data that is incorrect is not useful. So taking the time to ensure the submission is accurate is important.
File Upload
Submitted
10/28/14 10:20
Name of person or organization submitting comments:
Michael J Barry, MD; Chief Science Officer, Healthwise
Please identify which stakeholder group you represent:
Researcher
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
See uploaded document.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
See uploaded document.
File Upload
Submitted
10/29/14 12:51
Name of person or organization submitting comments:
Sandra Spivey
Please identify which stakeholder group you represent:
Patient
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Agree. This makes a lot of sense. If the public is funding the research, the public should be able to access the studies.
Production of a draft final report, scientific abstract, and standalone results table
Agree
PCORI peer review process of the draft final report
Vital. Very well thought through.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
I recommend that patient advocacy groups associated with the type of medical condition covered in the research, are involved in producing the patient-oriented results report. I have found that contract writers don't often understand the nuances of how patients interpret such research.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Agree
Public posting of full final reports on PCORI’s website
Agree
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Agree
Timeline of the proposed peer review and public release process
Agree.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
I am impressed at the level of thought put into this complex process and how stakeholders have been involved in writing and reviewing the process report.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please add any comment about your rating
The layered approach in getting the results into the right hands is exceptional.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Once this process is established, the investigators and awardee institutions ought to include a discussion of possible pitfalls in their original proposal for funding.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Without peer review, the results can be spun out of control by groups who may have not been advised or included in the peer review.
Submitted
11/2/14 20:52
Name of person or organization submitting comments:
Donna Z. Bliss, PhD, RN, FAAN on behalf of the Wound Ostomy and Continence Nurses’ Society Center for Clinical Investigation
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Disagree
Please provide any additional comments that you have not already provided about any aspect of this proposal.
The Wound Ostomy and Continence Nurses’ Society Center for Clinical Investigation (WOCN Society CCI) agrees that PICORI’s proposed approach to adopt the procedure developed by the National Institutes of Health to implement the FDA Amendments Act (FDAAA) to post an abstract and a table of key results from PICORI funded studies through the National Library of Medicine’s clinical trials registry (ClinicalTrials.gov) adequately meets the law requiring dissemination of results within 90 days of study completion without jeopardizing an investigator’s potential for publishing the findings in top tier journals due to an impression of prior publication. We also support the embargo of 12 months after the final report is accepted before posting the full final report on the PICORI website for similar reasons and believe this should be adequate time for a peer-review journal to publish a submitted manuscript reporting the study results. However, the proposed procedures for detailed peer review of the study’s methods in the draft final report after the study is completed seem misplaced considering PICORI was the funder of the research. We recommend that this type of review should take place when a proposal is submitted and prior to funding. In the draft final report, the review investigators should only need to explain any changes or deviations in methods or analyses from what were initially proposed and funded by PICORI. The review by PICORI after the study is completed should focus primarily on these changes. The rather extensive review of methods proposed seems more appropriate for to be conducted journal reviewers that did not have the opportunity to review them previously (prior to funding).
Submitted
11/3/14 9:34
Name of person or organization submitting comments:
Lauren Agoratus
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
We agree with this process of registration on clinicaltrials.gov
Production of a draft final report, scientific abstract, and standalone results table
We agree with the use of a draft final report and abstract. We particularly agree with the use of a stand-alone results table which should offer a succinct visual support, especially for consumers.
PCORI peer review process of the draft final report
We strongly support the peer review process which includes: a. Submitted the draft final report b. Review team c. Review conference d. Review summary e. Revisions to the draft final report f. Formal acceptance g. Begin 90-day period Although we undersand the importance of utilizing "nationally recognized experts in this field, we would strongly recommended including conusmers on the review team as a key component of this proposal is making the information available to the public.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Regarding "translation for patients and the general public" we disagree with a readability level of 8th grade as the recommended readability level for patient materials is 6th grade (source: U.S. National Library of Medicine- National Institutes of Health.) Indeed, heatlh literacy is the largest barrier to health care access. In addition, we recommend that peer review include consumers, as mentioned previously. Finally, a 500 word abstract may not provide sufficient understanding for the layperson who may not have access to the full report. Most important with regard to making the research available would be information to consumers on how this data is helpful to them. This is particularly true of how the data affects subgroups such as underserved populations experiencing health disparities.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
We agree with this recommendation.
Public posting of full final reports on PCORI’s website
We disagree with merely posting on the PCORI and awardee websites. The information can be shared as "news you can use." We would recommend utilizing social media such as Facebook and Twitter for the public. The information must also be accessible to people with disabilities online. Lastly, the information should be translated into other languages such as Spanish to increase public access.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
We agree with the recommendations for the responsibilities of the awardee.
Timeline of the proposed peer review and public release process
We agree with the timeline which will avoid "prior publication" in journals yet also not delay the results to the public.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please add any comment about your rating
Our primary concern is information access to the public.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please add any comment about your rating
No additional comments.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
We agree that the policy on conflicts of interest will faciliate credibility and transparency.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
Again, consumers should be included in peer review.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Thank you for the opportunity to provide input.
Submitted
11/4/14 10:32
Name of person or organization submitting comments:
Ann Bonham, PhD; Chief Scientific Officer, Association of American Medical Colleges
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The AAMC supports PCORI’s goals for PCORI research project registration at the site appropriate to the study design. Specifically, the AAMC supports the Awardee Institutions’ registration of clinical trials or observational comparative effectiveness studies at Clinical Trials.gov, clinical registries at the Registry of Patient Registries (RoPR), and evidence synthesis studies at PROSPERO.
Production of a draft final report, scientific abstract, and standalone results table
As indicated in the draft guidance, the Awardee Institution must submit a draft final report to PCORI within three months of the completion of data analysis specified in the study protocol. The AAMC believes that the requirements for the timing and content of the report as well as the specifications sections listed in the proposed process are appropriate for the purposes of peer review. The proposed process for engaging in the peer review of the draft final report appears to provide reasonable steps through which PCORI can meet its obligations prior to publication to ensure that the evidence and analyses presented support the conclusions of the report and the study methods adhere to PCORI’s methodology standards.
PCORI peer review process of the draft final report
The AAMC supports PCORI’s outlined requirements for submission of the draft final report, the review team, review conference, revisions to the draft final report, and formal acceptance. Given the importance of balancing Awardee Institutions’ administrative responsibilities with meeting federal requirements, the AAMC strongly recommends that PCORI assess the process as implemented after one year to evaluate the effectiveness of the steps in facilitating PCORI’s responsibilities and the consequences of the process.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
One of PCORI’s most critical responsibilities is ensuring that the results of PCORI-funded research are understandable to and effectively disseminated to the patients and populations that could benefit from the findings. In addition to the translation of a medical abstract into a lay abstract for patients and the general public, the AAMC recommends that PCORI provide a description of the context of the research findings, the relevance of the research results, and an easily comparable rating of the strength of the evidence. Posted abstracts should be appropriately indexed and tagged to allow for continued accessibility by patients and the general population. In particular, special attention should be paid to focusing communications for special groups of patients, especially populations where there is a longstanding history of disparities. Patient and/or community review of the lay abstract before posting would further demonstrate their involvement and could improve the uptake.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
In order to ensure consistency among posted information for PCORI supported projects, the AAMC recommends that PCORI take a leadership role, working with its investigators, in posting the abstract, the stand alone table, and ancillary information to clinicaltrials.gov and other sites as appropriate.
Public posting of full final reports on PCORI’s website
The AAMC supports PCORI’s outlined requirements for posting the final report of each study on the PCORI website no later than 12 months after the final report is accepted by the PCORI review board. However, the AAMC also urges PCORI to consider how it can help investigators and others in developing the most effective dissemination strategies for the results of PCORI-funded research. A truly patient centered approach may require a strategy that goes beyond posting results on the PCORI website or publication in a peer-reviewed journal. PCORI could provide examples and assistance to ensure that the results of patient centered outcomes research reach those who could most benefit from the information and who may not be searching for it on the PCORI website.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
We note that PCORI may delay posting beyond 12 months to coordinate posting with publication of a peer-reviewed journal version of study findings. It is possible that due to differing recommendations from PCORI’s peer review panel and that of the journal, the two abstracts may describe or emphasize differing analysis and results. The AAMC recommends that PCORI provide information to describe and explain any differences that may exist in the journal’s published abstract (to which the PCORI website links) and the updated abstract presented on PCORI’s website.
Timeline of the proposed peer review and public release process
We note that PCORI may delay posting beyond 12 months to coordinate posting with publication of a peer-reviewed journal version of study findings. Given the importance of balancing Awardee Institutions’ administrative responsibilities with meeting federal requirements, the AAMC strongly recommends that PCORI assess the process as implemented after one year to evaluate the effectiveness of the steps in facilitating PCORI’s responsibilities and the consequences of the process.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Given the importance of balancing Awardee Institutions’ administrative responsibilities with meeting federal requirements, the AAMC strongly recommends that PCORI assess the process as implemented after one year to evaluate the effectiveness of the steps in facilitating PCORI’s responsibilities and the consequences of the process. The AAMC urges PCORI to consider how it can help investigators and others in developing the most effective dissemination strategies for the results of PCORI-funded research. A truly patient centered approach may require a strategy that goes beyond posting results on the PCORI website or publication in a peer-reviewed journal.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
File Upload
Submitted
11/5/14 9:36
Name of person or organization submitting comments:
David W. Baker, MD, MPH
Please identify which stakeholder group you represent:
Researcher
PCORI peer review process of the draft final report
The proposed method does not guarantee that the final report will be reviewed by someone with the proper content expertise to judge the research. In addition, there will be only a single methodological reviewer. Medical journals maintain a list of content experts and request these individuals to review articles consistent with their expertise. In this manner, journals work to ensure that articles get the best review possible. Journals also have manuscripts reviewed by multiple reviewers. Every journal editor will tell you it is typical to have significant disagreement between reviewers for a paper, and the editor must use judgment to decide which of the concerns are substantive versus individual opinion. I strongly believe this peer-review process improves the quality of research and the final product. It is therefore likely there would often be discordance between the recommendations from the reviewers at a journal and the PCORI methods expert. How would these be reconciled? Imagine that a PCORI report is approved, but then a journal requires changes in analyses or additional analyses, and possibly changes in the conclusions and discussions. PCORI's proposed method could result in a published manuscript that differs from the PCORI report, a highly problematic state of affairs. I suggest you consider an alternative approach that I think would meet statutory requirements, optimize review quality, and meet your goals of rapidly disseminating findings: 1) Methodological review - Doing the methodological review after work has been completed is like checking for defects at the end of the assembly line. Everyone in manufacturing and healthcare quality and safety agree this is an outdated approach. Instead of doing this as your propose, I suggest you require a methods report BEFORE work begins (i.e., patient enrollment, or the start of data synthesis) and require certification after the work has been completed that the investigators adhered to the methods plan. This would obviate the need for the methods review after work has been fully completed, speeding time to dissemination. 2) Manuscript submission - Grantees should be required to submit a manuscript with their main results within 3 months of completion of their work. This manuscript should also be sent to PCORI. The comments from the journal reviewers should be sent to the PCORI methods expert who reviewed the methods report (see 1 above) to make PCORI aware of any other problems in methods that were not identified on the PCORI methods review. In essence, PCORI is doing the required peer-review with supporting information from multiple independent reviewers. This would give the needed content expertise and review by multiple experts. Hopefully, it would also result in reconciling discordant opinions between journal and PCORI methods experts. 3) Final PCORI report - content from the submitted or published manuscripts should be used as the foundation for the final report. In this way, the two are fully consistent. Manuscripts often do not have room to include details of the methods needs for clinicians to understand the generalizability of the results or to implement similar interventions. So, the final report should be an expanded version of the manuscripts or have additional sections required. For most of my final reports for grants, I have taken this approach of using material from manuscripts. The problem is that some researchers are very slow to produce manuscripts. So, PCORI would need to ensure that grantees write manuscripts promptly for this to work.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
I think this should be the responsibility of the grantees rather than PCORI staff. You should consider having training sessions for grantees to learn how to do this well, and then have PCORI staff work to improve the initial draft. In my experience with many press releases, I almost always end up re-writing these.
Timeline of the proposed peer review and public release process
Your proposed timeline is significantly longer than the current state-of-the-art at many journals, in which reviews are completed within four weeks of submission, revisions are required to be made within a brief period of time, an editor expeditiously makes a decision to adjudicate discordant reviews and accept or reject an article, and the article is published online soon thereafter. It is quite possible that the proposed process could slow time to publication in both scientific journals and publication/distribution to patients and stakeholders.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please add any comment about your rating
I think the prompt dissemination of results is not guaranteed by this process and may actually be slower than the best case scenario of current processes of dissemination in journals
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
I think this could create a burdensome and duplicative process for grantees who have to reconcile discordant comments from PCORI experts and journal reviewers
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
11/5/14 10:24
Name of person or organization submitting comments:
Lindsay Hendryx, Louisiana Clinical Data Research Network
Please identify which stakeholder group you represent:
Researcher
Production of a draft final report, scientific abstract, and standalone results table
Stakeholders expressed concerns about the 12-month timeframe for producing the final report. While completing the research process understandably takes time, stakeholders mentioned concerns that after 12 months, "interest may have dissipated or the industry changed.) My personal comments piggyback on what was discussed at the public forum hosted by PCORI on September 29. It seems it would be beneficial to all stakeholder groups if the final report and manuscript publication processes were somehow combined. This would reduce the number of onerous tasks on the research teams and could possibly produce public information on findings more quickly.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
There is no mention of patient involvement in the review and translation process. To remain true to the mission of patient-centeredness and to optimize the knowledge and experience of patients, some considerations include: -Inclusion of patients alongside researchers in the peer review process -Involvement of patients in the translation process; also consider a rubric for research institution approval of PCORI translation: has this translation been reviewed by patient stakeholders at the institution where research was conducted? Is the lay abstract culturally accurate?
Public posting of full final reports on PCORI’s website
Other avenues of public posting should be strongly considered. One suggestion is that any clinic or community venue in which data is collected should be provided with a culturally appropriate, patient-friendly "report" that can be posted and available for patient and public use.
Timeline of the proposed peer review and public release process
Patients and community members have expressed high levels of trust in the scientific process of research. However, several of our CHAB members and other patient stakeholders have expressed a necessity for research findings to become publicly available in a more timely manner. This should be taken into consideration for both patient and research stakeholders.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
I do agree with this process because I feel that peer review is essential to the validity of research, but I also wonder if there might be an "interim" stage, in which funded research is made available to the public (which they have expressed they want in a timely manner) but with a disclaimer that it is under peer review and pending final complete status.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
These comments are a compilation of feedback provided by members of the LaCDRN Community Health Advisory Board, which is our patient representation in governance. Because of the length and literacy level of the proposal, we felt that a more patient-centered version was needed for fair communication with our patient/community stakeholders, so we drafted a summary (uploaded here). Stakeholders were given the option to respond to the summary or to the full proposal, which we disseminated. Our Clinician Board was provided the same documents, but provided no feedback to us; they may have responded directly to this comment form.
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//Peer Review Process Summary.pdf
Submitted
11/5/14 13:11
Name of person or organization submitting comments:
Renaux
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The registrations process should be done but these sites need to be user friendly and streamlined as well as easy to maintain.
PCORI peer review process of the draft final report
The suggestion to conduct peer review of a the final report of the project is burdensome to an already over-burdened system of peer-review process down for grant reviews as well as publication. The review should be conducted by qualified staff members of the funding organization. PCORI should devote more of its effort and funds to funding projects - with the availability of funds there should be a much higher pay line than has been the case.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
A brief summary abstract would be reasonable - more than that is not indicated. Publication in peer-reviewed scientific journals should be the main thrust. There needs to be a significant time period allowed for reporting because projects are usually underfunded in both time and money and therefore processing results takes time.
Public posting of full final reports on PCORI’s website
PCORI should aim for abbreviated reports
Timeline of the proposed peer review and public release process
A reasonably long time period to respond to this (e.g. 1 year after completion) needs to be in place as noted above grants are underfunded in both time, money and personnel these days and digesting the data takes time.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
PCORI should review their initial grant review process- Reviews should be blinded to give all a fair chance at an unbiased review. The pay line should be higher for proposals. PCORI has had unspent money in cycles and seems to take pride in only funding 9% or applications. This is appalling- when funds are available MORE applications should be funded. I
Submitted
11/5/14 13:28
Name of person or organization submitting comments:
Shawntelle Dillon
Please identify which stakeholder group you represent:
Family member or Caregiver
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
In the draft section (Page 7, Item 1. Translation for patients and the general public) regarding translating the medical professionals' abstract into a lay person's abstract geared toward an 8th grade reading level, it is proposed that this will be done by PCORI staff. Might it be possible that this translation could be done by patient/stakeholder members of your review panel pool (for example), then submitted to PCORI staff for final review and editing?
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Submitted
11/5/14 13:31
Name of person or organization submitting comments:
Lawrence Scahill
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Agree
Production of a draft final report, scientific abstract, and standalone results table
The concern for many researchers would be the timeline. It often takes six months to a year to get findings published in a journal. High profile journals may not accept papers - if the data have been released prior to publication.
PCORI peer review process of the draft final report
It was not clear to me who would review the draft report. What is the results of the review?
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
The issue is when - i.e., how son after completing the study? If on clinicaltrials.gov - why duplicate? If not a clinical trial - then it makes sense.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
and the other way around. The abstract on PCORI provides link to clinicaltrials.gov.
Public posting of full final reports on PCORI’s website
If on clinicaltrials.gov - why duplicate?
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Why?
Timeline of the proposed peer review and public release process
This is critical - give investigators time to submit the paper.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
In clinical trials - this is a potential problem.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Disagree
Please add any comment about your rating:
I remain unconvinced about the need/purpose of the "peer review."
Submitted
11/5/14 15:08
Name of person or organization submitting comments:
CaReNet
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
I am well familiar with ClinicalTrials.gov and would easily be able to look for PCORI's process there.
Production of a draft final report, scientific abstract, and standalone results table
This makes a lot of sense.
PCORI peer review process of the draft final report
I like this this idea.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Excellent idea.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Great!
Public posting of full final reports on PCORI’s website
very good
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
yes
Timeline of the proposed peer review and public release process
It seems reasonable
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
as long as the patient/public version is readily available as quickly as possible
Submitted
11/5/14 16:19
Name of person or organization submitting comments:
Nicole Garro
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Maternal and child health information available through mass media is often confusing and contradictory, posing considerable challenges in delivering important research results and related recommendations, which may be based on limited evidence or flawed interpretation. For this reason, the March of Dimes has devoted significant resources to identifying ways to communicate most effectively with multiple target audiences, including pregnant women, healthcare professionals, and caregivers. Our organization has dedicated similar degrees of effort to facilitating the communication and use of maternal and child health best practices for health care professionals. We strongly believe that engaging consumers in the translation process would help ensure that information is clear to both consumers and caregivers. Testing comprehension is a critical step in developing materials to convey information to consumers. Focus groups or other methods of engaging the “average consumer” may be effective means for testing level of comprehension. While assuring the information is written at the appropriate grade level is important, it is also important to convey context and limitations of research to aid in interpretation of these results.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Neither Agree nor Disagree
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Please see attached letter.
File Upload
Submitted
11/5/14 18:13
Name of person or organization submitting comments:
james travis
Please identify which stakeholder group you represent:
Patient
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Excellent
Production of a draft final report, scientific abstract, and standalone results table
Would this also stand as a progress report for further funding?
PCORI peer review process of the draft final report
Excellent
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This should have been done by NIH years ago. When I was a member of a study section in the 1980s this was mentioned but never followed up by administrative staff. If the PRs institution didn't push nothing much was done to show the public how their tax money was being spent.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Great
Public posting of full final reports on PCORI’s website
Great
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Absolutely
Timeline of the proposed peer review and public release process
This must be variable since each press release usually is in concert with publication of findings and can't be set in concrete.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
already mentioned
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
mentioned above
Submitted
11/6/14 7:47
Name of person or organization submitting comments:
Carla Zema
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Fully support
Production of a draft final report, scientific abstract, and standalone results table
Fully support
PCORI peer review process of the draft final report
Fully support - However, the quality of this step of the process is dependent on the quality of and the direction given to reviewers. Having served as both an application reviewer as well as a reviewer for peer review journals, I can see the potential for Applicants getting conflicting information from their application review panel and the reviewers of their study results. PCORI, or their vendor, must provide high quality oversight and direction to reviewers to ensure that feedback from the application review, which help to shape study design and analysis decisions, are considered when reviewing the study results.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Fully support - I caution PCORI in setting a target reading level for abstracts. Strategies to improve the readability, such as using introductory clauses in sentences, can actually raise the measured reading level. Reading level alone is not a perfect indicator of readability for consumer audiences. Efforts to reach a target reading level may be counterproductive to achieving a consumer-friendly readability.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Fully support
Public posting of full final reports on PCORI’s website
Fully support
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Fully support
Timeline of the proposed peer review and public release process
Fully support
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
This is a necessary step since results may change given input from the peer review process.
Submitted
11/6/14 10:56
Name of person or organization submitting comments:
American Occupational Therapy Association (AOTA)
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The American Occupational Therapy Association (AOTA) agrees with PCORI’s processes for registration of trials, registries, and registries.
Production of a draft final report, scientific abstract, and standalone results table
It should be feasible to report primary outcomes within 3 months after the completion of data analysis.
PCORI peer review process of the draft final report
AOTA commends PCORI for its outreach to consumers and stakeholders for peer review. We agree with the processes outlined in the proposal but we do have one comment regarding the provision “PCORI would have the option of engaging a qualified vendor to perform the peer review of draft final reports. However, the review process and expertise of the review team would be the same as if PCORI performed the peer review.” We recommend careful scrutiny of the vendor’s staff expertise because some writers and communication specialists may not fully understand the nuances of the study’s context, methodologies, and limitations.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
AOTA supports PCORI’s knowledge translation product: a “500-word (lay) abstract for patients, consumers, and the general public” and agrees that it addresses the law’s section ‘‘Release of Research Findings”- making the findings comprehensible and useful to patients and providers. We do recognize that “usefulness” will depend upon where patients and providers are on the continuum of health care. The questions patients ask differ markedly in the acute phase of an injury/disease compared to patients with the same condition but in a chronic or rehabilitative phase. Another approach for providing information could be creating products for different stakeholders (patient, provider, etc.) or temporally.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
AOTA agrees with the proposed procedures and timeline for posting results tables to ClinicalTrials.gov and relevant links (i.e., abstract and results table) to PCORI’s website.
Public posting of full final reports on PCORI’s website
We concur with PCORI’s proposed timeline for posting the funded study’s full final report on PCORI’s website. Regarding the issue of withholding the final report so that it will not violate journals’ embargo policies, with the increasing number of online journal issues there may be opportunities for expediting publication of research that has been supported by PCORI. If however, such agreements cannot be reached with journals, it is fair that PCORI may choose to “delay posting beyond 12 months to coordinate posting with publication of a peer-reviewed journal version of study findings.”
Timeline of the proposed peer review and public release process
Some of the proposed timelines may be challenging (e.g., revisions of final draft report) if the scientists do not respond in a timely manner, but overall they seem reasonable.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
AOTA finds PCORI’s proposal of peer review and public release of PCORI’s research findings balanced, maintaining scientific rigor and review with the unique needs of patients and providers.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
11/6/14 15:06
Name of person or organization submitting comments:
Advanced Medical Technology Association (AdvaMed)
Please identify which stakeholder group you represent:
Life Sciences Industry
PCORI peer review process of the draft final report
AdvaMed Comment Letter Attached -- [excerpt] Since PCORI’s inception, AdvaMed has appreciated and supported PCORI’s efforts to be open and transparent in its processes. We have been especially pleased with PCORI’s outreach to the medical device industry for input on proposals and participation in meetings, webinars and other public forums. We therefore find it notable that the Proposal did not include any discussion regarding industry input. As we have commented many times in the past, AdvaMed’s member companies have on staff clinical and scientific experts who are committed to developing innovative devices, diagnostics and other advanced medical technologies and whose valuable clinical experience can enhance the quality of the PCORI research during all phases. AdvaMed recommends that the Proposal be amended to clarify that clinical, scientific and technical experts from device manufacturers may be among those chosen as methodologists or content experts as appropriate to participate on review teams as part of the peer review process for particular studies. AdvaMed can be helpful in facilitating this process by reaching out to our member companies to recommend such experts.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
AdvaMed Comment Letter Attached -- [excerpt] Finally, it is important to remember that PCORI-funded research is intended to be patient-centered. The research findings that will be posted to the PCORI website will be used by real patients, and their caregivers, to help them understand and digest important information so they can make critical health care decisions for their own unique situations and preferences. It is critical that the research abstracts and ancillary information be summarized in such a way that it is clear and accurate, yet understandable to the general public.
Public posting of full final reports on PCORI’s website
AdvaMed Comment Letter Attached -- [excerpt] We note that the final report published in a peer-reviewed journal may differ to some degree from the report that was accepted by PCORI. Such changes may be editorial or more substantial, but either way could render the final published report inconsistent with the final report accepted by PCORI. One solution may be to provide a link from PCORI’s website to the journal publication, noting the potential for such changes. However, inconsistent information could create concern by some parties – e.g., physicians, patients and caregivers – that the information itself is inaccurate. PCORI could also consider accepting the final journal article as the final PCORI report. PCORI will need to develop a systematic way to address such inconsistencies. We are concerned about the posting of ambiguous, inconclusive or incomplete research results. Such findings must be very carefully managed. While there may be some benefit to patients in posting such research findings, these can also be confusing and difficult to understand, particularly if data is interpreted differently in a final journal publication. AdvaMed recommends that PCORI develop approaches for dealing with such research findings, particularly as such research may relate to specific medical technologies. Again, we are happy to work with PCORI to discuss and develop potential approaches.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
File Upload
Submitted
11/6/14 16:27
Name of person or organization submitting comments:
Association of Academic Physiatrists Research Committee
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
From the attached AAP Research Committee comments document - It is not clear how this will be handled if the PCORI funded project is not a clinical trial and therefore not registered with clinicaltrials.gov, such as a developmental or methodology project. The examples given of the types of research funded by PCORI and where it should be registered (e.g., clinical trials, registry studies, and evidence syntheses) don’t appear to address methodology projects supported by PCORI. It seems likely that such studies, while perhaps having immediate relevance to other researchers, may not be immediately relevant to patients and clinicians. Are methodology study results exempt from this requirement or how are they to be handled?
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
From the attached AAP Research Committee comments document - The document suggests that PCORI has and will find ways to make the early peer review and dissemination of findings acceptable to journals that will ultimately publish the study results, which is critically important. One would not wish to create a disincentive for getting good research funded via PCORI if this were seen as potentially jeopardizing the opportunity for publication in high impact academic journals. It will need to be clarified if all journals will accept this process. In particular higher impact journals often refuse publication if there has been any prior dissemination of results. It would be important to develop clear instructions for authors about how they can meet embargo and prior publication requirements of specific journals while also complying with PCORI’s requirements.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
From the attached AAP Research Committee comments document - It appears that the underlying rationale for these requirements is to make evidence rapidly available to consumers of that evidence. However, findings from individual studies may or may not align with the conclusions reached by larger evidence synthesis efforts. In some cases this is due to methodologic weaknesses (e.g., an unblended study may not reach the same conclusions as a blinded study), but in other cases the discrepancies in conclusions relate to differences in patient populations, the nuances of treatment or outcome measurement, or purely random factors, given that p<.05 still allows for 5% erroneous conclusions. Given that the peer review process is presumably limited to the individual study funded by PCORI, it’s not entirely clear whether the “message” of rapid dissemination is, “Given these results here are the ways in which clinical practice should change” vs. “Given these results, this study should be synthesized along with other similarly strong studies on this topic to arrive at recommendations about the ways in which clinical practice should change.”
File Upload
Submitted
11/6/14 17:12
Name of person or organization submitting comments:
Monique Hedderson
Please identify which stakeholder group you represent:
Researcher
Production of a draft final report, scientific abstract, and standalone results table
The scientific application is hard to organize and too long and repetitive. Several things are asked for in multiple places. The methods section is particularly difficult to organize. It is also frustrating that PCORI keeps changing the application every cycle.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Unclear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
11/6/14 17:24
Name of person or organization submitting comments:
American Cancer Society Cancer Action Network
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
See attached
Production of a draft final report, scientific abstract, and standalone results table
See attached
PCORI peer review process of the draft final report
See attached
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
See attached
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
See attached
Public posting of full final reports on PCORI’s website
See attached
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
See attached
Timeline of the proposed peer review and public release process
See attached
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
The effectiveness of the proposal will depend heavily on implementation details (e.g. language level, organization of information, etc.) that are hard to know based on the broad framework proposed.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
See attached
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
See attached
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please provide any additional comments that you have not already provided about any aspect of this proposal.
See attached
File Upload
Submitted
11/6/14 18:18
Name of person or organization submitting comments:
American Nurses Association
Please identify which stakeholder group you represent:
Clinician/Clinician Society
PCORI peer review process of the draft final report
It's important in the PCORI peer review process that there is appropriate participation of consumers as well as mutlidisciplinary experts.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
It's important that PCORI balance the burden with the benefits for following the proposed additional peer review process. It's unclear the extent of PCORI's transparency on making these peer reviews public. Also, it's unclear what action PCORI will take if the peer reviews are negative.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Submitted
11/6/14 18:50
Name of person or organization submitting comments:
Jeffrey Elias
Please identify which stakeholder group you represent:
Training Institution
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Registration of Clinical Trials seems appropriate where the trial fits the registration criteria for a trial and for participation.
Production of a draft final report, scientific abstract, and standalone results table
The production of the final draft report, abstract, and standalone results table seems to be on target for what would be expected based on PCORI mandate.
PCORI peer review process of the draft final report
The PCORI peer review process represents the contracting agency's (PCORI) report and review process regardless of the constitution of review panels and the review panel guidelines (focus on methods). Some issues that could arise: --PCORI peer review not accepted as comprehensive peer review. --PCORI peer review cannot be replicated when data are provided for outside review, or cannot be replicated if data are not provided for outside review. That is, PCORI review will not be replicated as it might be with a manuscript submitted for review at a standing journal with data available. --Overall funding bandwidth too small to have multiple studies replicate or have overlapping data.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Some issues that could arise: --Data translation: Physicians and other care providers will potentially make decisions or provide options based on PCORI findings. Will the reports and the data be clear and valid enough to allow those decisions? Who will decide that issue? --Unlike other research, Patients as stakeholders will have to understand risk and options based on PCORI report otherwise stakeholder promise for inclusion is not supported. Numerical literacy and scientific literacy is a significant issue.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Appropriate as suggested.
Public posting of full final reports on PCORI’s website
This makes reports accessible.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Not clear how this process will work if original data is not shared outside of PCORI. Will data be scrubbed by PCORI for release to public? What would be the guidelines for data scrubbing? Not many journals are likely to be interested in surrendering space to already published data. Larger journals will want the data. Smaller journals will want data released to those outside of those funded so reliability can be assessed.
Timeline of the proposed peer review and public release process
The timeline seems very tight on the contracting end. This might induce mistakes and short-cuts.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Submitted
11/7/14 5:38
Name of person or organization submitting comments:
Martin Gould
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The suggested process requiring that studies be registered at these sites is appropriate for the studies' design.
Production of a draft final report, scientific abstract, and standalone results table
This effort is appropriate in that it does meet PCORI's authorizing legislation and requirements, as well as its intent to promote transparency and "rapid" dissemination of research findings to professional and non-clinical/lay audiences.
PCORI peer review process of the draft final report
This effort is noteworthy in that the process conceptual framework (i.e., Submitting the draft final report, review team, review conference, review summary, revisions to the draft final report, formal acceptance, 90-day period) is eminently logical, feasible and thorough.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This effort in noteworthy in that the approach acknowledges the real fact that research findings or evidence must be usable as well as robust. One of the most over looked aspects of evidence-based medicine or evidence-based practice and its effectiveness is that those who produce and summarize research evidence must attend more closely to the needs of those who might use it. These people include clinicians and patients with a range of health literacy levels.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
To reiterate, the suggested process requiring that studies be registered at these sites is appropriate for the studies' design.
Public posting of full final reports on PCORI’s website
To reiterate, this effort is appropriate in that it does meet PCORI's authorizing legislation and requirements, as well as its intent to promote transparency and "rapid" dissemination of research findings to professional and non-clinical/lay audiences.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
This strategy is noteworthy in that it is should ensure a more transparent, ethical and broad dissemination of research findings and evidence. Such public posting should ensure patients, clinicians, and policy makers that the PCORI-funded research findings and recommendations are trustworthy.
Timeline of the proposed peer review and public release process
This effort is noteworthy in that the process conceptual framework and timeline (i.e., Submitting the draft final report, review team, review conference, review summary, revisions to the draft final report, formal acceptance, 90-day period) are eminently logical, feasible and thorough.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
Nothing to add here.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please add any comment about your rating
Nothing to add here.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Nothing to add here.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Nothing to add here.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
For the past 15 years, there has been an ongoing effort in the field of evidence-based medicine/evidence-based practice to improve the quality of peer-reviewed research through the use of explicit assessments of the quality of research data and findings on which "evidence of effectiveness" is based. At this point in time, some may argue, there is a risk that too much evidence is being (randomly) is subjected to too many evidence-based practice guidelines which ultimately mitigates against the: (a) meaningful determination of the robustness of the research or the strength of the research findings level of evidence; (b) quality of the peer-reviewed final product; (c) value-added or utility of the evidence to medical clinicians, health practitioners, or patients; and (d) ability of clinicians, practitioners and patients to readily integrate the research evidence into daily practice. PCORI can avoid having its research findings and evidence falling prey to these conditions. On way to mitigate against such sheer randomness and variability, and to promote rapid and effective uptake by key stakeholders is to ensure that: (a) the peer-review process include a clear focus on strength-of-evidence and outcomes (measures) that matter to patients such as quality of life, symptom improvement, cost/expense reduction, morbidity, and mortality; (b) patients are integrated into every facet of the PCORI peer-review and report production process; and (c) there be some framework or taxonomy be created that allows for the peer-reviewed report and evidence to be cross-walked or cross-referenced to other (like) bodies of evidence (e.g., Cochrane's, BMJ's, etc.)
Submitted
11/7/14 7:39
Name of person or organization submitting comments:
John Mortimer
Please identify which stakeholder group you represent:
Patient
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Please implement these proposals - full transparency is good science and without good science we cannot get good medicine
Submitted
11/7/14 8:37
Name of person or organization submitting comments:
ThinkWell & Empower2Go
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
Yes, ThinkWell Http://ithinkwell.org and Empower2Go http://empower2go.org support registration. We also see more accountability to ethical process as needful. The public is in need of ways to validate the research methods through an oversight organization as many questionable trials are offered with MDs as medical directors. There is a conflict as the patients have been taught to respect an MD’s authority, which translates to if an MD has his/her name on it, the trial is registered, the inventor has published papers and the trial was submitted to an IRB this must be good research. Sadly this is not always true. As for trial expenses, compensation for adverse events and costs for pre and post trial testing as well as inclusion/exclusion criteria should be clearly laid out upfront and the terms of engagement standardized. For example if a patient in a trial is expected to pay pre-testing and costs of the medical device or the intervention or biological materials but the trial itself is “free” how can the public differentiate appropriate from inappropriate research when they are asked by an ethics challenged, non compliant group to pay upfront and be included as part of a trial? Problem Solving Suggestions: • Competing interest declaration. At the very least, if a company is directly profiting from a trial itself then that competing interest should be prominently displayed in the listing or be disallowed. The public has a right to know. • No use of clinical trials listings as marketing tools.. If patients are convinced to pay for the via website promotional material indicating that they are legit because their trial is listed the listing should be deleted. • Keeping the food and drug administration informed about new trials as they are listed. In this way the regulatory agencies can examine whether they need to investigate these new trials for compliance and would have all information about investigators, intervention and geography on hand. This would save time, resources and money in the event of complaints or questionable practice. • That it be suggested that governmental or academic funding be conditional on trial registration, institutional review board approvals and that investigators maintain a track record or registering all trials and reporting trial results within 12 months.
Production of a draft final report, scientific abstract, and standalone results table
Yes on the same site where the trial is registered and in a standardized format
PCORI peer review process of the draft final report
Yes for anything PCORI funds
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Not only comprehensible but standardized so the results can be compiled for use in other publications
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
We maintain that registration be enabled on a free database and that the reporting of the trial results take place to this trial registration site.
Public posting of full final reports on PCORI’s website
Yes with access to de-identified trials data
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Yes
Timeline of the proposed peer review and public release process
We suggest submission of publication to registration site 12 months post trial and not 30 months as is sometimes agreed to unless there are clear and reasonable circumstances for a delay
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
This needs to be more clearly defined and a dynamic process so we see what works in practice
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
Fine for a PCORI standard with PCORI funding however rather vague about how this will be paid for and who is ultimately responsible. It would be tragic if it adds extra fees to the obligations of already underfunded entry level researchers or graduate student research
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
as above
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Neither Agree nor Disagree
Please add any comment about your rating:
This would be fine as long as the peer review does not slow the process substantially, a time deadline/standard for peer review is needed, also this restricts peer review to PCORI and does not leave the results open to other stakeholders which could be a detriment for the promotion of transparency
Submitted
11/7/14 9:41
Name of person or organization submitting comments:
Caroline Struthers
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
I fully support this proposal
Production of a draft final report, scientific abstract, and standalone results table
I fully support this
PCORI peer review process of the draft final report
I fully support this
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This is absolutely crucial
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
I fully support this
Public posting of full final reports on PCORI’s website
I fully support this
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
I fully support this
Timeline of the proposed peer review and public release process
There must be enforcement of the timelines, and effective sanctions if they are not met without legitimate reason
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Any competing obligations should be declared before funding is awarded, and an evaluation of feasibility given at that point. If it is not feasible to meet the obligations of the proposal, then the research should not be funded.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
11/7/14 9:43
Name of person or organization submitting comments:
Nancy Keating on behalf of the Society of General Internal Medicine
Please identify which stakeholder group you represent:
Researcher
PCORI peer review process of the draft final report
Note: these comments are also uploaded as a word file. The legislation regarding PCORI peer review is not ideally worded and the process for peer review at the end of a research project will be complicated to implement. We appreciate the effort of PCORI to avoid violating the law and still let researchers do their work. Nevertheless, there remain concerns. Concerns • Projects have many pieces and analyses are never all done at once, and often work is done well after a grant ends. • One solution would be to wait until all work is done, and submit the report, but that will not likely achieve the goals of the legislation. • The draft policy suggests that the review done at the end of the process will consider the entire project; however, the protocol and analysis plan were already peer-reviewed at the time the grant was funded. • It is unclear who the candidates for the review will be and if PCORI will be able to maintain a large group of highly qualified reviewers. This program is likely to be a substantial administrative burden for PCORI. • There is no plan to dissolve disagreements between investigators and reviewers about whether the methods followed PCORI standards, which could be prone to error and bias. • The exact “end date” is not clearly specified. Suggestions • Allow projects to submit work in pieces, perhaps by Aim. • Focus this stage of peer review on the analysis and interpretation, since the protocol already underwent peer review, and after analyses are done, it is very likely too late to change the study design. An exception would be for projects where the protocol changed following funding; this should be described in the submitted report. • Consider an open peer review process where researchers can nominate the methodologic reviewer and the reviewer is known to the researcher. • Consider providing payment to reviewers to increase the likelihood of having a large number of highly-qualified reviewers. • There should be a process to resolve disagreements between investigators and reviewers about whether the methods followed PCORI standards. • There should be a process for investigators to request delay of publishing of findings of the PCORI review if a paper is still undergoing peer review for publication so that the PCORI report is not published before the work can be published in the peer-reviewed literature. • There should be some consideration regarding analyses that are conducted more than 90 days after the project ends. • Clarify the “end date” as after the end of any no-cost extensions. • Juggling the journal peer review process with the PCORI peer review process is likely to be complex for researchers; anything that can be done to simplify the PCORI process would be useful to help maintain the scientific productivity of PCORI researchers.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Neither Clear nor Unclear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
but timing of posting must be coordinated with researcher who is trying to publish their work
File Upload
Submitted
11/7/14 10:44
Name of person or organization submitting comments:
Lisa Simpson MB BCh MPH FAAP, President/CEO, AcademyHealth
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The proposed process appropriately requires that PCORI funded researchers make the public aware of their studies and the questions to be addressed by requiring the awardee institutions to register the studies at sites appropriate for their design, including Clinical.Trials.gov for clinical trials or observational comparative effectiveness studies, RoPR for patient registries, and PROSPERO for evidence syntheses and systematic reviews. However, not all types of patient-centered outcomes research studies are represented by these sites. AcademyHealth strongly recommends that HSRProj—Health Services Research Projects in Progress—be added to the list of registration sites for PCORI funded studies. HSRProj contains descriptions of more than 12,000 ongoing and completed health services research projects funded by federal and state agencies, foundations, universities, private organizations, and international organizations. Funded by the National Library of Medicine and coordinated jointly by AcademyHealth and the Cecil G. Sheps Center for Health Services Research at the University of North Carolina Chapel Hill, HSRProj includes the most up-to date information on ongoing projects in the field of health services research, including comparative effectiveness research and patient centered outcomes research. Already, HSRProj includes information on 279 PCORI-funded projects (http://vsearch.nlm.nih.gov/vivisimo/cgi-bin/query-meta?v%253Aproject=hsrProj&v%253Asources=hsrProj_db&input-form=advanced-hsrProj&query=&binning-state=agency%253d%253dPatient-Centered%20Outcomes%20Research%20Institute%20%2528PCORI%2529). As a leading resource for researchers, policymakers, funders, and other stakeholders interested in understanding the landscape of ongoing health services research, HSRProj should include ongoing PCORI-funded studies.
Production of a draft final report, scientific abstract, and standalone results table
The proposed process and deliverables strike an appropriate balance between timeliness and the realities and complexities of conducting research. First, the proposal adheres to the spirit and the letter of the Affordable Care Act to generate timely, relevant, and understandable research for general consumption by (1) requiring that researchers submit a 500-word abstract and standalone results table within 90 days of completion of data analysis, and (2) that these materials be translated and disseminated via PCORI.org within 90 days. At the same time, PCORI recognizes the realities of research by (1) allowing PCORI-funded researchers to seek publication in journals concurrent with PCORI peer review of the results and (2) delaying the publication of full research articles on PCORI.org until the results are published by a journal or 12 months after PCORI’s publication of the abstract and results table, whichever comes first. Peer-reviewed, scientific journals are the preeminent mode of dissemination for members of the scientific community, and journal publication remains the cornerstone of scientists’ professional development and their career advancement. In developing a peer review process that respects scientists’ need to “publish or perish,” PCORI ensures that the nation’s best and brightest researchers will actively seek opportunities to conduct PCORI-funded research and contribute to the knowledge base critical to improving health outcomes and care. To further strengthen the proposal, AcademyHealth urges PCORI to consider best practices of federal agencies that fund research and obtain written assurances from major journals or the International Committee of Medical Journal Editors (ICMJE) that they are in fact willing to publish a full article even after the abstract and results table are published on PCORI’s website. In addition, PCORI could make funding opportunities even more appealing to researchers by explicitly allowing for the presentation of preliminary findings and/or final results at professional meetings prior to publication. As with journal publications, scientific presentations are essential to the professional development of researchers. In recent years, there has been a trend toward greater restriction of scientific presentations by federal and other funders that require pre-approval of such presentations or prohibit the presentation of results, altogether. This trend is slowing our learning and hampering efforts to accelerate the adoption of proven strategies in clinical and systems practice. Thus, in the interest of transparency, supporting rapid learning and broad dissemination, AcademyHealth recommends that PCORI adopt a policy whereby its researchers are allowed to present findings between the time that the initial abstract and results table are submitted and the final report is published by PCORI and/or the scientific journal, whichever comes first. In addition, we recommend that there be no required “pre-approval” of presentations by PCORI staff, as such prior review is contrary to the fundamental principle of scientific freedom and would slow progress in the mission of PCORI.
PCORI peer review process of the draft final report
As currently drafted, the process is conspicuously vague about specifically who will serve on the peer review team as “content experts,” other than a “methodologist selected by PCORI from among nationally recognized experts in the field” to conduct the methodological review. AcademyHealth’s Methods Council (http://www.academyhealth.org/about/about.cfm?itemnumber=2533) —comprised of experts in the variety of fields that represent health services research and health policy—was established to assist AcademyHealth in developing strategies for professional development in health services research methods. The Methods Council developed guidelines for methods reviews that we hope will serve as a useful reference for methods reviewers on PCORI review teams, available at: http://www.academyhealth.org/files/pdf/HSRMetSugGuide.pdf. One of the key elements of this guideline is the recommendation that editors work with reviewers to ensure reviewers are appropriately familiar with the methods addressed in the study, and that all reviews should provide substantive information to support comments related to the project’s methods. AcademyHealth believes that “content experts” as written is too broad. To preserve the integrity of peer-review— considered globally as the “gold standard” across all scientific disciplines of science—AcademyHealth recommends that the review team be composed of independent, scientific experts and not PCORI staff or contractors. PCORI program officers have an important role in the research and production process, and contractors may be used to manage that process under PCORI’s direction. But the quality, rigor, and soundness of the science should be judged by independent, scientific peers.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
The proposal as currently drafted does not address whether, or at what stage, in the peer review process the studies’ raw data and code would be made available to reviewers and/or the public to see if results are reproducible. Such raw data may be provided to reviewers so they may attempt to reproduce the results as part of their scientific review, very few health and medical journals (e.g., Journal of the American Medical Informatics Association), release data and code publicly for all articles, or in a format that is readily accessible. In other scientific disciplines, including the physical sciences, raw data are often made available publicly at the time of journal publication for purposes of reproducibility (see: http://openeconomics.net/resources/data-policies-of-economic-journals/ and http://www.nature.com/nphys/about/ed_policies/index.html). Some journals are even going so far as to integrate data and code to produce “executable papers” that will promote transparency and reproducibility (see: http://www.elsevier.com/physical-sciences/computer-science/executable-papers). AcademyHealth supports transparency and accountability in science, and making raw data available for purposes of reproducibility will only strengthen public trust in PCORI and the science it supports. However, it is likely possible that some PCORI-funded research will rely in part on datasets, e.g., Medicare claims, where the researchers are prohibited by the data use agreements they sign from re-releasing the data publicly. Therefore, AcademyHealth suggests that PCORI make its studies’ raw data available publicly where possible, bearing in mind that exceptions must be made for data where there are restrictions on re-release in data use agreements to protect privacy as defined by the Health Information Privacy and Accessibility Act (HIPAA). AcademyHealth also recommends that PCORI look to other scientific disciplines—particularly the physical sciences—to understand best practices and inform further development of its policy on raw data release and reproducibility.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
AcademyHealth agrees with proposal, with suggestion to add HSRProj (see above).
Public posting of full final reports on PCORI’s website
AcademyHealth agrees that it is appropriate for PCORI to post the final report on its website, and as previously discussed above, we strongly support PCORI’s proposal to allow PCORI-funded researchers to seek journal publication before the final report is released. One key challenge with this model, however, is that the final report that is approved and posted on PCORI.org may look very different than the article that is published by the journal. Peer review is an iterative process whereby, depending on the reviewer(s) and the specific feedback, researchers may be asked as a condition of journal publication to run new analyses that generate new findings and/or substantively expand/modify the discussion of the findings and their implications. These new findings and additional context or updated interpretations, once published in the journal, may then be perceived by the public as inconsistent and/or conflicting with the findings from the original analyses in the final report posted by PCORI. Such perceptions could erode public trust in PCORI and patient-centered outcomes research. To ensure the transparency, clarity, and public trust of research results, AcademyHealth recommends that PCORI require researchers seeking journal publication to include a “reconciliation statement” for inclusion in the PCORI final report and the published journal article that highlights any substantive differences and/or changes in the findings and/or interpretation and conclusions.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
AcademyHealth agrees, with caveats outlined under “public posting.”
Timeline of the proposed peer review and public release process
AcademyHealth agrees that the proposal is consistent with both the spirit and letter of the law.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Neither Clear nor Unclear
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
The feasibility will in great part depend on how scientific journals will view this process, and their willingness to publish findings from PCORI studies. Should preeminent health journals act restrictively, research institutions may be unwilling to support their investigators in pursuing PCORI-funding opportunities.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
See above, as long as the peer review is by independent scientists external to PCORI staff and/or contractors.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
PCORI has laid out a reasonable approach for a peer review process that is quite new to the field of health services research, and science more broadly. AcademyHealth believes it will be critical to ensure that the implementation of the process be accompanied by a robust system for monitoring and evaluating the process—to include input from researchers, journals, and end user audiences such as patients and health care providers—so that information is made available for mid-course corrections and continuous improvement in the approach.
Submitted
11/7/14 11:05
Name of person or organization submitting comments:
PCORI Advisory Panel on Clinical Trials
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
design. Grantees should publish both positive and negative results. Postings on the PCORI website should be easy to find and include a link to relevant external posting(s). Registration at ClinicalTrials.gov should take place prior to the enrollment of the first patient (as is being required by many journals now), and ideally much before that. In addition to addressing issues related to transparency and ensuring that the study design and objectives are stated up front, such postings may be helpful for subjects who may be interested in participating in such studies. PCORI may want to consider revising the time requirement to also align with project milestones. Agreement regarding the relevant external posting site should be agreed to up front and be part of the contract between PCORI and awardees, and should be a milestone for PCORI awards.
Production of a draft final report, scientific abstract, and standalone results table
Final reports by grantees to PCORI should include a scientific abstract, a summary of results written in lay language, and a results table. The results table should include essential elements for conducting a systematic review. The format of the draft report should contain a clear re-statement of the study objectives or a link to the CT.GOV posting at the front. The discussion section should put the study findings into the broader context of the prior literature. PCORI should be funding research that advances the science for a given research topic. So it is important to review the results and contrast those results with other relevant studies. As currently proposed, limitations of the study appear within both the discussion and conclusions sections, which may lead to some confusion and has the potential to dilute such discussion. Limitations need to be clearly stated and understood. PCORI may want to consider having limitations as a stand-alone section, or provide clarity on where they should be presented. While the discussion section also includes “considerations specific to certain sub-populations”, PCORI may want to consider “implications for relevant stakeholders” – which is not adequately captured with the current format. Inclusion of such a section may help to ensure that PCORI funded research adequately meets the needs of patients and relevant stakeholders.
PCORI peer review process of the draft final report
PCORI should give feedback to grantees on final reports. This should include suggestions regarding the success of the overall study design and patient engagement as well as specific feedback on the researchers’ success in enrolling a diverse population of patients when appropriate. PCORI has an important role in helping to advance methodological standards for patient-centered outcomes research. As the methodology standards will continue to evolve over time, it is important that a particular study be evaluated based on the methodology standards that were in place at the time that the study was awarded, and not to judge the study and its design by standards that did not exist at the time. For the peer-review process, while PCORI will be following a process similar to that used by medical journals, it is important to note that there may be opportunities for PCORI to improve upon some of the challenges/issues with peer-review by medical journals. 1) Effective peer-review should ensure that the study design and objectives are clearly described and appropriate, that results and interpretation of the results are sufficiently detailed, consistent with the study design and free of bias, and that the conclusion is aligned with the study design and results, and avoids unsubstantiated interpretation and speculation. 2) PCORI should consider the appropriateness of the peer-reviewers based on their qualifications and expertise as so-called content experts. 3) There can be better “quality controls” to ensure that manuscripts are being reviewed by appropriate referees and that they have done an adequate job. Although the assigned “methodologist” and program officer will be responsible for ensuring that the study design and methods adequately conform to PCORI’s methodology standards – it is important that the “content reviewers” also be familiar with PCORI’s values and methodological standards. One recommendation is to ensure that all peer-reviewers go through training on the PCORI values and methodological standards. PCORI envisions involvement of patients and patient-representatives throughout the entire research process, but patients and patient-representatives are noticeably absent from the peer review process. Although the proposed process does incorporate lay summaries of the research, further consideration should be made to how patients (and other relevant stakeholders) can be involved.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
One of the most important aspects of research is sharing the results with the public. This not only leads to a better-informed populace, but it may also help in recruiting people to future studies. PCORI should give specific guidance to investigators on how to prepare reports for the public that are written in lay language. In terms of participant notification, PCORI should require that some results be shared with participants prior to publication of those results. Such sharing should be done in a way that is consistent with journal policies on prior publication. Specific guidance should also be developed for investigators on how to report such information to study participants, including an “opt out” procedure on consent forms. PCORI need not start from “scratch;” NIH DAIDS and its Office of HIV/AIDS Network Coordination as well as the non profit CISCRP has resources and policies on this issue. PCORI should also require that clinicians and their professional societies be included in the active dissemination of results. In particular, clinicians will be an important link to patients and will likely help translate the results of trials to their patients.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
This should be required even if, as is well known, many studies have negative outcomes. The public should be made aware of all results of the study, positive and negative. Other researchers may also benefit because this may result in fewer repetitive studies.
Public posting of full final reports on PCORI’s website
PCORI should post results and publicize the results of PCORI-funded trials. Publication of final reports on a central site will help the general population and the research community learn about results of research. The final reports can be posted on PCORI’s website and also on other websites that collect data on best practices, e.g. rtips.cancer.gov/rtips, AHRQ, SAMSHA, etc. PCORI should create a webpage for each project that includes timelines, status updates, milestones (including accrual), and links to every conference presentation & journal article by the investigators. Study subjects should be given the URL. This should also include a public notice if a trial is prematurely stopped for any reason, including safety, insufficient patient recruitment, or budgetary concerns.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
All research results must be shared in a public forum and arena, regardless of whether the research resulted in positive or negative results. PCORI awareness of all article submissions will help this process.
Timeline of the proposed peer review and public release process
It is pertinent to share outcomes of the research conducted through PCORI. However, an adequately prepared report cannot be completed within the proposed three month period. A sound analysis of completed studies will not be as detailed if put together in a short window. The proposed timeline for the full report should be extended.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
There is repetitive language used in much of the PCORI proposal. This language makes it difficult to understand the main points of the proposal.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
The PCORI proposal is both reasonable and feasible though the timeline should be adjusted. Researchers should have one year, rather than 3 months, to prepare final reports and publicize study results.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
The timeline proposed by PCORI should be revised, as noted above.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
In general, rigid rules here might not fully appreciate some of the complexities of the subject. It would at least be helpful if the 90 day rule could be interpreted as 90 days following the publication (or acceptance) of research findings in a peer-reviewed journal. Arguably research results would not be finalized until they have gone through that journal peer-review process and so would not be considered final until the final version of the paper is ready.
File Upload
Submitted
11/7/14 12:12
Name of person or organization submitting comments:
AllTrials Campaign
Please identify which stakeholder group you represent:
Researcher
Please provide any additional comments that you have not already provided about any aspect of this proposal.
It's great to see PCORI joining so many other organisations in recognising the unacceptable adverse impact on patient care that arises from unreported trials. However these proposals could and must go further: PCORI must require reporting of all previous trials it has funded. There is no excuse for permitting such important information to be withheld from doctors, researchers, patients and payers.
Submitted
11/7/14 12:17
Name of person or organization submitting comments:
National Consumers League
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
We agree with requiring the registration of studies at NIH’s clinical trials.gov, Registry of Patient, or Prospero. This provides yet another avenue for making the public aware of studies and the questions to be addressed. Because the awardee is responsible to register, we recommend that PCORI establish a process that ensures that registration has actually occurred.
Production of a draft final report, scientific abstract, and standalone results table
No comments
PCORI peer review process of the draft final report
We agree with the proposal that research findings undergo peer review before the findings are posted. Peer review ensures that research and evidence is credible and trustworthy, thus making it easier to disseminate and implement new findings. However, we do ask for clarification about who is included in the peer review team. It is not currently apparent from the proposal if peer review includes patient or consumer representation, as does the merit review process for PCORI proposals. The legislation states that the peer review “shall be composed of experts in the scientific field relevant to the research under review.” We suggest that the true expert is the patient and that to stay true to PCORI vision of engaging patients in every step of the process, patients should be part of the peer review team. Including patient and consumer representation in the peer review process, as the Cochrane Collaboration does for their systematic reviews, is another opportunity for patients and consumers to ensure that the research is relevant to them and focuses on meaningful outcomes. As an integral part of the peer review, they can better support research within their community and disseminate results of that research back to the community. Transparency in the peer review process can impact how the research is ultimately received and accepted by consumers and patients. Transparency builds trust and credibility and is how PCORI and researchers are held accountable. Greater trust and credibility makes dissemination and implementation of evidence easier.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
We appreciate the emphasis on making research findings comprehensible and useful to patient and providers in making health care decisions. For the lay abstract we suggest that the 8th-grade reading level be the maximum level, and consider a lower level as well, for text as well as tables and ancillary information. One out of five Americans read at the 5th-grade level, the average American reads at the 8th-grade level. For older Americans, two out of three read below 5th-grade level. In addition, make sure the language used in the lay abstract is accessible and culturally sensitive. We suggest PCORI consider publishing lay abstracts in additional languages besides English, especially Spanish. To ensure the research results can be used by patients as they make health care decisions, PCORI should consider including the following in any results summary and/or in the final report: • why the research results are important to patients, specifically how they impact outcomes important to patients (pain relief, etc.) • benefits and risks of the treatment or intervention • who was included in the study – women, children, ethnic, groups, etc. • limitations of the study. What works in a controlled clinical setting may not be effective in the real world where patients have intervening health and environmental factors that may impact their treatment outcomes. What are the gray areas, and what further research is needed?
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Since PCORI will post on its website the lay abstract, medical abstract, standalone results table and ancillary information on the PCORI site, it is important that the PCORI website database of “Research and Results” be updated so that patients and family caregivers can easily access the information. Currently the “Research and Results” function on the PCORI website is neither useful nor accessible to consumers. For example, when a patient wants to search for a study on a specific condition or disease, the “search our awards” prompt does not make sense and should be changed to “search our studies or research.” Sample searches should be provided to guide consumers, using consumer friendly language and search categories that are relevant to them. Currently, the emphasis on the "Research and Results" page is on what PCORI has funded and funding announcements, and seems designed for researchers, not patients. There should be a specific search page for patients and family caregivers, explaining why the research is important and how they can use the results to make health care decisions. For research results not yet available, it would be helpful to list the approximate date of when research findings will be publicly available for each study.
Public posting of full final reports on PCORI’s website
See above comments on how the final report and research results are displayed on the PCORI website. Also, clarify the process and timeline for the public posting of the full final report – weighing the need to delay to coordinate for journal publication against making information publicly accessible in a timely matter. Explain why there needs to be a delay in posting the report, and how posting the report impacts potential journal publication.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
no comments
Timeline of the proposed peer review and public release process
We encourage dissemination of the research findings broadly. The majority of consumers will not go to the PCORI website, or to medical journals for information, so there is a need for a broad based dissemination strategy, focused on going to where patients get their information. To reach under represented communities disseminate where diverse patients are able to access information
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please add any comment about your rating
no comment
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
We believe that involving consumers and patients in every step of the research, review and dissemination process is what is required to implement the vision of PCORI – conducting and dissemination of research that is centered on outcomes that matter to patients. The proposal needs to clarify that patients are part of the peer review process and that public release of findings will be done in a way that is accessible to consumers.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
No comment
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
We agree. It is important to undergo peer review before the findings are posted to ensure that research and evidence are credible and trustworthy, thus making it easier to disseminate and implement new findings. However, the peer review process should include the patient and consumer voice.
Submitted
11/7/14 12:19
Name of person or organization submitting comments:
College of American Pathologists
Please identify which stakeholder group you represent:
Clinician/Clinician Society
File Upload
Submitted
11/7/14 13:06
Name of person or organization submitting comments:
Elizabeth McGlynn on behalf of PORTAL CDRN
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
PORTAL response: Registration of clinical trials, clinical registries, or evidence synthesis studies at the corresponding registration site may be appropriate. It is not clear whether the CDRNs qualify as “registries” for purposes of registration at the AHRQ RoPR site. Does this apply only to PPRNs? If not, more information is necessary to explain the rationale for CDRN registration. For example, does each individual disease cohort need to register?
Production of a draft final report, scientific abstract, and standalone results table
PORTAL response: 1. Clinical trials and registry-based observational comparative effectiveness research studies often produce multiple scientific papers, and it can be difficult to identify the primary analysis and paper. PCORI should provide an operational definition for the “primary analysis” or paper. Extension of this requirement to all ancillary analyses and papers is burdensome to the Awardee Institution and researchers. 2. The primary analysis of a study may take place before the formal end of the award. Thus, it is not synonymous with a final report. The language in the document should reflect this distinction. 3. PCORI should clarify whether a journal-ready manuscript that contains and highlights the required elements for the PCORI draft final report would be acceptable. Allowing this would substantially reduce the burden on the investigators, who otherwise would be required to submit a report and a manuscript that are largely duplicative. This would require PCORI to accept minor modifications to its reporting format, however. 4. PCORI should give consideration to using a structured abstract format for the required 500-word abstract. 5. A single standalone
PCORI peer review process of the draft final report
PORTAL response: 1. As stated, the Principal Investigator of the PCORI grant is responsible for all communications with the PCORI project officer managing the peer review. The Principal Investigator should have the option of delegating the responsibility for communication with PCORI to the lead investigator on the specific study. 2. Will PCORI develop a structured report form for its peer reviewers? If so, this should be publicly available to assist investigators in preparing the PCORI report. 3. While allowing simultaneous review of a report by PCORI and a journal is advisable, this process may lead to important discrepancies between the information in a PCORI final report and in the published paper. For example, if different questions are raised about inclusion criteria or covariates in a multivariate model, the number of enrollees in the study or the strength of association (e.g. magnitude and confidence intervals for relative risk) may differ. Resolution of any discrepancies between these two reports may require substantial effort by the investigators, and this process is made more difficult by the lack of detail allowed in the proposed PCORI results summary. One approach to resolving this would be to substitute the published paper (or a link or reference to it) for the final report as soon as possible after it is published.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
PORTAL response: No concerns.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
PORTAL response: No concerns.
Public posting of full final reports on PCORI’s website
PORTAL response: 1. The ability to delay posting of full final reports beyond 12 months is critical, since multiple journal submissions may be required. 2. As noted previously, by the time of journal publication, there may be multiple discrepancies between the published paper and the final report accepted by PCORI. PCORI should propose an approach to reconcile these discrepancies, such as substitution of the published paper for its prior final report in its publicly available site.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
PORTAL response: No concerns.
Timeline of the proposed peer review and public release process
PORTAL response: The current timeline for PCORI peer review considers the process in isolation from the simultaneous timeline for publication in a scientific journal. It would be instructive (either in the document itself or as part of the process of assessing the PCORI peer review policy) to overlay the PCORI timeline with several plausible scenarios for scientific publication, in specific: a. How would the timelines overlap for a paper submitted to and ultimately accepted by a high-impact journal such as NEJM or JAMA? b. How would the timelines overlap for a paper rejected by NEJM or JAMA but then submitted to and ultimately accepted by a prominent specialty journal? c. How would the timelines overlap for a paper requiring several submissions before publication? The current timeline also considers the process in isolation from the simultaneous grant timeline. What part of the timeline is included under grant funded activities? What parts are not included under grant funded activities?
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
PORTAL response: No additional comments
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Unclear
Please add any comment about your rating
PORTAL response: “Somewhat unclear” because, as noted previously, important concerns about the peer review and public release process require further clarification by PCORI.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
PORTAL response: 1. Discrepancies between the required format of the PCORI final report and that of scientific journals will require substantial duplication of effort. This reduces the feasibility of the proposed approach. 2. Differences between the final report accepted by PCORI and the paper ultimately published in the scientific literature may require substantial effort on the part of investigators to reconcile. This reduces the feasibility of the proposed approach.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
PORTAL response: No additional comments
Please provide any additional comments that you have not already provided about any aspect of this proposal.
PORTAL response: The PCORI process for peer review should be reconciled as much as possible with the established processes of federal agencies such as FDA and CDC. If this has already occurred, it should be mentioned in the final guidance for peer review.
Submitted
11/7/14 13:49
Name of person or organization submitting comments:
Biotechnology Industry Organization (BIO)
Please identify which stakeholder group you represent:
Life Sciences Industry
File Upload
Submitted
11/7/14 13:56
Name of person or organization submitting comments:
Larry Peiperl MD, Chief Editor, PLOS Medicine
Please identify which stakeholder group you represent:
Journal/Publisher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
We applaud PCORI for taking the bold step, ahead of many journals, of requiring prospective registration of protocols for observational studies. We believe this requirement will enhance the integrity of reporting and clearly differentiate pre-specified from post-hoc analyses.
Production of a draft final report, scientific abstract, and standalone results table
We feel that the brief abstract and summary results table will be of limited usefulness without public availability of the the full study data and results. Rather than delaying data availability for many months until a “definitive” peer reviewed publication appears in a journal, only to be revised or refuted when further experts are able to re-analyze the study data, PCORI could work with journals that permit data sharing at an appropriate point prior to appearance of a “final” publication.
Timeline of the proposed peer review and public release process
Rather than assuming that journals will refuse to publish work that has been made available in preliminary format, PCORI could work with journals that allow pre-acceptance posting of preliminary versions. This is an established process and there are a number of places where prepublication versions of papers are now available for public scrutiny; these include clinicaltrials.gov and biorxiv.org. Doing so would allowing public comments (with appropriate disclosure of competing interests) to begin without delay, and potentially to be integrated into the journal peer review process.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
The Draft Proposal does well by requiring prospective pubic registration of studies and a minimum standard of review prior to full availability. We appreciate that PCORI has had to balance the time needed for adequate vetting of study results with the need to make properly vetted results widely available as rapidly as possible for the benefit of patient care. We believe, however, that the Draft Proposal leans too far in the direction of deference to existing processes by some, but by no means all, respected medical journals.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Neither Clear nor Unclear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Disagree
Please add any comment about your rating:
Rather than accepting that journals will take 9 months or more to peer review and publish a “definitive” version of a study following 3 or more months of initial review by PCORI, the Institute could collaborate with journals that are willing and able to coordinate journal and PCORI reviews as concurrent processes or even a single combined process.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
PCORI-sponsored research should receive the widest possible dissemination and should be available for re-use and adaptation to benefit patients, healthcare providers, and communities. We believe this would be best accomplished by establishing the expectation that researchers publish their studies under Creative Commons attribution licenses (https://creativecommons.org/licenses/), and by ensuring funding to cover reasonable article publication charges by journals that provide open access publication. Otherwise, important work intended to benefit the public will remain behind subscription paywalls that limit access outside of large institutions, and locked into copyright restrictions that discourage sharing and adaptation for broad public use.
File Upload
Submitted
11/7/14 14:34
Name of person or organization submitting comments:
Thomas Ferkol, M.D.
Please identify which stakeholder group you represent:
Clinician/Clinician Society
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//PCORIfinal11-14.pdf
Submitted
11/7/14 15:18
Name of person or organization submitting comments:
American College of Physicians
Please identify which stakeholder group you represent:
Clinician/Clinician Society
File Upload
Submitted
11/7/14 16:12
Name of person or organization submitting comments:
Pharmaceutical Research and Manufacturers of America
Please identify which stakeholder group you represent:
Life Sciences Industry
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//PhRMA Comments to PCORI.pdf
Submitted
11/7/14 16:23
Name of person or organization submitting comments:
GlaxoSmithKline
Please identify which stakeholder group you represent:
Life Sciences Industry
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
In addition to requiring that awardees register with the applicable site (ClinicalTrials.gov, RoPR, PROSPERO), postings on the PCORI website should be easy to find and include a link to relevant external postings. This will ensure that individuals can easily find external postings related to the study. There is a time requirement that registration on ClinicalTrials.gov take place prior to the enrollment of the first patient. This should be the minimum requirement as the ClinicalTrials.gov posting, besides addressing issues related to transparency and ensuring that the study design and objectives are stated up front, may also be helpful for subjects who may be interested in participating in such studies. PCORI may want to consider revising the time requirement to also align with project milestones. The relevant external posting site should be agreed to up front and be part of the contract between PCORI and its awardees, and should be a milestone for PCORI awards.
Production of a draft final report, scientific abstract, and standalone results table
The format of the draft report should also contain a clear re-statement of the study objectives or a link to the ClinicalTrials.gov posting. This should be provided up front. The discussion section should also contain information about the study findings within the broader context of the prior literature. PCORI should be funding research that advances the science for a given research topic. So it is important to review the results and contrast those results with other relevant studies. The limitations of the study appear within both the discussion as well as conclusions section. Including limitations in both sections may lead to some confusion and has the potential to dilute such discussion. Limitations need to be clearly stated and understood. PCORI may want to consider having limitations as a stand-alone section, or provide clarity on whether limitations should be included in either the discussion or conclusions section. While the discussion section also includes “considerations specific to certain sub-populations,” PCORI may want to consider “implications for relevant stakeholders” which is not adequately captured with the current format. Inclusion of such a section may help to ensure that PCORI funded research adequately meets the need of patients and relevant stakeholders.
PCORI peer review process of the draft final report
PCORI has an important role in helping to advance methodological standards for patient-centered outcomes research. As the methodology standards will continue to evolve over time, it is important that a particular study be evaluated based on the methodology standards that were in place at the time the study was awarded, and not to judge the study and its design by standards that did not exist at the time. Doing so would be problematic for both PCORI and the awardees, as the study may be judged as not sufficiently meeting methodological standards. For the peer review process, while PCORI will be following a process similar to that used by medical journals, it is important to note that there may be opportunities for PCORI to improve upon some of the challenges and issues with peer review by medical journals. • There is often a great deal of variability in the type of comments that an author receives from peer reviewers. They range from highly detailed and specific criticism on the study design and results that may be motivated by personal interests and agendas (despite the reviewer’s self-determination of whether or not they have competing/conflicting interests) to insufficient and haphazard review (i.e. looks good, no comments). Effective peer review should ensure that the study design and objectives are clearly described and appropriate, that results and interpretation of the results are sufficiently detailed, consistent with the study design and free of bias, that the conclusion is aligned with the study design and results and avoids unsubstantiated interpretation and speculation. In other words, it should be clear what the study is – and what it is not. • PCORI should consider the appropriateness of the peer reviewers based on their qualifications and expertise as so-called content experts. Journals often invite scientists to review manuscripts that are outside of their area of expertise. It is up to the individual reviewer to determine whether or not they have sufficient expertise to critique a given study or topic. • There can be better “quality controls” to ensure that manuscripts are being reviewed by appropriate reviewers and that they have done an adequate job. For instance, many reviewers may not review relevant citations to ensure that such citations are used appropriately and with the proper context, and that the study has considered all such sources rather than “cherry-picked” citations. In practice this is sometimes problematic because reviewers may not take the time to review the relevant literature or may not have access to all of the relevant articles to do so, thus relying on their own knowledge of the literature. Although the assigned “methodologist” and program officer will be responsible for ensuring that the study design and methods adequately conform to PCORI’s methodology standards, it is important that the “content reviewers” also be familiar with PCORI’s values and methodology standards. One recommendation would be to ensure that all peer reviewers go through training on the PCORI values and methodology standards. Although PCORI envisions that patients and patient-representatives be involved throughout the entire research process, patients and patient representatives are noticeably absent from the peer review process. Although the proposed process does incorporate lay summaries of the research, further consideration should be made to how patients (and other relevant stakeholders) can be incorporated into the peer review process. While there are likely some challenges with regard to the technical nature of scientific manuscripts, this may underestimate the ability of patients or others to review such information. PCORI, embracing the principles of transparency, may wish to consider employing open-review where the reviewer’s comments (and the identity of the reviews) are publically disclosed. This provides a way to ensure accountability and the highest standards for the review – although it could have a negative impact as some reviewers may not want to participate in an open-review. If an open (as opposed to blinded) review takes place then the reviewers could also potentially participate in the review conference. PCORI may also want to consider inviting relevant patients and other stakeholders to participate in such conferences which may help to facilitate dissemination and adoption of the study results. It should be noted that most of the timelines are in calendar days, while the “revision to the draft final report” is in “working days.”
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
PCORI proposes the 500-word (lay) abstract be readable at the 8th grade level, however, a 6th grade reading level may be more suitable according to health literacy principles. PCORI should also consider numeracy principles in the creation of lay summaries. PCORI should consider expanding the length of the lay-abstract. There is often an art to writing abstracts, especially for large and complex studies, and it sometimes is difficult to squeeze everything into an abstract. When creating lay summaries, PCORI should consider providing additional space for additional information that may help to facilitate clarity and understanding (e.g. a results table). We suggest increasing the word limit to at least 750 words. PCORI should seek opportunities to engage patients and other relevant stakeholders in the review of the lay abstract. This is critical to help ensure that the study and the results are meaningful to patients. It may be beneficial to also invite patients and other stakeholders to draft a “stakeholder perspective” on the study which may help to further demonstrate how PCORI is funding research that meets patients’ needs and interests. PCORI should look at opportunities for further translation of the full study report appropriate for patients and consumers although such translations may be outside of the scope of the current reporting requirements.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
There may not be a mechanism within ClinicalTrials.gov to provide a link (we have not found one) to the PCORI website. Thus PCORI should be sure this is possible before finalizing this proposal. Is this mechanism mainly based on avoiding concerns about prior publication or is it a desire to have the trials posted and easily connected with the PCORI website? Understanding this issue may facilitate refining the proposal.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
PCORI may wish to specify that notification to PCORI occur with a specified number of days after submission to a journal. It may be beneficial to indicate that notice to PCORI occur after submission, as opposed to simultaneous with or before, so as not to give the impression that PCORI may have an opportunity to input on the manuscript. The awardees should also notify PCORI if a manuscript was not accepted for publication, the reasons why, and if they plan to submit elsewhere. Parallel review between the PCORI and peer review journal is somewhat problematic, as it could potentially result in two publications which may diverge in some aspects. PCORI should look for ways in which the two peer review processes can complement one another.
Timeline of the proposed peer review and public release process
Who will hold the PCORI peer reviewers to the proposed time lines? Peer review at journals can drag on for months. PCORI’s proposed frames are optimistic. Ninety days from end of research to draft report, and 12 months to publish are indeed stretch targets. The 90 day period from PCORI’s acceptance of the final report to the posting of an approved “consumer friendly abstract” would seem to be enough time to allow for PCORI to draft, and the Awardee to approve, even with the assumption that there will be several iterations back and forth.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
PCORI should also consider peer review of study proposals and/or study designs. We would argue that this is even more important than the results review. Researchers may get contradicting advice from the PCORI peer reviewers. How will this be handled? Who is the arbiter of review comments, i.e., who is the PCORI equivalent of the editor and makes the final stamp of approval? Would the final draft report be the draft manuscript that gets submitted to the journal? The wide variation in external peer review (timelines and processes) makes ‘keeping it simple’ complex. Further clarity and transparency for minimal standard expectations of PCORI awardees seems relevant yet distinct from broader PCORI goals for systematic harmonization of contracts with external processes such as ClinicalTrials.gov (pre-start of the work) and archive of the completed work (e.g. in publications and open access systems). Comments to improve clarity of the proposal: • Page 2, title and first figure. Add (D/I) after dissemination and implementation so that the D/I in the figure has a precedent use. • Page 3. Should be “PCORI’s authorizing law mandates…” instead of “law’s…” • Page 3. First and second sentences are lengthy and run-on. • Page 4. “…top-flight journals…” could perhaps be strengthened to “target manuscript submissions to journals with readerships aligned with the study design, execution, and analyses.”
Submitted
11/7/14 16:32
Name of person or organization submitting comments:
Jorge H. Ramírez
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
1. Over half of US phase 3 clinical trials were retrospectively registered (2002-2014) http://www.bmj.com/content/347/bmj.f6449/rr/690626
PCORI peer review process of the draft final report
I only knew today about the existence of this draft and its deadline
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Poorly
Please add any comment about your rating
1. Over half of US phase 3 clinical trials are unpublished http://chaoticpharmacology.com/2014/08/13/response-to-j-castellani-phrma-an-ounce-of-data-i-e-64740-data-values/ 2. How many trials are never registered? 4. Who will have access to the raw data? 5. Does anyone at pecori have access to this data?
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please add any comment about your rating
Arguments are very clear, but not cogent at all, this proposal is weak and the status quo will continue as usual.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Is feasible but not very useful (raw data have to be disclosed)
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Neither Agree nor Disagree
Please add any comment about your rating:
Over half clinical trials are never published anyway.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
No further comments.
Submitted
11/7/14 16:33
Name of person or organization submitting comments:
daniel e. korin, MD
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
No comments
Production of a draft final report, scientific abstract, and standalone results table
The report details the more traditional peer review processes and some of the challenges. What is missing a much more elaborated description of the mechanisms to include patients, caregivers, and other consumers organizations into the "total peer-review," including opportunities for those important stakeholders to contribute to a minority report.
PCORI peer review process of the draft final report
As above
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Needs to spell out how that version will be developed and, if there are costs involved, who will pay.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please add any comment about your rating
Patient-centeredness should be evident at every phase of the project, including effective communication to patients impacted by the problem(s), caregivers, etc.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
As above
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Neither Agree nor Disagree
Submitted
11/7/14 16:57
Name of person or organization submitting comments:
Doris Peter for Consumer Reports
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Timeline of the proposed peer review and public release process
See attached document
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Poorly
Please add any comment about your rating
This relates to the mandate to release the results in a timely manner. We urge PCORI not to hold the public hostage to the journal peer review process. See attached comments.
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
We support rigorous peer review, just not double peer review (PCORI, and then journal); and then holding back the publication of the PCORI-funded results based on the journal's publication timeline (embargo). The research should be allowed to be published through PCORI without precluding later publication in an academic journal.
File Upload
https://www.pcori.org/sites/default/files/webform/pcori-comment//PCORI_peer_review_comments.docx
Submitted
11/7/14 16:57
Name of person or organization submitting comments:
Sue Ellen Kline
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
It is possible that some of the research may qualify for multiple registration options, i.e., these categories are not mutually exclusive. For example, a patient registry could be designed to evaluate comparative effectiveness. Will the study be required to be listed on both clinicaltrials.gov and RoPR?
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This is extremely important and, I believe, is a key aspect of PCORI's function as a liasion between academically trained researchers and the lay public. Unfortunately, the funding applications that I reviewed in Q4 2013 did not properly address this issue, i.e., most/almost all applications copied the exact same medical professionals section into the patients and general public section. I am glad to see PCORI taking the lead on ensuring this is done properly.
Public posting of full final reports on PCORI’s website
In addition to the website I would like to see plans to disseminate more widely. For example, many poor, rural and/or elderly do not have ready access to a computer, and even if they do, know how to navigate the internet. I recommend a discussion about other lay dissemination strategies beyond the internet and the PCORI website.
Timeline of the proposed peer review and public release process
Just to clarify, the 90 day release of findings required by the legislation will be satisfied by the Awardee's submission of the "draft final report" to PCORI? Then the Awardee will have up to 12 months to submit and publish the peer-reviewed manuscript?
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please add any comment about your rating
Excellent job balancing the time constraints in the legislation with the realities of publishing in the peer-reviewed medical literature. Kudos on finding a solution to that challenge!
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
Please add any comment about your rating
Very few questions and thank you for the condensed communication format (i.e., only nine pages to review was very helpful).
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Only thought is if what gets publicly disclosed within the 90 day time frame ends up differing with eventual peer-reviewed publication particularly where the Results table is concerned.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Per above, from my perspective peer-review is critical before posting results exactly because those comments benefit from a rigorous third party (i.e., expert reviewers that were not associated with the research and their comments will provide critical unbiased evaluation of the research, results and conclusions). Also, once results and conclusions are "published" e.g., via the PCORI website should a peer-review determine inaccurate it will be more difficult to disclose the correct data.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Thank you very much for the opportunity to participate in the excellent work that PCORI is undertaking to improve by including patients/caregivers in US healthcare decision-making.
Submitted
11/7/14 16:59
Name of person or organization submitting comments:
Wilson D. Pace, MD, FAAFP
Please identify which stakeholder group you represent:
Clinician/Clinician Society
File Upload
Submitted
11/7/14 17:04
Name of person or organization submitting comments:
Andrew Sperling
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
File Upload
Submitted
11/7/14 18:26
Name of person or organization submitting comments:
National Health Council
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
The National Health Council (NHC) is pleased to submit comments on PCORI’s Proposal for Peer Review of Primary Research and Public Release of Research Findings. The NHC is the only organization that brings together all segments of the health community to provide a united voice for the more than 133 million people with chronic diseases and disabilities as well as their family caregivers. The NHC is committed to promoting a health care system that strives to engage patients and their caregivers in making decisions about their health. Integral to this system is the ability for patients and caregivers to access and use information as it becomes available to inform their decision-making. The NHC appreciates the opportunity to comment on PCORI’s proposed process for making research findings publicly available. In particular, the NHC commends PCORI for the inclusion of a step to generate separate abstracts for medical professionals and patients/caregivers. We believe this is an important component to ensuring that research findings are accessible and “comprehensible” to patients. A major gap in PCORI’s outlined process, however, is in how PCORI intends to convey the usability of research findings. Translating technical information derived from research into usable, actionable information for a wide range of stakeholders, regardless of their scientific background, is rooted in PCORI’s authorizing language and has been a key focus for PCORI since its inception. According to the authorizing legislation, PCORI is charged with producing high-quality and usable research. To that end, the NHC recommends PCORI build into this process a mechanism for evaluating the usability of research findings. We believe this is a critical and necessary piece of the translation process that will promote appropriate use of these findings. In developing a process to evaluate usability, PCORI should leverage the work of the PCORI Evaluation Group (PEG), which has already begun the work to develop usability metrics. While the PEG has generated measures that assess uptake of findings by stakeholders as a way to gauge how useful the research was, the determination of usability cannot be limited solely to after the release of findings. There should be a process and similar metrics to assess usability of findings prior to results being released to provide support to patients and caregivers in understanding how research findings may be applied to their circumstances. We also strongly recommend that investigators be required to self-evaluate the usability of their results against the aforementioned metrics and include their assessments in the draft report and medical professional summaries that they produce. These evaluations can be incorporated through a brief summary and/or a simple grading scale which translates into various levels and dimensions of usability. We recommend the implementation of a similar process and requirement for the PCORI-generated patient-oriented abstract, which can be led by PCORI, the investigator, or the contractor. Finally, and perhaps most notably, the NHC is concerned with the clear absence of patient involvement in this proposed process. Patients are best equipped to discern whether research findings and other outputs, such as summaries and abstracts, are comprehensible and/or usable. Furthermore, this omission would seem to contradict PCORI’s desire to involve patients at every stage of the research process, including dissemination of research findings. There are a number of different roles that patients could play to provide input on whether the research findings are comprehensible to the average patient and whether the findings are actually usable in the context of making decisions about care. Therefore, we strongly recommend PCORI identify ways to meaningfully incorporate patients into this proposed process to ensure that findings are genuinely evaluated through the lens of a patient.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please add any comment about your rating
The NHC acknowledges and appreciates PCORI’s thorough review of the ACA legislation, as detailed in the proposal. PCORI’s proposed process meets many of the requirements set forth in the authorizing legislation; however, we believe that critical components are missing. Most notably, the process lacks a step to evaluate for the usability of research findings. As the legislation states, in releasing research findings PCORI must “convey the findings of research in a manner that is comprehensible and useful to patients and providers in making health care decisions.” We believe the efforts outlined in the proposal do not sufficiently assist patients and other stakeholders with making determinations of usefulness, which is distinct from comprehension. For example, a summary written appropriately for a low health literacy audience is not the same as being written so that a patient can make connections between how the findings might be applied in the context of his or her clinical decisions. Thus, the NHC strongly recommends that PCORI incorporate into its process a step to evaluate the usability of research findings. By articulating how this research may be used in decision-making, PCORI will be ensuring that users will be able to quickly assess how “useful”
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
Please add any comment about your rating:
We appreciate PCORI’s efforts to adhere closely to the legislative language of the Affordable Care Act, which outlines numerous provisions for sharing study results. We understand that implementing and executing all of the requirements, which are intended to ensure that study findings are both of high quality and accessible to a broad audience, can lead to a lengthy, multi-step process. Despite these challenges, the NHC encourages PCORI to continue to refine its process and identify areas where efficiencies could be gained. For example, building in roles for patient advisors and patient reviewers of the research outputs will allow for relatively quick, simple, and clear evaluations of whether the findings will be comprehensible and usable to the broader patient population. Given the rapidly changing environment and continuous flow of information, the proposed average timeframe of 10 months for sharing research results with the public could potentially reduce the impact they could have in practice. Thus, we believe that expediting this process to improve timeliness will help to maximize relevance and applicability of findings to patient and stakeholder decision-making.
Submitted
11/7/14 20:07
Name of person or organization submitting comments:
Community Campus Partnerships for Health
Please identify which stakeholder group you represent:
Patient/Caregiver Advocacy Organization
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Very Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
File Upload
Submitted
11/7/14 21:09
Name of person or organization submitting comments:
American Society of Health-System Pharmacists
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Agree
File Upload
Submitted
11/7/14 21:28
Name of person or organization submitting comments:
Deans of Nursing Schools Coalition: Eileen Sullivan-Marx, NYU
Please identify which stakeholder group you represent:
Researcher
PCORI peer review process of the draft final report
Thank you for this opportunity to comment. The science underlying nursing practice is consistent with the mission of PCORI: enabling people to make informed healthcare decisions, and improve healthcare delivery and outcomes. Clinical research generated by nurses clearly demonstrates that nursing interventions improve patient outcomes by addressing strategies and interventions that engage patients and families in their everyday health decisions. The following studies highlight work conducted by nurses over the past 30 years that actively involved patients and their families in the design and conduction of the research. For example, nursing research has shown that heart failure patients who can assess and make decisions about their own symptoms in terms of managing their care reduces emergency visits and hospitalization stay. Nurse scientists were responsible for designing the Braden Scale, a nationally used instrument that predicts a patient’s risk for pressure sores. A team led by a nurse scientist discovered that Coping Skills Training (CST) improves diabetic teens’ abilities to develop positive health behaviors that improve their control of diabetes. Nurse scientists led the way in demonstrating that African American men at risk for hypertension can lower their blood pressure and prevent cardiac damage – but it requires developing a trusting relationship with their health care team. Scientists with a nursing background also tested and identified strategies that worked to enhance safe sex practices to inner-city black middle school students. The Nurse Family Partnership clearly demonstrates through two decades of research that nurse - mother/family engagement reduces not just stress in daily living but also reductions in the number of pregnancies and a decrease in mortality rate among mothers and children. And other nurse scientists were the first to discover that personal characteristics (e.g. gender) played a factor in the efficacy of pain control. Given the direct linkage between interventions and outcomes, nursing research has led to significant changes in the nation's health. By increasing the capacity of nursing scientists on review panels with leadership responsibility, such as chairing review panels, would propel the mandate of PCORI to improve the quality and relevance of evidence available to consumers. Respectfully submitted, Eileen Sullivan-Marx, Dean, NYU College of Nursing Bobbie Berkowitz, Dean, Columbia University School of Nursing Kathleen Rideout, Dean, University of Rochester, School of Nursing Margaret Grey, Dean, Yale University School of Nursing Mary Kerr, Dean, Case Western Reserve, Francis Payne Bolton School of Nursing Antonia Villarruel, Dean, University of Pennsylvania, School of Nursing Patricia Davidson, Dean, Johns Hopkins, School of Nursing Linda McCauley, Dean, Emory University, School of Nursing Linda Norman, Dean, Vanderbilt University School of Nursing Marion Broome, Dean, Duke University School of Nursing
Submitted
11/7/14 22:42
Name of person or organization submitting comments:
American Nursing Informatics Association (ANIA)
Please identify which stakeholder group you represent:
Clinician/Clinician Society
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The American Nursing Informatics Association (ANIA) supports PCORI’s plan to have the planned research registered at one of these three entities. This approach should help alleviate the issue of journal rejections due to prior publication of findings.
Production of a draft final report, scientific abstract, and standalone results table
ANIA agrees with PCORI’s proposal for each grantee to submit the draft final report with a scientific abstract and a results table that can disseminate understandable data on its own. This set of documents should provide a rich source of integrated information that fully describes the research and its outcomes.
PCORI peer review process of the draft final report
Peer review of the draft final report has ANIA’s support. We do encourage transparency or openness of the peer review process, including identification of the grantees and the reviewers. While publication of a list of all peer reviewers used by PCORI is important, such a list does not address the ethical and quality issues of closed peer review. Another concern is the emphasis in the peer-review plan on assessing the fit of the research methodology to the PCORI’s Methodology Standards. Review of the methodology for this fit should be done during the grant application process, not after the research is completed.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
ANIA strongly supports creating summaries and abstracts of PCORI -funded research for consumption by patients and other members of the general public. With its focus on patient outcomes, PCORI’s research will be of keen interest to everyone who has been or will be a healthcare consumer. ANIA encourages PCORI to use a 5th-grade reading level as the benchmark for these materials, since this is the reading level most recommended for healthcare materials.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
ANIA applauds the requirement to post research results within 90 days after the approval of the final report. Timely access to patient outcomes research is important to practitioners and consumers alike. Given the time delays inherent in changing practice, the sooner results are available for consideration, the better. ANIA encourages PCORI to include diverse healthcare-related sites in those considered appropriate. Nursing, healthcare-consumer, and allied-health sites need to be part of these other sites in order to disseminate results widely and quickly to all who can benefit. ANIA also encourages extensive use of diverse social media to notify many different audiences about the availability of PCORI-funded research results.
Public posting of full final reports on PCORI’s website
ANIA supports the plan for the timing of making the full final report available on the PCORI website.
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
ANIA agrees with this portion of the plan and recommends use of social media to inform diverse audiences of published articles. ANIA encourages PCORI to support grantees in looking beyond medical journals to the numerous quality journals in healthcare for publication opportunities.
Timeline of the proposed peer review and public release process
ANIA believes the timeline is appropriate and feasible.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Very Clear
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Submitted
11/7/14 23:39
Name of person or organization submitting comments:
Annals of Family Medicine
Please identify which stakeholder group you represent:
Journal/Publisher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
The Annals of Family Medicine supports the registration of clinical trials.
Production of a draft final report, scientific abstract, and standalone results table
Posting of research reports should be timed in concert with publication in professional journals. Good research is seldom “final” when the major effects report is posted or published. Timing should accommodate the iterative process reporting, review, revision, further analysis and multiple publications of study findings.
PCORI peer review process of the draft final report
Peer review – both internal and independent - is essential to scientific rigor, evidence quality, professional acceptance and public trust. Quality, qualified, expedited review will require resources.
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
Effective dissemination of research findings will require reporting in language accessible to patients, communities and other stakeholders. Dissemination should be timed in concert with presentation at professional meetings and publication in professional journals.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Dissemination should use diverse and new avenues of communication and be timed in concert with traditional, trusted methods, including presentation at professional meetings and publication in professional journals.
Public posting of full final reports on PCORI’s website
“Final reports” should be posted and disseminated widely, after peer review and publication have provided full opportunity for comment, critique and correction.
Timeline of the proposed peer review and public release process
The timeline is ambitious and potentially problematic. See our uploaded document. The timeline is unclear in places and may not provide adequate time for essential processes, including review, revision, editing and production. The final report must meet - at a minimum - the internationally accepted standards of peer-reviewed, indexed professional journals.
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Somewhat Well
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Clear
Please add any comment about your rating
The proposal is somewhat unclear on the timeline and definitions of steps in the reporting, review and dissemination processes.
Please comment on the proposals feasibility given any competing obligations that awardee institutions funded by PCORI, or their investigators, may face.
Investigators will face difficulties in following the prescribed PCORI process, managing the realities of the research process and the meeting expectations journal editors. Peer-reviewed journal publications are the professional capital of academic careers, research funding (federal and private), service on national expert panels and effective public advocacy. PCORI will need to work with investigators, institutions and journals to meet mutual needs and to reach shared goals.
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Strongly Agree
Please add any comment about your rating:
Peer-review precedes publication. If reports are posted for public information, prior peer review is even more essential, to help correct and clarify key findings and implications of the research.
Please provide any additional comments that you have not already provided about any aspect of this proposal.
Please refer to our uploaded document.
File Upload
Submitted
11/7/14 23:51
Name of person or organization submitting comments:
National Pharmaceutical Council
Please identify which stakeholder group you represent:
Life Sciences Industry
File Upload
Submitted
11/7/14 23:58
Name of person or organization submitting comments:
Leif Solberg and Juliana Tillema
Please identify which stakeholder group you represent:
Researcher
Registration at ClinicalTrials.gov, RoPR, or PROSPERO
This is perfectly reasonable and important
Production of a draft final report, scientific abstract, and standalone results table
There is no problem with having a draft final report - the problem is a potentially censoring review process and in public reporting, making it difficult to later publish in a prominent journal
PCORI peer review process of the draft final report
The proposal wording makes it sound like the review can produce required changes that are disliked by the reviewer. While we respect and support the methodology standards, like any standards they are open to a wide variety of interpretations. Since any objection can be presented as a methodological problem, this means that findings and recommendations disliked by the reviewer or PCORI could be forced to change. That would short circuit the scientific process and make it less likely that controversial findings could be presented or later appear in journal publications if they conflicted with the opinions of the PCORI reviewers
Providing a version of the results summary that will be useful and comprehensible for use by patients and the general public
This is a good idea, but it should not precede publication of the complete results in peer-reviewed journals, since frequently peer review at journals leads to additional analyses and significant changes in findings. In that case, the public version would appear to be incorrect and confusing to the public. This does not help the public.
Submission of results table to ClinicalTrials.gov and/or other sites as appropriate, with a link to the abstract and standalone results table on PCORI’s website
Public posting of results prior to journal submission and peer review threatens the ability to later publish those findings
Public posting of full final reports on PCORI’s website
Same issues as above
Awardee institution notification to PCORI of submission to journals and subsequent publication of papers resulting from funded projects
Notification and subsequent publication is no problem and is really an advantage to all concerned
Timeline of the proposed peer review and public release process
Very problematic, both because a good study should be submitting articles for publication prior to the final report and because of the prior publicity problems noted above
Please rate how well the proposal meets PCORI’s obligations under its authorizing legislation
Neither Well nor Poorly
Please rate how well the proposal provides a clear understanding of the peer review and public release process
Somewhat Unclear
Please add any comment about your rating
The peer review process doesn't say anything about the potential for censorship of controversial findings or provide any safeguards against that
The law requires that PCORI both conduct peer review of our funded primary research and make it available publicly within a specific timeframe. We propose to conduct peer review first, then post the results, so they can benefit from the review process. Do you agree or disagree with the sequencing of this process?
Somewhat Disagree
Please add any comment about your rating:
See above comments
What's Happening at PCORI?
The Patient-Centered Outcomes Research Institute sends weekly emails about opportunities to apply for funding, newly funded research studies and engagement projects, results of our funded research, webinars, and other new information posted on our site.
Image
