Reviewer FAQs
Click on the links below to learn more about PCORI's Merit Reviewer application process. top
General FAQs | Becoming a Reviewer | Training | Technology | Evaluation | Online Merit Review | In-Person Merit Review | Travel | Compensation
General FAQs
What are the differences between Scientific Reviewers, Patient Reviewers, and other Stakeholder Reviewers?
PCORI’s Scientific Reviewer community includes members of the health and healthcare research communities, with technical or specialized research backgrounds.
PCORI’s Stakeholder Reviewer community includes patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders. PCORI defines stakeholder reviewer communities as follows:
- Patients (Persons with current or past experience of illness or injury, family members or other unpaid caregivers of patients, or members of advocacy organizations that represent patients or caregivers)
- Clinicians (Providers of health care in a clinical setting, including physicians, nurses, physician assistants, rehabilitative professionals, pharmacists, mental healthcare providers, complementary and alternative healthcare providers, and professional societies serving clinicians)
- Researchers (Those who conduct clinical research, including investigators or funders of research and organizations or associations representing the research community)
- Purchasers (Those who purchase health benefits for employees and their dependents, including individual businesses as well as local, state, regional, and national business groups, coalitions that represent businesses, and health coalitions)
- Payers (Those who function as financial intermediaries in the health system, including private insurers and public insurers, and organizations representing insurers, such as America’s Health Insurance Plans)
- Industry (Companies that design, invest in, or manufacture diagnostics, devices, pharmaceuticals, electronic records systems, and mobile apps, and organizations representing the life sciences industry, such as the Advanced Medical Technologies Association)
- Hospitals and Health Systems (Organizations where care is delivered, including public and private hospitals and health systems, urgent care centers, retail health clinics, and community health centers, and organizations representing these facilities)
- Policy Makers (Those who help craft public policy at any level of government, including federal, state, and local government officials; federal, state, and local units of government; and organizations that represent policy makers)
- Training Institutions (Those that deliver health professional education include public and private universities and colleges, individuals affiliated with the delivery or administration of health professional education, and trade or professional associations representing these institutions, organizations, and individuals)
PCORI assigns Scientific Reviewers, as well as Patient and Stakeholder Reviewers, to every application for online review. PCORI recognizes that some Reviewers have multiple roles, including as patients or caregivers, and asks that Reviewers distinguish the Reviewer type with which they primarily identify.
Does PCORI have a Standing Panelist designation for reviewers?
Our changing needs and our more targeted funding strategy have led us to phase out the Standing Panelist designation. We will continue to collaborate with our excellent returning reviewers, including those who have served as Standing Panelists.
Can I serve as a Reviewer if I also plan to submit an application to a PFA?
Yes. However, PCORI does not permit applicants (Principal Investigator and key personnel) to serve as Reviewers for the same PFA that their application is currently being reviewed.
I have the opportunity to serve on a PCORI Advisory Panel. Am I still eligible to serve as a PCORI Reviewer?
Yes. Members of Advisory Panels can also serve as PCORI Reviewers.
Will applicants be able to see my reviews directly?
Yes, applicants will be provided with the written critiques for their application, but the anonymity of the Reviewers will be preserved.
Will my name be published as a Reviewer?
PCORI gives Merit Reviewers the opportunity to be included in PCORI’s publicly available Annual Report. Please note that this is intended to acknowledge and express our appreciation for the services and contributions of PCORI Merit Reviewers. PCORI will not identify which panels reviewers served on.
Becoming a Reviewer
How does PCORI select Reviewers for each cycle?
PCORI assigns applications to Reviewers based on availability, experience, and corresponding fit with the applications in the panel. Individuals who have been invited to serve as a PCORI Reviewer will be asked to indicate their availability to serve before a new cycle begins. In this correspondence, a Reviewer will indicate whether he or she is able to complete all assignments and attend the in-person Merit Review meeting. After participant availability is determined, PCORI assesses its need for specific Reviewer types based on the quantity and topic areas of applications received for that cycle. A subset of available Reviewers are then invited to self-rate their expertise and indicate any conflicts of interest (COIs) on the applications in each panel for a particular PFA.
Can non-US residents become PCORI Reviewers?
PCORI will primarily recruit merit reviewers who reside in the United States and its territories to be good stewards by effectively managing the costs of our Merit Review process. If you live in other countries in North America (Canada, Mexico) you are welcome to apply. We may recruit reviewers to serve during a review cycle if they have expertise that is not represented by our current U.S.-based reviewers. If you live outside of North America, we cannot accept your application to review for PCORI. For more information, visit our Become a Merit Reviewer page.
When will I be notified of my application status as a PCORI Reviewer?
PCORI vets Reviewers on an ongoing basis and will notify you of your status within approximately one week of receiving your application. (Check your spam/junk folder, just in case.) If you have any questions, contact us at [email protected].
If I was previously accepted to the PCORI Reviewer pool, do I need to reapply to participate in future cycles?
No. As a member of our reviewer pool, a Merit Review Officer (MRO) may contact you to assess your availability to serve on an upcoming cycle if your experience and expertise align with the applications we receive.
If I am no longer able to serve as a Reviewer for a particular cycle, what is the process for alerting PCORI?
Reviewers should send an email to [email protected] and email their MRO if they are currently working with one.
What if I indicated that I was available but was not assigned applications for Merit Review?
There is always a possibility that you may not be assigned to review applications. PCORI’s aim is to match the content of the application to the expertise and experience of the Reviewer, while also developing a panel representative of the patient and stakeholder communities. In the event that you are not assigned an application to review for this cycle, PCORI will keep you in consideration for future review cycles.
How will I know if I have not been selected as a Reviewer for this current cycle?
If you were selected to indicate COI and expertise on applications but not selected, you will receive an email notification from an MRO informing you of your reviewer status for the current review cycle.
If I am not accepted to be a PCORI Reviewer, how else can I get involved with PCORI?
We appreciate your interest in PCORI and have several other opportunities to become involved with PCORI. Please visit PCORI’s website or send an email to [email protected] to learn more.
Training
Is previous training required to be a PCORI Reviewer?
No, previous training is not required to be a PCORI Reviewer. However, once Reviewers are assigned to review within a cycle, they are required to complete the PCORI Merit Reviewer training. These training materials are updated each cycle to reflect changes in the PFA and merit review process.
When will I receive training?
Once application assignments have been made, the assigned Reviewers will be provided with training materials.
Where can I find Reviewer training materials and sample reviews?
If you are a Reviewer for the current review cycle, you will receive a direct link to the training from your MRO.
If I have already taken PCORI Reviewer training, do I need to take it again?
Yes. PCORI is continually improving its Merit Review process based on reviewer feedback. As policies and procedures change from cycle to cycle, all Reviewers are required to take the provided training each cycle.
Technology
Why must I have computer and internet access to serve as a PCORI Reviewer?
Reviewer tasks are completed online. Tasks include all training, correspondence, viewing applications in the PCORI Online system, and submitting critiques and scores in the system.
Do I have to own a computer?
No, PCORI does not require ownership of a computer to be a Reviewer, but PCORI Reviewers need to have access to a computer and internet on a regular basis.
Can I use a tablet computer to conduct the online review portion?
No, we encourage Reviewers to use a computer to conduct the online review portion when writing, entering and submitting them into the system along with your scores. Tablet computers are not compatible with the PCORI Online system.
What technology is required to serve as a PCORI Reviewer on a virtual panel?
In addition to having computer and internet access, Reviewers need access to headphones with a microphone when serving on a virtual panel. Use of a laptop camera or webcam is encouraged but not required.
Evaluation
What is PCORI’s scoring rubric?
The PCORI scoring system is based on a nine-point scale. Numerically high scores (for example, scores of 7 to 9) reflect applications that have major weaknesses, while numerically low scores (for example, scores of 1 to 3) are associated with applications that have minor weaknesses. Reviewers are asked to utilize the whole scoring range.
How do Reviewers develop the overall score and narrative?
Each criterion score should have its own consideration, but the overall score should reflect the merit of the application as a whole. The overall narrative should validate the score chosen by the Reviewer by stating the overall strengths and weaknesses that support the score. The narrative should provide feedback that will be informative and helpful to the applicants to strengthen their application and is an opportunity to expand upon what has already been highlighted. Contact your MRO with any other questions about scoring and comments.
Are Reviewers required to identify three strengths and three weaknesses for each criterion?
No. There is no required number of strengths and weaknesses for written critiques. The number of strengths and weaknesses for each criterion should correlate with the criterion score. For example, a score of 1 would correlate with an exceptionally strong application with essentially no weaknesses.
Do Patient and other Stakeholder Reviewers need to review the “Protection of Human Subjects” section?
No. Only Scientific Reviewers are required to evaluate the Protection of Human Subjects section.
What is the difference between patient engagement and patient centeredness?
Patient-centered research seeks to answer questions that matter to patients and/or their caregivers. Additionally, patient-centered research will measure outcomes that are important to patients and/or their caregivers. Patient and stakeholder engagement, on the other hand, refers to the integration of patients and other stakeholders as partners in the development of the research plan and in key elements of the proposed project.
How do I review a resubmission?
An applicant may resubmit an application that was not funded. An application is considered a resubmission if it has previously completed PCORI’s merit review process, including receipt of the summary statement and is resubmitted to the same funding announcement.
When resubmitting an application, applicants include a resubmission letter with their revised application.
The resubmission letter is an opportunity for applicants to provide an overview of how the application has been strengthened in its scientific merit and responsiveness to the current PFA. Simply responding to previous reviewers’ concerns is not sufficient; the application must be programmatically responsive and demonstrate methodological rigor and patient-centeredness. When evaluating a resubmission, reviewers should determine how well the resubmitted application and resubmission letter meet PFA requirements, address Merit Review criteria, and describe whether and how the application has improved overall.
Online Merit Review
Are the applications vetted before being assigned to Reviewers?
PCORI staff screen all applications for programmatic responsiveness to PFAs and administrative compliance. If a Reviewer has a concern with a particular application, he or she is encouraged to reach out to his or her MRO.
When will I be able to access my assigned applications?
Based on availability, PCORI will invite Reviewers to self-rate their expertise (using the application abstracts) and indicate any Conflict of Interest (COI) using information about key personnel. PCORI MROs will assign Reviewers based on an individual’s experience and corresponding fit with the applications in the panel. Approximately one week after Reviewers identify COI and expertise, applications will be assigned in PCORI Online and released to Reviewers.
What if I have a conflict of interest for an application that is assigned to me?
Please indicate your COI in PCORI Online and notify your MRO. Any applications you have a COI with will be assigned to a different Reviewer.
I accidentally marked an application as having a conflict, what should I do?
Notify your MRO as soon as possible via e-mail and at [email protected].
If I am in the process of reviewing an application and identify a potential issue regarding non-responsiveness, whom should I notify?
Contact your MRO immediately.
Do I need to submit all critiques at once in PCORI Online?
No. However, we do encourage Reviewers to submit their critiques early and as they are finished. This allows the MROs to review and ensure any questions or necessary edits are addressed before you complete all your reviews, and before the critique deadline.
Will I see my peers’ critiques of the same application?
Yes. Reviewer critiques will be available prior to the in-person/virtual panel discussion so that all Reviewers in the panel without a COI can be familiar with the application and critiques.
In-Person/Virtual Merit Review
What should I bring to Merit Review?
We request that all Reviewers bring laptops to the in-person panels, as computer access is required for scoring purposes. A limited number of laptops will be available if they are requested at the time of registration. Please note that iPads and other computer tablet or mobile devices will not suffice.
One of my assigned applications was selected for discussion at the in-person/virtual Merit Review. How do I know what the critique presentation order will be?
At the in-person/virtual meeting, Reviewers are asked to present their online critiques for all assigned applications scheduled for discussion. The presentation order generally is as follows: Scientific Reviewer, Patient Reviewer, Scientific Reviewer, Stakeholder Reviewer, and Scientific Reviewer, but may vary based on program funding announcement. Scientific Reviewers will be notified by their MRO, prior to the In-Person/Virtual Merit Review, of which applications they will be the first versus last presenter.
After the In-Person/Virtual Merit Review will I be able to revise my online critique comments/scores?
Once critiques have been submitted as final in preparation for the in-person/virtual Merit Review meeting, no other revisions can be made. Note that there is another opportunity to score and provide comments for applications discussed at the in-person Merit Review. Once the overall application scores are submitted at the in-person Merit Review, they cannot be changed.
What happens once the in-person Merit Review is completed?
The PCORI staff review the applications discussed at the in-person/virtual Merit Review and determine which applications should be recommended for funding to the Selection Committee based on the results of Merit Review, programmatic balance, and PCORI’s strategic priorities.
What do applicants receive after funding decisions have been announced?
After funding decisions are publicly announced, a summary statement for each application submitted to that cycle will become available to applicants in PCORI Online. For applications that are discussed at the in-person/virtual Merit Review, the summary statement includes a final overall average application score, a summary of the application discussion at the in-person/virtual Merit Review, and preliminary online reviewer critiques. For applications that are not discussed at the in-person/virtual Merit Review, the summary statement includes only the preliminary online reviewer critiques. Summary statements no longer include any preliminary review scores.
How does PCORI assess its Merit Review process and learn about opportunities for improvement?
After each review cycle, Reviewers are invited to complete a post-review survey. Additionally, a random subset of Reviewers are invited to participate in more in-depth group telephone interviews to discuss the Merit Review process. As a learning organization, PCORI continuously evaluates and revises processes, including application review protocols. Please send comments on the Merit Review process to [email protected].
Travel
I already registered to be a Reviewer. Do I still need to register to attend the in-person/virtual Merit Review?
Yes. All confirmed Reviewers will be sent information on how to register to attend the Merit Review. All Reviewers should register once they receive the link for the event.
I registered to attend the in-person/virtual Merit Review. How and when will I know that my hotel room is confirmed?
When you register to attend the in-person/virtual Merit Review, please indicate the number of nights that you will need to stay at the hotel. You will receive a hotel confirmation within one week of the event. If you do not receive this email confirmation within one week of the event, contact [email protected].
Do I need to contact someone else about my transportation?
All Reviewers should also contact the travel vendor listed on the Merit Review registration site via phone to reserve their flight or train, as needed.
How do I set up travel arrangements?
PCORI will assist Reviewers in all booking matters. Soon after Reviewers receive assigned applications, Reviewers will receive an email with instructions for how to register for in-person Merit Review and book travel arrangements. (Remember to check your spam/junk folder, just in case.)
Please do not make flight or hotel reservations on your own, as PCORI will be unable to reimburse you. PCORI will cover up to a 2 night stay depending on what days your meeting takes place.
Will PCORI cover travel to the in-person/virtual Merit Review?
Yes, PCORI will pay for all Reviewers’ travel and lodging for any meetings. PCORI will reimburse for local travel (car, taxi, and shuttle) after the Reviewer submits a reimbursement expense form. For further details, see the compensation guidelines below.
Do I need to attend the entirety of the Merit Review?
Yes. PCORI requires Reviewers to attend the entire Merit Review meeting to ensure fair and consistent scoring for all applications, and asks that Reviewers refrain from making any travel arrangements that will require them to leave early. Panels are expected to conclude at 3 p.m. EST on the second day of Merit Review. If the in-person Merit Review concludes earlier than anticipated, Reviewers will receive the full stipend for the second day.
Can I travel business class when I attend the in-person Merit Review?
No. PCORI will only cover coach tickets, unless there is a unique situation that requires a participant to travel under other conditions (e.g., medical condition).
I have a condition that requires special accommodation. How may I request this?
During the process of registering for Merit Review, PCORI will ask if you request special accommodations. Please inform us of any regarding your travel, lodging, or meal arrangements at that time.
Will PCORI pay to change my flight on the second day of Merit Review if my panel ends early?
In the event that the panel meetings ends earlier than anticipated, PCORI will not pay for change fees for travelers to depart on earlier flights. Travelers can make changes but will be responsible for paying any change fee associated with the change.
What if I am no longer able to attend the in-Person/Virtual Meeting?
If, due to an emergency, you need to cancel your attendance to the meeting, we ask that you provide notice no less than 48 hours prior to your arrival date. Additionally, if your flight reservation has already been made and you cancel your attendance to the meeting, you may use the flight again for travel, but will be responsible for following the carrier’s instructions and associated change fees.
To cancel your registration, lodging or travel, email [email protected] and alert your Merit Review Officer as soon as possible.
Compensation
How will I be compensated for serving as a PCORI Reviewer?
All PCORI Reviewers for all PFAs will receive $100 for each application reviewed as well as a $200 stipend for each day of the in-person/virtual meeting. All travel and lodging will be coordinated by PCORI for the in-person meeting. If meals and ground transportation are not provided, PCORI will provide reasonable reimbursement. All PCORI reviewers for all PFAs will receive an electronic voucher to cover the cost of breakfast and lunch for virtual Merit Review, depending on the duration of the meeting.
What is the compensation process for in-person Merit Reviews?
Stipends for reviewing online applications are transferred to your account via Bill.com before the in-person Merit Review. Stipends for attending the in-person Merit Review are transferred to your account after the in-person Merit Review. You will receive a reimbursement form on the final day of the in-person meeting. You can submit the completed expense report and scanned receipts to [email protected]. PCORI Finance staff will be on-site at Merit Review to help process reimbursements. Receipts are required for all reimbursements. For panelists serving on a virtual Merit Review panel, stipends for reviewing applications online and attending the virtual panel are transferred to your account after the virtual panel.
Are PCORI stipends taxable?
Yes. All PCORI stipends are taxable, no matter the amount received. Any Reviewers, Chairs, or Mentors who receive stipends totaling $600 or more during the calendar year will receive a 1099-MISC in accordance with IRS requirements. Reimbursement for travel expenses is not taxable and will not be included on the 1099-MISC.
If I am a federal employee, will PCORI pay my stipend?
Reviewers are responsible for using the guidelines for their federal agency/organization as the deciding factor to determine their stipend eligibility. PCORI will handle stipend and reimbursement requests based on information provided by the reviewer.
Updated: April 2, 2024
