Merit Review Criteria for Implementation PFA
Limited PCORI Funding Announcement: Implementation of PCORI-Funded Patient-Centered Outcomes Research Results
Criterion 1. Importance of research results
- Does the application sufficiently describe the original evidence gap that the PCORI-funded research addressed, as well as how the findings from this research address this gap?
- Are the research results proposed for implementation clearly described? Does the application describe the clinical relevance and strength of the findings from the PCORI-funded study proposed for implementation?
- Does the application sufficiently discuss how the PCORI-funded research results augment, strengthen, or complement the existing body of evidence? If the results contradict the existing evidence base, does the application address and justify the appropriate next steps, whether further research, dissemination, or changes in practice?
Criterion 2. Readiness for implementation
- Does the application describe how further uptake of these results, beginning with the project proposed, will lead to a change in practice and improved health care and health outcomes?
- Is the proposed implementation project an appropriate next step toward integrating and using the evidence in real-world settings? What will this next step accomplish (e.g., adapting an intervention or incorporating evidence to promote uptake in practice settings, reaching larger numbers of end-users, reaching broader audiences including diverse populations and settings)?
- For Phased Implementation Projects: Does the proposed implementation approach— once honed and demonstrated in Phase 1—have clear potential to succeed in the larger steps proposed in Phase 2? Is the overall scope and scale of the proposed project, in terms of anticipated scaleup and reach, significant and reflective of the large budget request?
- Have the proposed implementation sites been identified? If so, has the applicant demonstrated the readiness of the implementation sites, including the identification of site champions and key decision makers? If not, has the applicant provided a rationale for why this is not possible, along with acceptable assurances that all implementation sites can be activated within the initial project phase?
- For Phased Implementation Projects: Does the application make the case that selected sites are ready for implementation in line with the Phased Implementation Project timeline? For example, do Phase 1 sites have sufficient support and infrastructure to launch implementation by the end of the first year of the project? Will participating Phase 2 sites and systems be ready for scaleup within the timeframe proposed?
- Does the application sufficiently address the relevance of the PCORI-funded evidence proposed for implementation to the targeted end-users and implementation settings?
- Does the application sufficiently describe the target group of the proposed implementation activity? Does it describe the setting in which the implementation will take place? Are the PCORI-funded results generalizable to these targeted users and settings?
- Are these targeted end-users and settings representative of additional audiences who stand to benefit beyond this proposed implementation project?
Criterion 3. Technical merit of the proposed implementation project (project design and evaluation)
- Does the application provide a clear approach for implementing the described PCORI-funded research results?
- Are the chosen implementation strategies appropriate for this effort? Consider the extent to which they are tested, evidence-based, and consistent with principles and findings from implementation science.
- Do the proposed strategies consider factors that may help or hinder the integration of the PCORI-funded research results in the proposed project, including specific barriers to user implementation and how to mitigate them?
- Are the proposed project activities clearly described, and are these activities likely to result in successful uptake of the evidence and lead to meaningful changes in practice and improvements in health care and health outcomes?
- For Phased Implementation Projects: Does the application clearly describe specific activities that will occur in each phase of the project, and explain how these activities will accomplish the goals of each phase (i.e., demonstrating feasibility and impact of Phase 1 implementation activities and accomplishing significant scaleup in Phase 2)?
- If the applicant is proposing to adapt an effective intervention, is the adaptation well justified? Does the adapted intervention capture the core elements of the original tested intervention?
- Does the application propose appropriate measures and describe the plan for evaluating success in sufficient detail, including an appropriate balance of measurable outcomes that document both of the following?
- The successful execution of implementation activities (i.e., reach, site-level adoption, and fidelity)
- The impact of these activities on end-users in the immediate and longer term (i.e., changes in knowledge, behavior change, healthcare utilization, and health outcomes)
- For Phased Implementation Projects: Additionally, does the application clearly describe what data and outcomes (both implementation and intervention) will be included in the Interim Evaluation Report submitted after completion of Phase 1? Will these be sufficient to demonstrate feasibility and impact of Phase 1 implementation activities and to warrant advancement to Phase 2?
- Does the application use a D&I framework or model to inform the project design and evaluation outcomes? Alternatively, does the application adequately describe a logic pathway that shows how the proposed implementation approach is likely to lead to meaningful changes in knowledge, behavior, and practice?
- Are the proposed timeline and specific project milestones realistic?
- For Phased Implementation Projects: Does the application provide a detailed timeline describing which Phase 1 and Phase 2 activities will occur and when? Does this timeline allow for sufficient data collection and evaluation of implementation activities at Phase 1 and Phase 2 sites?
- Does the application address scalability, including a clear path for future efforts to bring these research results toward yet wider use across more systems, settings, or sites?
- For Phased Implementation Projects: Beyond the scaleup expected to occur during the proposed project, does the application describe what additional steps will be needed to accomplish further scaling, and how the proposed project will directly inform or support these efforts?
Criterion 4. Project personnel and environment
This criterion should assess the appropriateness (e.g., qualifications and experience) of the project personnel/team and the capacity of the environment to support the proposed project.
- How well qualified is the project team (e.g., PIs, collaborators, and other stakeholders) to conduct the proposed activities? Does the application describe the project team’s expertise relevant to moving evidence into practice?
- Does the investigator (or co-investigator) have demonstrated experience conducting projects of similar size, scope, and complexity?
- (Dual-PI option only) Does the Leadership Plan adequately describe and justify roles and areas of responsibility of the PIs? Specifically, do the investigators have complementary and integrated expertise? Further, are the leadership, governance, and organizational structures appropriate for the project?
- Is the level of effort for each team member appropriate for successful conduct of the proposed work?
Criterion 5. Patient-centeredness
- Does the application describe how the proposed implementation project has the potential to help people make more informed healthcare decisions or to improve healthcare delivery and/or health outcomes?
Criterion 6. Stakeholder engagement
- Does the application demonstrate that the relevant stakeholder perspectives – including those of the relevant patients or caregivers – have informed the development of the proposal and describe how these stakeholders will be meaningfully engaged throughout the project?
- Does the application demonstrate that personnel (e.g., the frontline staff delivering the intervention or directly supporting the implementation activities) at the proposed implementation sites are clearly interested in the proposed implementation project and are committed to participating as active partners in the project? Have these staff provided input on, or endorsed, the activities they will undertake during the proposed project?
- Does the application demonstrate that decision makers at the proposed healthcare systems and settings where implementation will occur are sufficiently committed to the proposed implementation project, as well as sustaining successful interventions beyond the PCORI-funded project? Does the application describe how these decision makers will be meaningfully engaged throughout the project?
- For Phased Implementation Projects:
- Does the application include meaningful letters of support from the relevant decision makers at Phase 1 and 2 sites/systems, whose support will be critical for ensuring successful use and scaleup of the program during the project, as well as sustainability when the project is completed?
- Do these individuals have the appropriate level of organizational influence and PCORI Cycle 1 2021 Limited PFA: Implementation of PCORI-Funded Patient-Centered Outcomes Research Results 22 decision-making authority? Does the application describe any relevant in-kind support or contributions as additional evidence of commitment and buy-in from participating sites/systems?
- Does the application describe the specific criteria these decision makers will use to make decisions regarding the ongoing maintenance of the program being implemented and the extent to which the proposed project evaluation activities will inform these decisions?
- For Phased Implementation Projects:
- Does the application indicate the relevant regional or national stakeholder organizations whose support will be critical to extending the impact of the PCORI-funded research findings to broader venues? Does the application describe how the project will engage or work directly with these stakeholders?
Posted: February 15, 2019; Updated: January 28, 2021