Our Evaluation Work Flows from Our Strategic Plan: Our strategic plan focuses on funding and conducting highly relevant research likely to change practice and improve patient outcomes; on disseminating and promoting implementation of the results of that research; and on influencing how clinical research is done by others, so a greater proportion of all clinical research is useful to patients and other healthcare decision makers.
Our evaluation process focuses on whether we are accomplishing what we set out to do. Are our strategies and priorities advancing us toward our goals? Are we having the desired impact of improving health outcomes?
Our Evaluation Plans Are Organized into a Framework
The PCORI Evaluation Framework 3.0 contains all of the questions our stakeholders and staff have submitted about our work and describes how we’ll go about answering them. It addresses how we conduct our day-to-day work, how we are progressing toward our strategic goals, and ultimately, if and how "research done differently" will make a difference for patients and public health.
Our framework reflects our strategic plan and evolves along with our work because, even as we attempt to answer the initial questions, new ones arise. Read more here about our discussion at the Advisory Panel on Patient Engagement January 2014 Meeting and at the Board of Governors May 2014 Meeting.
We Use a "Dashboard" to Report to Our Board
We use a visual representation of metric tracking, or "dashboard," for reporting to our Board of Governors what we are learning and how we are incorporating our findings into our work.
Our dashboard continues to evolve, and we update it and its supporting materials quarterly to present to the Board in a public meeting.
View our Dashboards
Current Dashboard: May 8, 2017
Past Dashboards: March 21, 2017 | December 13, 2016 | October 31, 2016 | May 23, 2016 | March 22, 2016 | December 7, 2015 | September 28, 2015 | May 4, 2015 | March 24, 2015 | December 8, 2014 | September 15, 2014 | May 5, 2014 | January, 28 2014 | September 23, 2013 | May 6, 2013
How We Evaluate Progress Toward Our Goals
As described in our strategic plan, PCORI has three goals: to increase useful information, to speed the use of that information, and to influence research to be more patient-centered. Each of three Strategy Committees, made up of members of our Board and Methodology Committee, focuses on one of these goals and the metrics and evaluation activities associated with it. Read more here about the Background on PCORI’s Strategic Goals.
Goal 1: Increase Useful Information
The Science Oversight Committee focuses on this goal.
The ultimate test of the usefulness of the information that PCORI produces will be whether people use it. But, because such results are at least a few years away, we are developing criteria to assess the potential usefulness of information from PCORI-funded studies and examining how well these criteria are reflected in all our processes, particularly those for determining which studies to fund.
We presented a first draft of these usefulness criteria to a joint session of the PCORI Advisory Panels and to the PCORI Evaluation Group (PEG).
- Usefulness criteria slides presented at the PCORI Advisory Panels at the January 2014 meeting
- Blog explaining the usefulness criteria
- April 2014 PEG slides explaining the usefulness criteria
We presented a refined draft of the usefulness criteria and initial results to the PCORI Addressing Disparities Advisory Panel.
- Evaluation Framework & Usefulness Criteria slides presented at the Addressing Disparities Advisory Panel January 2015 meeting
To assess progress toward our first goal, we also examine our processes for planning and managing our portfolio and the portfolio itself. We collect and analyze data on:
- Applicants’ understanding of our Methodology Standards and how well projects adhere to relevant standards
- Implementation of our topic prioritization and merit review processes and the results they are yielding
- Composition of our portfolio and how it compares with portfolios of other funders
- Success of projects in staying on track and the effectiveness our project management
Goal 2: Speed the Uptake and Use of Information
The Engagement, Dissemination, and Implementation Committee focuses on this goal.
We have developed a broad spectrum of measures to assess the dissemination, uptake, and impact of information from PCORI-funded comparative clinical effectiveness studies when they are completed, beginning in a couple of years. This spectrum begins with measuring early indicators of the dissemination of results for all PCORI studies. It then progresses to indicators of uptake and use of results, to be measured for a substantial subset of PCORI studies. It ends with indicators of the impact of PCORI results on health decision making, healthcare quality, and health outcomes, which we plan to measure for a small set of studies.
Goal 3: Influence Research to Be More Patient-Centered
The Research Transformation Committee focuses on this goal.
We had anticipated that we would influence other researchers and research funders after we had demonstrated the merits of our approach, with research results judged by end-users to be useful and then applied. However, we find that many people are already following our example. They tell us that they have been influenced by observing our work or participating in some aspect of it. We are collecting and documenting instances brought to our attention and featuring some of them in our quarterly reports to our Board of Governors.
We have developed a set of measures to track and understand our influence and are now, or soon will be, measuring:
- Endorsement, promotion, and dissemination of PCORI work
- Use of PCORI Methodology Standards for Patient-Centeredness
- Use of PCORI approaches to:
- Topic generation and research prioritization
- Merit review
- Communication and dissemination of findings
- Use of PCORI guidance on patient-centered comparative clinical effectiveness research
- Use of PCORI-supported curricula or training
- Collaborations, including co-funding, with other funders
- Amount of total patient-centered outcomes research (PCOR) funding that comes from funders other than PCORI
We also plan to measure the following, but we will wait a few years for the programs to mature:
- Use of evidence from the PCOR Methods program research portfolio
- Use and support of PCORnet, the National Patient-Centered Clinical Research Network
Posted: November 13, 2014; Updated: September 28, 2017