How Are We Evaluating Our Portfolio?
We analyze and report on the research we fund to ensure that our portfolio is meeting the needs prioritized by our stakeholders. We review our portfolio to determine the distribution of projects by therapeutic area, populations of interest, study design, intervention type, systems level, and location on the care continuum, such as prevention, diagnosis, or treatment.
We compare the distribution of the funded projects to that of the suggested research topics submitted to the PCORI website by stakeholders and the general public. We also examine how our funded projects align with other organizations’ priorities for comparative clinical effectiveness research.
With this information, we can identify gaps in research funding and trends to help us make strategic decisions about how to prioritize topics for future funding.
How Are We Evaluating Merit Review?
We administer surveys to funding applicants and merit reviewers and also conduct group interviews with a subset of our reviewers to learn how well our unique process is working and collect suggestions for improvement. The feedback has already led to substantial changes.
In additional assessments, we analyze the scores across all cycles to better understand how reviewers apply our criteria. We also examine how our unique merit review system affects the projects we select for funding.
PCORI Merit Reviewer Survey
We conduct surveys to compare perspectives on the merit review process from patient, scientist, and other stakeholder reviewers; to improve merit review at PCORI; and to contribute to the study of the effects of including patients and stakeholders in merit review.
- Webinar: PCORI Merit Review: Learning from Patients, Scientists, and Other Stakeholders
- Results presented to Board from a trend-over-time analysis of the Merit Reviewer Survey, September 15, 2014
- PCORI Merit Review Webinar: An Applicant and a Couple of Reviewers Walk into a Webinar: Lessons Learned from PCORI Merit Review
- Presentation on the Engagement Resources from the Applicant and Reviewer Survey at the January 2015 Advisory Panel on Patient Engagement Meeting
PCORI Merit Reviewer Group Interviews
We convene group interviews with merit reviewers after every merit review cycle to obtain their perspectives on our review process, to follow up on results from our surveys, and to facilitate discussion of ideas about merit review. The information is used to improve merit review processes at PCORI and contribute to the study of patient and other stakeholder inclusion in review of funding applications.
PCORI Merit Reviewer Score Analysis
We analyze scores to understand the role each of the PCORI merit review criteria plays in evaluation of funding applications and the impact of including patients and other stakeholders in review of health research funding applications. The information assists in our effort to improve PCORI merit review and ensure we incorporate multiple perspectives while maintaining the high quality of proposals funded.
- Peer Review Congress: In September of 2013, Laura Forsythe, presented on the relationships among scores of the 3 reviewer types, changes in scoring after in-person discussion, and the effect of inclusion of patient and stakeholder reviewers on PCORI’s Cycle I review process at theSeventh International Congress on Peer Review and Biomedical Publication.
- Analysis of PCORI Review of Application on Rare Diseases Presented to the January 2015 Advisory Panel on Rare Disease
- Analysis of Rare Diseases Applications Presented at the May 2015 Board of Governor’s Meeting
- Follow-up Analysis of Letters of Intent (LOIs) on Rare Diseases Presented to the May 2015 Rare Disease Advisory Panel
- Methods Consultation Panel for Pragmatic Clinical Studies: Evaluation and Recommendations Presented to the May 2015 Clinical Trials Advisory Panel
- Engaging Patients and Stakeholders in Research Proposal Review: The Patient-Centered Outcomes Research Institute, Annals of Internal Medicine (2014)
How Are We Evaluating Engagement in Research?
We are constantly working to identify and spread best practices for engagement in research. To this end, we have developed several tools for obtaining information from our awardees.
Pilot Projects Engagement Data Collection
We funded our first research through the Pilot Projects. To learn as much as possible, as quickly as possible, from these first awardees, we developed a self-report instrument for awardees to complete throughout their work.
- Pilot Projects Engagement Data Collection Tool
- ARM Presentation: This presentation focused on the assessment of patient and stakeholder engagement in the PCORI pilot projects at the AcademyHealth 2014 Annual Research Meeting (ARM).
- Webinar Slides
- Manuscript: Forsythe, L. P., Ellis, L.E., Edmundson, L., Sabharwal, R., Rein, A., Konopka, K., Frank, L. Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned, Journal of General Internal Medicine.
Tools to Collect Information on Research Engagement
Using what we learned from the pilot projects, we developed tools to collect information on engagement in research from PCORI-funded researchers (via their regular project reporting) and their patient and stakeholder partners (Ways of Engaging - ENgagement ACtivitiy Tool, WE-ENACT).
The collected information will help us describe engagement in research and identify best practices to guide PCORI applicants, awardees, and others interested in conducting patient-centered outcomes research.
- Presentation on WE-ENACT from the March 2014 PEG Meeting
- Presentation of the preliminary WE-ENACT results at the December 2014 Board of Governors Meeting
- Presentation of the preliminary WE-ENACT results at the January 2015 Advisory Panel on Patient Engagement Meeting
- Understanding Engagement in Research Presented at the May 2015 Board of Governor’s Meeting
- Understanding Engagement in Research Presented at the June 2015 Advisory Panel on Patient Engagement Meeting
- Evaluating Engagement in Research: Promising Practices from PCORI’s Portfolio presented at the October 2015 PCORI Annual Meeting
- Engagement Report deployed to PCORI Researchers (For use through March 2017)
- Engagement Report deployed to PCORI Researchers (PCORI Online version)
- WE-ENACT self-report tool deployed to Patients and Stakeholders Engaged in PCORI Research
PCORnet Ways of Engaging—ENgagement ACtivity Tool
PCORI has also developed a tool called PCORnet Ways of Engaging—ENgagement ACTivity (Net-ENACT) to collect information that describes engagement of patients and other stakeholders in the networks—both Clinical Data Research Networks and Patient-Powered Research Networks—that make up PCORnet.
Results will be used to learn how research networks implement engagement, identify best practices, and determine the effects of engagement.
- Baseline Net-ENACT self-report tool deployed to PCORnet Primary Investigators and Engaged Stakeholders
- 6-month Net-ENACT self-report tool deployed to PCORnet Primary Investigators
- 6-month Net-ENACT self-report tool deployed to PCORnet Engaged Stakeholders
Published Research that PCORI Has Commissioned or Conducted on Engagement in Research
- Integrating Patients’ Voices in Study Design Elements with a Focus on Hard-to-Reach Populations
- Eliciting Patient Perspective in Patient-Centered Outcomes Research - A Meta-Narrative Systematic Review
- Eliciting Patient Perspective in Patient-Centered Outcomes Research - Expert Interviews Part 1
- Eliciting Patient Perspective in Patient-Centered Outcomes Research - Expert Interviews Part 2
- Eliciting Patient Perspective in Patient-Centered Outcomes Research - Expert Interviews Part 3
- Eliciting Patient Perspective in Patient-Centered Outcomes Research - Expert Interviews Part 4
- A Systematic Review of Approaches for Engaging Patients for Research on Rare Diseases
- Conceptual and Practical Foundations of Patient Engagement in Research at the Patient-Centered Outcomes Research Institute
How Are We Evaluating PCORI Events?
We survey participants before, immediately after, and six months after each PCORI-hosted workshop or other event that includes patients and other stakeholder. These surveys track the involvement of participants in patient-centered outcomes research. We use the results to improve effectiveness of events and understand the influence we are having on building a patient-centered outcomes research community. View results from these surveys.
How Are We Evaluating Stakeholder Views?
We use a variety of approaches, including surveys and focus groups, to collect information on patient, clinician, researcher, and other stakeholders’ attitudes and behaviors related to comparative clinical effectiveness research (CER) and engagement in research. The goal is to identify and spread best practices by recognizing views from key communities and learning about barriers to, and facilitators of, CER and engagement in research.
Survey of Patient and Clinician Views on CER and Engagement in Research
We commissioned the survey firm InCrowd to collect information on attitudes and knowledge about CER and engagement in research. This survey included patients with rare conditions or chronic diseases, as well as caregivers and primary care clinicians, including physicians, nurses, and physician assistants.
- Manuscript: Forsythe, L. P., Frank, L., Walker, K.O., Anise, A., Wegener, N., Weisman, H., Hunt, G., Beal, A. Patient and Clinician Views on Comparative Effectiveness Research and Engagement in Research, Journal of Comparative Effectiveness Research.
PCORI Patient, Caregiver, and Clinician Surveys
We are performing nationally representative surveys to collect information on attitudes toward CER and engagement in research among various healthcare communities: patients with chronic diseases or rare conditions; caregivers of family members or friends with chronic or rare conditions; and primary care clinicians, including physicians, nurse practitioners, and physician assistants.
- Nov 4, 2015 Webinar: Listening to Patients, Caregivers, and Clinicians: Meeting Stakeholder Needs for Comparative Effectiveness Research- A PCORI Survey
- June 2015 Presentation to the Board of Governors: PCORI Stakeholder Surveys Patients, Caregivers, Clinicians, & Researchers
- Blog: Surveys Show Great Interest in Patient-Centered Clinical Research
PCORI Researcher Survey
Through surveys of researchers, we are collecting data about CER practices, experiences with PCORI funding-application submission, barriers to and facilitators of engaging patients and other stakeholders in research, and perceptions of our programs.
- Nov 18, 2015 Webinar: Listening to Researchers: Meeting Stakeholder Needs for Comparative Effectiveness Research- A PCORI Survey
- January 2015 Presentation to the Board of Governors on the PCORI CER Stakeholder Surveys: Update and Preliminary Results from Researchers
Health Information National Trends Survey (HINTS)
HINTS routinely collects nationally representative data about the American public's use of cancer-related information. We are collaborating with HINTS researchers to add survey items to assess awareness of, perceptions of, and interest in patient engagement in health research.
Other Methods for Collecting Stakeholder Views on Patient-Centered CER
To guide our future direction, we are planning to collect opinions through focus groups and interviews about PCORI’s progress. We will learn from different stakeholder communities, such as payers, clinicians, purchasers, and industry, how PCORI’s work can be most useful to them.
Posted: November 7, 2014; Updated: February 2, 2017