Evaluating the PCORI Way: Does "Research Done Differently" Make a Difference? We Plan to Find Out

At PCORI, we take a unique approach to funding comparative clinical effectiveness research—ensuring that we focus on outcomes important to patients and that patients and others across the healthcare community have a meaningful role in guiding our work. Patients and other stakeholders help us determine which research topics we consider funding and help us review proposals. We also require that patients are included as meaningful partners in the research we support, helping to develop and conduct the studies and disseminate the results.

We call this "research done differently." We’re committed to this approach because we believe strongly that it will focus the research we fund on producing useful information that can answer the questions of greatest importance to patients and those who care for them.

Of course, as we’re a young and still-growing research organization, a major question remains: will our approach yield the results we strive to achieve? As proponents of evidence-based practice, we’re eager to find out the extent to which this research leads to more-useful healthcare information. So we’re pleased to announce the formation of the PCORI Evaluation Group, a task force that will advise PCORI on how best to measure and evaluate the effectiveness of our work, from the usefulness of the studies we fund, to the extent to which the findings from these studies affect health decisions.

The task force includes members of our Board of Governors and Methodology Committee, PCORI staff, and four external advisors selected based on their expertise in specific areas of evaluation research.

PCORI Evaluation Group

Board of Governors

• Gail Hunt, National Alliance for Caregiving, President and CEO
• Robert Jesse, MD, PhD, Department of Veterans Affairs, Principal Deputy Under Secretary for Health (deceased, September 2, 2017)
• Robert Zwolak, MD, PhD, Dartmouth-Hitchcock Medical Center, Vascular Surgeon and Director of the Non-Invasive Vascular Laboratory; Dartmouth Medical School, Professor of Surgery; Veterans Administration Medical Center, Chief of Surgery

Methodology Committee

• Naomi Aronson, PhD, Blue Cross and Blue Shield, Executive Director, Association Technology Evaluation Center
• Michael S. Lauer, MD, National Heart, Lung, and Blood Institute, Director, Division of Cardiovascular Sciences
• Robin Newhouse, PhD, RN, University of Maryland School of Nursing, Chair and Associate Professor, Organizational Systems and Adult Health, Chair of the Methodology Committee

External Advisors

• Kimberly Bailey, MS, Families USA, Research Director and Director of Health System Reform; PCORI Patient Engagement Advisory Panel, Member
• Claire Brindis, DrPH, University of California, San Francisco, Caldwell B. Esselstyn Chair in Health Policy, Director, Philip R. Lee Institute for Health Policy Studies, Professor of Pediatrics and Health Policy
• Floyd Jackson Fowler, Jr., PhD, University of Massachusetts Boston, Senior Research Fellow, Center for Survey Research
• Beverly A. Parsons, PhD, InSites, Executive Director, American Evaluation Association, President-elect

PCORI Staff

• Laura Forsythe, PhD, MPH, Program Officer, Research Integration and Evaluation
• Lori Frank, PhD, Program Director, Research Integration and Evaluation
• Michele Orza, ScD, Senior Advisor to the Executive Director
• Suzanne Schrandt, JD, Deputy Director, Patient Engagement

The task force will hold its first meeting in Washington, DC, on December 13, to be followed by monthly teleconferences during the next year.

Framing Questions, Measuring Progress

At the initial meeting, the group will have two charges. The first is to begin developing an evaluation framework that reflects PCORI’s strategic plan. The framework will include the questions about PCORI’s work that should be answered, and the measures or indicators, methods, and data that can be used to answer them. As the group begins to develop this framework, some of the general questions it may consider are:

• Does PCORI’s approach to research make a difference in the quality of the research and the usefulness of the resulting information? In how the information is used? In the health outcomes that result?
• What aspects of PCORI’s approach make a difference in the quality of the research? In the degree to which our research questions are patient-centered? In how well the research team engages stakeholders? In how well the studies follow PCORI’s methodology standards? In how well our program staff manage their research portfolios to achieve maximum impact?
• What aspects of PCORI’s approach make a difference in the usefulness of the resulting information? In how well study results are disseminated? In how much our work has contributed to enhancing infrastructure to conduct patient-centered CER?

The task force might also evaluate specific aspects of PCORI’s approach to research, such as how stakeholder engagement can be done most effectively and meaningfully.

The group will need to prioritize the questions they consider. Members will identify which answers are most important to advance the field of patient-centered outcomes research, as well as decide which questions we at PCORI can answer ourselves and which are better suited for outside groups to examine.

A second charge is to discuss how PCORI can define and measure achievement of its strategic goals, which were approved by our Board in November. The task force will consider how we can determine whether we have produced useful health information, accelerated the use of research results, and moved research funded by other organizations toward being more patient-centered.

We Want to Hear From You

As the task force sets about its work, we plan to invite input from the healthcare community and report on its activities and recommendations. Just as we need your input to help guide the research we support—whether you’re a patient, caregiver, researcher, clinician, payer, policymaker, or any other stakeholder—so, too, do we value your thoughts on how we can best gauge whether we’re making a difference.

View the presentation slides from the December 13 meeting of the PCORI Evaluation Group.

So we ask you to share with us the evaluation questions about PCORI’s work that are the most important to you.

Questions received by 5 pm ET on Thursday, December 12, will be provided to the task force for consideration during its first meeting, but this is just the first of many opportunities for you to provide input and feedback.

We’re confident that the task force’s discussions, informed by your ongoing contributions from across the healthcare community, will serve as a dynamic source of guidance for us as we work toward meaningful evaluation of our approach to supporting patient-centered comparative effectiveness research.