Results Summary

What was the project about?

Research teams usually decide on research agendas, or issues and questions to study in research projects, even when community stakeholder organizations, or CSOs, such as healthcare networks and patient organizations, are involved.

In this project, the research team explored a process for helping CSOs create research questions and set research agendas.

What did the research team do?

The research team worked with six CSOs in California, New York, and Oklahoma. They developed an agenda-setting process with five steps. In this process, CSOs

  1. Chose the community’s 15 most pressing issues
  2. Created rules for decision making informed by what matters most to the community
  3. Reduced the list to the top three issues, based on the rules created in step 2
  4. Collected and weighed information on the top three issues
  5. Decided on a final issue by weighing the costs and benefits

To help CSOs decide on the final issue in the fifth step, the research team created a six-topic decision guide. The topics were

  • Risk-benefit ratio: The chance of a major change in the health of the community
  • Reduced uncertainty: The chance that strong science supports the research
  • Feasibility: The chance that the changes required by the research could happen in the real world
  • Sustainability/durability: The chance that changes from the research, and their effects, would last
  • Reach: The chance that many patients or community members could benefit from the research
  • Person-centered and community-aligned: Confidence that the research will strengthen health care and address the needs, values, and goals of the community

Each topic had a set of questions to help CSOs decide between issues.

CSOs used the decision guide in one of two ways. Some groups had discussions using a value tree picture. The picture showed each topic from the guide and a discussion question on a leaf of a tree. Other groups did a rating exercise. In this exercise, members used a scale of 1 to 10 to rate the chance that the research would improve each guide topic. They also rated their confidence in that rating and how important information on that topic was to decision making.

Patients, doctors, researchers, and CSO staff gave feedback throughout the study.

What were the results?

CSOs said that using the value trees helped them talk about important values that they didn’t usually discuss when making decisions.

CSOs said using the rating exercises helped them learn new things to think about before deciding to invest time and other resources into ideas for improving community health. But the groups also said they had a hard time rating certain topics, especially

  • Risk-benefit ratio
  • Reach
  • Person-centered and community-aligned

What were the limits of the project?

The decision guide may be too complex for lay audiences to use, especially in a short time. Future research could adapt the decision guide so that the questions make more sense to patients and community members.

How can people use the results?

CSOs can use these results to help set agendas for research.

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers questioned how representative the study sites were of the geographic areas they represented. The researchers responded that in this qualitative study, they could only study a limited number of geographic regions and involve a limited number of organizational partners. Therefore, they focused on studying divergent examples of primary care stakeholder voices and experiences rather than a representative sample. The researchers said their work is an exploratory study of an understudied topic, so they did not expect their findings to be generalizable to every population. To further justify their choices of organizations to study, the researchers added information about each organization to illustrate its unique qualities and how those qualities supported study implementation.
  • The reviewers noted that the researchers implemented a highly complex and time-consuming process for helping community organizations to prioritize their goals for engaging in research.  The reviewers pointed out that the process was so time consuming that the researchers were not able to present much information on the actual priorities identified in the sampled organizations. The researchers agreed that the process was complex but reminded the reviewers that the goal of this project was to develop methods for priority setting rather than to identify those priorities. To this end, the researchers felt that the study was quite successful as it concluded with a rubric for how to conduct priority-setting exercises that were less complex than those used in this study.
  • The reviewers wondered how small sample sizes in some regions affected study findings. The researchers agreed that there were imbalances in data collection across study sites but said they took care to examine patterns of how participants were assigned in different groups and concluded that these differences had only a modest effect on overall results.

Conflict of Interest Disclosures

Project Information

Zsolt Nagykaldi, PhD^
University of Oklahoma Health Sciences Center
$855,717 *
Development of a Patient-Directed Queries Network to Engage Patients and Prioritize Their Questions to Inform the PCORI Research Agenda**

Key Dates

September 2014
June 2020

Study Registration Information

^James Mold, MPH, MD was the original principal investigator for this project.
**This project was previously titled: Patient and Community Stakeholder Engagement in Identifying and Prioritizing an Agenda for Research in the Primary Care Setting

Final Research Report

View this project's final research report.


Has Results
Award Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 20, 2023