Results Summary
What was the project about?
In stakeholder-engaged research, patients, families, and other community members are part of a research team. As part of the team, community members can influence the design of a study and other parts of the research process so that the results are useful for their communities.
How research teams involve community members varies across studies. Some involve people in just one part or in a limited way, while others involve people much more. Having a consistent way to assess how involved community members are in a research project may help improve the process and the impact of stakeholder-engaged research.
In this study, the research team created a survey that researchers can use to assess how much community members are involved in research teams.
What did the research team do?
The research team used the Delphi method to create a survey with input from a panel of community members. The Delphi method helps panels share feedback and reach agreement through rounds of discussion and voting.
The panel included 19 patients, advocacy organization members, and researchers. Of the panelists, 63 percent were African American, 32 percent were White, and 5 percent were multiracial. The median age was 55, and 90 percent were women.
The research team started with a list of 11 best practices on how to engage community members in research. The team created survey questions about each practice. The team also developed definitions and scenarios about how research teams involved community members.
In each round of discussion, panelists gave feedback on the materials and decided to keep, change, or remove each survey question based on its importance. The panel reached consensus on the survey questions to include in the final survey.
Patients, caregivers, healthcare providers, and researchers helped design and conduct the study.
What were the results?
The final survey had 32 questions organized by eight best practices. These practices included:
- Focus on community views of health.
- Include community input.
- Make sure partnerships between researchers and the community last.
- Help community members learn and benefit from the research.
- Make sure research teams build on communities’ strengths.
- Make sure partnerships involve working together and sharing power.
- Have community members share results of studies.
- Build and keep trust.
Panelists and the research team also created a way to score studies on community engagement. The scoring can show a range of engagement from simply getting input from community members, to having community members be partners in making decisions about the research.
The research team also created a shorter version of the survey that takes less time to complete.
What were the limits of the project?
The research team tested the survey in English. It may not reflect the experiences of people who speak other languages.
Future research could translate the survey and test it with people who speak other languages.
How can people use the results?
Researchers can use the survey to assess how involved community members are in research studies.
Professional Abstract
Background
Stakeholder-engaged research includes patients, caregivers, community members, advocacy groups, and others as part of the study team to improve the quality and relevance of research. The level of engagement can vary from minimal engagement to fully collaborative partnerships. Assessment of the process and impact of stakeholder engagement on research requires a standardized measure developed from the stakeholder perspective.
Objective
To develop and validate a measure to assess stakeholder engagement in research studies from a stakeholder perspective
Study Design
| Design Element | Description |
|---|---|
| Design | Empirical analysis |
| Data Sources and Data Sets |
Data collected through a Delphi panel of 19 participants including patients, advocacy organization members, and researchers |
| Analytic Approach |
Standardized Delphi method with 5 rounds of soliciting feedback from panelists, panel voting, and consensus-building |
| Outcomes |
Measure to assess the level of stakeholder engagement in research studies |
Methods
This study used a five-round, standardized Delphi panel review to develop and validate a measure of stakeholder engagement. A Delphi panel review is a consensus-building method that involves multiple rounds of feedback from individuals with expertise on a topic.
Researchers used a list of 11 best practices for engaging community members in research that was previously developed from the academic literature plus questions to assess each best practice. In each round, panelists provided feedback on and indicated whether to keep, modify, or remove each best practice definition and corresponding questions. They also reviewed levels of engagement and their definitions and case scenarios corresponding to different levels of engagement. After reaching consensus with the panel, researchers created the final measure.
The panel included 19 patients, advocacy organization members, and researchers from across the United States. Of the panelists, 63% were African American, 32% were White, and 5% were multiracial. The median age was 55, and 90% were female.
Researchers examined internal consistency and convergent validity of the measure and completed cognitive interviews of survey questions to improve reliability and validity.
Patients, caregivers, healthcare providers, and researchers helped design and conduct the study.
Results
The final measure, the Research Engagement Survey Tool (REST), includes 32 questions organized by eight best practices for stakeholder engagement in research. The eight best practices include:
- Focus on community perspectives and determinants of health.
- Include input from partners.
- Sustain partnerships to meet goals and objectives.
- Foster co-learning, capacity building, and co-benefit for all partners.
- Build on strengths and resources within the community or patient population.
- Facilitate collaborative, equitable partnerships.
- Involve all partners in the dissemination process.
- Build and maintain trust in partnerships.
Based on final survey analyses, researchers determined a scoring schema based on a five-category continuum of engagement developed by panelists and researchers: (1) outreach and education, (2) consultation, (3) cooperation, (4) collaboration, and (5) partnership.
Researchers also developed a nine-question version of REST to decrease respondent burden.
Limitations
Broader use of the measure will require further testing of its performance in paper, phone, or other surveys. The research team developed the measure in English; it may not capture the experiences of those who speak other languages.
Conclusions and Relevance
In this study, the research team developed a measure of stakeholder-reported engagement in research studies with a panel of community members.
Future Research Needs
Future research could assess the measure’s performance in surveys of patient engagement in research and could develop versions of the measure for individuals who speak languages other than English.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers asked about the report’s references to item response theory, stating that it was not clear in the text why this work was done and how it contributed to instrument development. The researchers explained that they did not use item response theory in the traditional sense, instead applying the technique to help determine which items might be potential candidates for inclusion or exclusion in the condensed version of their final Research Engagement Survey Tool (REST).
- The reviewers questioned the development of tailored versions of REST for different demographic groups, pointing out that individual scale items showing differential responding by demographic group should probably be excluded instead, to ensure the consistency of the instrument. The researchers agreed that having different groups of people respond to different items on the same scale would not be helpful for scale development. However, they explained that they tested the tailored versions of REST to determine the feasibility of using such versions in the future. The researchers noted that future investigators might want to use a tailored subset of REST items for specific demographic groups.
- The reviewers requested more statistical information regarding the factor analysis described in the report. The researchers explained that they did not make any inferences from the factor analysis or create factors or subscales from the results, instead using the factor analysis descriptively as one component of a triangulation approach to identify which REST items i would be best to use in a condensed version of the measure.
Conflict of Interest Disclosures
Project Information
Patient / Caregiver Partners
- Lorretta Jones, Healthy African American Families II;
- Disparities Elimination Advisory Committee, Program to Eliminate Cancer Disparities at Siteman Cancer Center;
- St. Louis Patient Research Advisory Board
Other Stakeholder Partners
- Elizabeth Baker, Saint Louis University
- Edith Parker, University of Iowa
- Nancy Schoenberg, University of Kentucky
- Consuelo Wilkins, Meharry-Vanderbilt Alliance
Key Dates
Study Registration Information
^Melody S. Goodman, PhD, MPH, was the principal investigator when this project was initially awarded.