Pilot Project: Improving Ways to Discover Research Needs from Stakeholders

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

Understanding what matters to patients, family members, healthcare professionals, and other stakeholders helps researchers know what topics to focus on.

Two ways to ask patients and stakeholders about research needs are direct prioritization and best-worst scaling. In direct prioritization, participants put a list of research needs in order from most to least important. In best-worse scaling, researchers ask participants to decide between sets of research needs from a list. For each set, people choose the most important and least important research need. This method allows researchers to see whether the importance of each item changes depending on what other choices are available.

Project Purpose

This study compared direct prioritization to best-worst scaling to find out the research needs of people who are affected by autism or who do work related to autism.

Methods

The research team looked at publications by autism advocacy organizations and the federal government. The team also talked to 17 patients with autism, caregivers of patients with autism, healthcare professionals, and researchers. The research team used what they learned to make a list of topics for future autism research.

Two hundred nineteen patients with autism, caregivers, healthcare professionals, and researchers took an online survey about the research topics. The participants used either direct prioritization or best-worst scaling to choose the research topics that were most important to them. For the best-worst scaling surveys, participants were given different research scenarios and asked to select their most and least favorite part. For the direct prioritization surveys, respondents were given the list of research topics and asked to select a limited number as most important.

Researchers looked at whether there were differences in priorities depending on which method was used. The researchers also looked at what participants valued, and whether people with different characteristics gave different answers.

Findings

Both methods identified the same top priority for future autism research: learning what characteristics of children, families, and therapies lead to better or worse outcomes. The two methods produced different answers about lower research priorities.

Participants in the study had similar values when it came to deciding why a research topic was important. They cared about how many children with autism the research could help, how much a treatment cost, whether a treatment helped with multiple aspects of autism, and whether research focused on both the child and the child’s support system.

Study participants found the best-worst scaling method easier to understand than direct prioritization. People who used best-worst scaling were less likely to skip parts of the process. Different stakeholders came up with similar results more often using best-worst scaling than using direct prioritization.

Limitations

The research team found the study participants by asking people they knew at autism advocacy organizations or universities to participate. For this reason, the people in the study may have known more about autism research than the average patient or caregiver. The results might be different if people with more varied backgrounds were included.

Results might also be different if the research options included more information about the time and cost of doing research.

Importance of Findings

In the past, patients have not always been part of discussions about what to study, particularly in mental health. The best-worst scaling used in this study worked with people with autism and could be useful in other areas of mental health research.

Sharing the Results

The research team presented the study results at conferences. The researchers also wrote materials for patients, caregivers, advocates, other researchers, people who fund research, and people who work on policy about autism.

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

The evidence base for how to capture stakeholder input in the knowledge development process is underdeveloped. Although direct prioritization has commonly been used to identify future research needs, this method only generates rankings and is subject to bias. Best-worst scaling experiments are a novel, simple approach to modeling preferences for future research needs that yields both rank and weight.

Project Purpose

This study examined the comparative effectiveness of best-worst scaling and direct prioritization exercises regarding stakeholder priorities for autism research. Findings provide insights into how to engage stakeholders with psychiatric and cognitive impairment in the research prioritization process.

Study Design

Mixed-methods feasibility study

Participants, Interventions, Settings, and Outcomes

Participants were stakeholders including adults with autism, parents of children with autism, and professionals who provided autism services and conducted research (N = 236; n = 17 in developmental focus groups, n = 219 survey respondents).

Two strategies for prioritizing future research needs were compared: best-worst scaling and direct prioritization. In the best-worst scaling exercise, survey respondents were presented with a series of 18 future research scenarios from which they selected their least and most favorite things. Future research needs were characterized in terms of nine attributes that describe how stakeholders value future research. The best-worst scaling exercise presents survey respondents with a series of future research scenarios comprised of a random selection of five attributes to minimize burden from which they mark what is their least and most favorite thing about that scenario. In the direct prioritization exercise, survey respondents ranked 17 future research priorities directly by voting with a limited number of chits. Respondents were given a total of 12 chits, and could allocate up to four per priority. To rank nine valued attributes of research, respondents were given a total of six chits and again could allocate up to four per attribute.

A survey was fielded in focus groups and via the web among a national convenience sample of autism stakeholders. The group setting provided an opportunity for respondents to review the survey directions and ask questions; answers were provided to the entire group. Each focus group participant completed the survey individually.

Multinomial logit regressions and summary statistics characterize rankings of attributes and priorities for future research.

Pearson correlation coefficients were estimated to assess the correlation of the rankings generated by the two methods overall and stratified by stakeholder type. Missingness and survey completion were compared. Qualitative data were collected to assess acceptability, including cognitive difficulty and meaningfulness.

Data Sources

Preliminary provisional lists of valued research characteristics and knowledge gaps were synthesized from the strategic plans of advocacy groups and federal agencies and from an AHRQ Evidence-Based Practice Center comparative effectiveness review of therapies for children with autism. A qualitative iterative focus group design was used to explore stakeholder values about future autism research and to refine lists. Final values and gaps lists were used to construct best-worst scaling and direct prioritization exercises. The exercises were fielded in focus groups facilitated by two study team members and self-administered via the web.

Quality of Data and Analysis

For qualitative analyses, a values coding approach was used to explore what stakeholders valued, how these values impacted the way they understood and articulated knowledge gaps, and how values translated into beliefs about research priorities. Three members of the research team coded the transcripts to identify values and gaps. The preliminary provisional lists from the literature illustrated naturalistic sources of values and gaps that were corroborated and refined by the group participants to enhance trustworthiness of findings. Researchers corroborated revisions by both listening for suggested changes from stakeholder groups and asking for reactions to them. In addition, the groups provided iterative member-checking to validate study team interpretations. For quantitative analyses, multinomial logit regressions and summary statistics characterized rankings of attributes and priorities for future research and differences by method and stakeholder characteristics.

Findings

Stakeholders developed their values and priorities by framing their respective perspectives as autism stakeholders: living a full life as an adult with autism, being an effective parent, and helping families make the best treatment choices.

The best-worst scaling and direct prioritization exercises identified the same top stakeholder priority overall for future autism research: research to develop better information on treatment effect modifiers and the child, family, and intervention characteristics that lead to the best (and worst) outcomes. The methods diverged (r=0.24, p=0.54) on lower priorities for research needs.  Rankings were significantly correlated across stakeholder types in the best-worst scaling exercises (all p < 0.0001) but not the direct prioritization method. Rankings among self-advocates were different from others using the direct prioritization method (r=0.08, p=0.97), while rankings from others were correlated (p-values<0.05).

Both methods yielded consistent results overall regarding valued attributes of research (r = 0.70, p = 0.03). Stakeholders selected the proportion of children with autism affected by the research, family out-of-pocket costs for the treatment, whether or not the treatment addressed multiple aspects of autism in an integrated manner, and the focus of the treatment on the child and support environment as the most valued attributes. In both exercises, rankings were significantly correlated across stakeholder types (all p < 0.05).

Stakeholders found the discussion of research attributes engaging. The best-worst scaling exercises were easier to understand as evidenced by the fact that unique responses to the 18 scenarios were less frequently skipped than unique allocation of the 12 and six chits of the direct prioritizations. Importantly, the test-worse scaling exercises yielded more consistent results between stakeholders. Acceptability of the instrument did not vary by stakeholder type.

Limitations

Findings are drawn from a convenience sample of stakeholders recruited from academic and advocacy contacts. Findings may not align with the values of less educated and activated stakeholders. Mapping of values onto research scenarios to generate research priorities is imprecise. There may be greater variation in priority scores or different rankings with more complete information on time and financial costs.

Conclusions

The values and research priorities generated provide a road map for researchers and funders to pursue research that matters to autism stakeholders.

There is a need for better research prioritization methods, particularly in the field of mental health, where individuals with mental health needs have typically been excluded from discussions. Although conceptualizing future research is challenging, all stakeholders valued having a voice in the discussion. Best-worst scaling presents a series of simple queries that are feasible for adults with autism. Eliciting stakeholder priorities for future research investment through best-worst scaling is a meaningful exercise that can be extended to other realms of mental health research.