This document describes how PCORI created our Engagement in Health Research Literature Explorer (Literature Explorer). The Literature Explorer is an online repository for peer-reviewed literature that describes or evaluates stakeholder engagement in health research. It provides detailed information about our PubMed/MEDLINE search strategy, inclusion/exclusion criteria, article tagging definitions, and process for creating and maintaining this resource. We welcome your comments and inquiries about the Literature Explorer at [email protected].


PubMed/MEDLINE Search and Supplemental Sources

To keep this collection current, we regularly conduct a literature search that spans various sources. One component includes a recurring search of the PubMed/MEDLINE database, hosted by the National Library of Medicine.

Table 1 lists the terms we include in our search string. We created the initial search using previous systematic reviews,1,2,3 as well as a content analysis of search results. We periodically vet new search terms for inclusion in our search strategy to maximize our ability to capture relevant content about engagement in health research.

As we continue to improve our search strategy, expand the content we include, and refine how we categorize relevant articles, we frequently have a collection of articles identified for review but not yet available in the Literature Explorer. If you are using the Literature Explorer to conduct research, please connect with us at [email protected] so that we can discuss how our efforts to curate literature on engagement in health research can support your research needs.

Table 1: Terms Included in PubMed/MEDLINE Search String

  Concept: Stakeholders Concept: Engagement Concept: Publication Type Concept: Research Other Terms
Medical Subject Headings (MeSH)^
  • Community participation[MeSH]
  • Patient participation[MeSH]
  • Stakeholder participation[MeSH]
  • Clinical trial[ptyp]
  • Comment[ptyp]
  • Comparative study[ptyp]
  • Editorial[ptyp]
  • Evaluation studies[ptyp]
  • Letter[ptyp]
  • Meta-analysis[ptyp]
  • Observational study[ptyp]
  • Pragmatic clinical trial[ptyp]
  • Randomized controlled trial[ptyp]
  • Systematic[sb]
  • Citizen science[MeSH]
  • Comparative effectiveness research[MeSH]
Key Words^^
  • Caregiver(s)
  • Clinician(s)
  • Community(ies)
  • Consumer(s)
  • Employer(s)
  • Insurance
  • Lay
  • Partner(s)
  • Patient(s)
  • Payer
  • Pharmaceutical
  • Policy makers
  • Provider(s)
  • Public Stakeholder(s)
  • Collaboration
  • Collaborative
  • Engage
  • Engaged
  • Engagement
  • Engages
  • Engaging
  • Input
  • Involvement
  • Participation
  • Partner(s)
  • Partnered
  • Partnership
  • Clinical control(led) trial
  • Clinical trial
  • Comparative study
  • Observational study
  • Pragmatic clinical trial
  • Pragmatic trial
  • Randomized clinical trial
  • RCT
  • Randomized control(led) trial
  • CER
  • Collaborative research
  • Community-academic partnership
  • Community-engaged research
  • Comparative effectiveness research
  • Comparative effectiveness trial
  • Engagement method*
  • Participatory action
  • Patient and public involvement
  • Research
  • Conceptual framework
  • Conceptual model
  • NOT preference*
  • Practical framework
  • Practical model
  • Public Involvement Impact Assessment Framework

Criteria for Publication Inclusion and Exclusion

We screen articles for inclusion in the literature explorer in two phases, using the abstracts (Table 2) and full text (Table 3). Articles sourced from PubMed/MEDLINE and the Journal of Patient-Reported Outcomes are first screened at the abstract level followed by a full-text review. Articles from all other sources go directly to a full-text review.

We will provide additional information regarding the number of articles screened at each phase upon request.

Table 2: Inclusion and Exclusion Criteria for the Title and Abstract Screening Phase

Inclusion Criteria

  • In English

Exclusion Criteria

  • Not a peer-reviewed article
  • Not about health research (e.g., agricultural health, occupational health, environmental health)
  • Evaluates or describes engagement in health care rather than engagement in research
  • Evaluates or describes patient preference on health care


Table 3: Inclusion and Exclusion Criteria for the Full-Text Review Phase

Inclusion Criteria

  • In English
  • Describes engagement in health research (e.g., in planning, conducting, or disseminating research; NOT limited exclusively to comparative effectiveness research [CER])
  • Includes information relevant to at least one of the five engagement topics: (1) examples of engagement in health research studies; (2) detailed descriptions of engagement in health research studies; (3) evidence for engagement; (4) editorials or commentaries; and (5) models or frameworks

Exclusion Criteria

  • Not a peer-reviewed article
  • Not about health research (e.g., agricultural, occupational health, environmental health)
  • Evaluates or describes engagement in health care rather than engagement in research
  • Evaluates or describes patient preference on health care
  • For engagement topics 1 and 2 only: No description of the effects of engagement in research (i.e., only describes anticipated engagement or only mentions the engagement of partners without describing how learnings from partners were incorporated, changes were made as result of engagement, or some other perceived impact of engagement)
  • Patients/stakeholders involved only in delivering or administering the intervention; no other forms of patient/stakeholder engagement described

Article Tagging

For articles that meet the inclusion criteria, we review the full texts and apply applicable tags across three engagement-related categories: engagement topic, stakeholder involvement and phase of research engagement.

Engagement Topic (not mutually exclusive)

  • Examples of Engagement in Health Research
    • Manuscripts with a primary objective of reporting on a health research study that engaged partners in at least one phase of the research and describe at least one impact of engagement on the research study
  • Detailed Description of Engagement in Health Research
    • Manuscripts with a primary objective of reporting a detailed description of the planning, conduct, and/or impacts of engagement within the context of a health research study or research agenda
  • Evidence for Engagement
    • Manuscripts that include a formal evaluation of engagement within the context of a health research study, or a study with the primary objective to evaluate or synthesize engagement methods/impacts in health research
  • Editorial or Commentary
    • Manuscripts that express a theoretical view on engagement in health research, including scientific commentaries or opinion briefs
  • Measurement
    • Manuscripts describing instruments/measures, measurement topics, or evaluation frameworks for engagement in health research or related fields (e.g., community-based participatory research)4
  • Model or Framework
    • Manuscripts that introduce a new or revised conceptual framework or model to organize or contextualize concepts related to engagement in health research

Stakeholder Involvement (not mutually exclusive)

Stakeholder groups actively engaged in the research as described in the article, if applicable.

Table 4: Stakeholder Descriptions and Examples

Stakeholder Group




Providers of health care in a clinical setting, including physicians, nurses, physician assistants, rehabilitative professionals, pharmacists, mental healthcare providers, complementary and alternative healthcare providers, and members or representatives of professional societies that serve clinicians

Family physician in direct patient care in a small group practice, hospital floor nurse, psychologist in private practice, dentist, member of the American Medical Association or the American Nurses Association


Persons with current or past experience of illness or injury, persons at risk for a condition or who consume healthcare services, or members or representatives of advocacy organizations that represent patients

Person with type 1 diabetes, cancer survivor, person whose leg was amputated after a traumatic injury, person with bipolar disorder, patient council leader for a national heart disease advocacy organization, patient engagement specialist for a state-based newborn health coalition, executive director of a stroke patient advocacy foundation who also has personal experience with strokes


Family members or other unpaid persons who provide care to patients, or members or representatives of advocacy organizations that represent caregivers

Spouse of a patient receiving home hospice care for advanced amyotrophic lateral sclerosis (ALS), representative from the National Alliance for Caregiving

Hospitals/health Systems

Organizations where care is delivered, including public and private hospitals and health systems, urgent care centers, retail health clinics, and community health centers, as well as organizations that represent these facilities

Clinic administrator of a multispecialty physician practice, chief operating officer of a health system


Companies that design, invest in, or manufacture diagnostics, devices, pharmaceuticals, electronic records systems, and mobile apps, as well as organizations that represent the life sciences industry

Chief medical officer of a device manufacturer, representative from a pharmaceutical company, The American Medical Informatics Association


Companies that purchase health benefits for employees and their dependents, including individual businesses as well as local, state, regional, and national business groups; coalitions that represent businesses; and health coalitions

Owner of a family-run business; staff of a state chamber of commerce; health benefits manager of a large, ERISA-covered employer


Companies that function as financial intermediaries in the health system, including private insurers, public insurers, and organizations that represent insurers, such as America’s Health Insurance Plans

Chief medical officer of a health insurance cooperative; claims adjuster at a large insurer; policy analyst at an insurance trade association; insurers BlueCross BlueShield, Medicaid, and Medicare

Training institutions

Institutions that deliver health professional education, including public and private universities and colleges, individuals affiliated with the delivery or administration of health professional educations, and trade or professional associations that represent these institutions, organizations, and individuals

Dean of a nursing school, director of a residency program, manager of a continuing medical education provider

Policy makers

Persons who help craft public policy at any level of government, including federal, state, and local government officials; representatives of federal, state, and local units of government; and representatives from organizations that represent policy makers

State Medicaid director, representative of the Office of the National Coordinator for Health IT, policy analyst at the National Association of County and City Health Officials, National Conference of State Legislatures, National Association of Counties


Other groups not covered by the categories above, such as community organizers, faith-based leaders, subject matter experts, or researchers who are not part of the primary research team

Community organizer for Planned Parenthood, faith leader for a place of worship, subject matter expert on patient engagement who is not affiliated with the research team

Phases of Research Engagement (not mutually exclusive)

Phase(s) of research in which stakeholders were engaged as described in the article, if applicable (Table 5).

Table 5: Phases of Research Engagement Descriptions and Examples

Phase of Research in Which Engagement Occurs

Description of Engagement Activities

Example of Engagement Activities

Research topic and agenda

Identifying, refining, and/or prioritizing topics and populations to study

Providing insight about the concerns and priorities of the patient population or stakeholder groups

Research questions and aims

Developing and/or refining the research questions and aims of the study

Reframing aims of the research study to highlight health issues within certain communities 

Study comparators and content

Selecting, designing, and/or refining interventions to be tested in the study

Altering the delivery mode of the intervention to be more aceptable to patients

Study outcomes

Selecting and/or refining study outcomes and deciding how to measure them

Selecting endpoints that are important to patients and other stakeholders 

Study design

Contributing to a specific plan or protocol for conducting the study, including which inclusion criteria should be used to select participants or helping select which methods will be used to conduct the research

Suggesting to use a study design that takes patients’ preferences into account instead of using randomization, in order to better appeal to the study population

Participant recruitment and retention

Refining approaches and/or materials for finding patients who are eligible to participate in the study, enrolling them into the study, and/or making it easier for them to participate and remain in the study

Rewording the consent form to include clearer descriptions of study activities

Data collection

Developing and/or refining approaches and methods to best collect data from study subjects

Improving the timing of data collection to better align with clinic flow

Results review and analysis

Analyzing collected data or reviewing study results to help interpret and explain research findings

Assisting with the transcription and coding of qualitative data, and/or interpreting preliminary findings


Applying study results to real-world practice, health condition management, policy, or future research

Contributing to the development of clinical practice guidelines


Planning, creating, and/or presenting materials that share information about the study or the results of the study

Suggesting the best means of returning results and other study findings to study participants, and/or giving presentations at conferences to spread results to a wider audience

Additional tags and other bibliographic data may be available upon request. Please contact us at [email protected] to learn more.

Download this Document

^ National Library of Medicine’s controlled vocabulary thesaurus used for indexing articles for PubMed
^^ Key words for stakeholder and engagement concepts are used in combination in our search string. Please contact us at [email protected] to learn more.

In addition to articles identified via PubMed/MEDLINE searches, we screen articles for inclusion by reviewing all PCORI awardee articles with descriptions of engagement, highly relevant journals not yet indexed in PubMed (e.g., Research Involvement and Engagement, The Journal of Patient-Reported Outcomes), and other articles recommended by PCORI staff.

1 Concannon TW, Fuster M, Saunders T, et al. A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research. Journal of General Internal Medicine. 2014;29(12):1692-1701. doi:10.1007/s11606-014-2878-x.
2 Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: A systematic review. BMC Health Services Research. 2014;14:89. doi:10.1186/1472-6963-14-89.
3 Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015:4(2):133-45. doi: 10.2217/cer.14.79.
4 We recently added ‘Measurement’ as a special topic within the Literature Explorer. The current collection of articles was identified with a modified search strategy focused on identifying articles about existing measures and insights related to measuring engagement from literature reviews (published 2010-2020), individual studies or commentaries (published 2018-2020), and articles identified through supplemental reference mining for measure information. A forthcoming update will expand the application of this topic to all publications included in the Literature Explorer.

Posted: June 8, 2018; Updated: July 8, 2022

What's Happening at PCORI?

The Patient-Centered Outcomes Research Institute sends weekly emails about opportunities to apply for funding, newly funded research studies and engagement projects, results of our funded research, webinars, and other new information posted on our site.

Subscribe to PCORI Emails


Hand pointing to email icon