Summary of Program

The Patient-Centered Outcomes Research Institute (PCORI) is reopening this funding initiative to support patient-centered comparative clinical effectiveness research (CER) that addresses important questions regarding strategies to prevent unsafe opioid prescribing in primary care among patients with acute or chronic non-cancer pain. Due to the policy importance and continued evidentiary needs, PCORI leadership remains committed to support high-quality research in this area. When first issued, prevention of unsafe prescribing was identified as an important research gap that was complementary to PCORI’s targeted funding announcement (initially released in October 2015) addressing clinical strategies for managing and reducing long-term opioid use for chronic pain. As before, this current funding announcement excludes patients on chronic opioid therapy. For this solicitation, PCORI is particularly interested in receiving applications that target settings of high need, such as dentist offices, emergency departments, rural areas, and states with high rates of opioid prescribing. PCORI is also particularly interested in receiving applications that compare specific strategies of interest to payers (e.g., pharmacy benefit management strategies such as prior authorization, step therapy, and quantity limits).

Through this PCORI Funding Announcement (PFA), PCORI seeks to fund studies that have a sufficient sample size to address the research questions below and studies that will generate information that is readily generalizable to the broader population.

Competitive applications must address at least one of the two priority research questions described in this PFA.

The two priority research questions are:

• What is the comparative effectiveness of different payer or health-system strategies that aim to prevent unsafe opioid prescribing while ensuring access to non-opioid methods for pain management with the goal of reducing pain and improving patient function and quality-of-life outcomes, while reducing patient harm?
• What is the comparative effectiveness of different patient- and provider-facing interventions that facilitate improved knowledge, communication, and shared decision making about the relative harms and benefits of opioids and alternative treatments on prevention of unsafe prescribing and improved patient outcomes?

Applicants should also:

• Focus on patient populations who are potentially new users of opioids or patients who have used opioids for less than three months, with either acute or chronic pain or both. All chronic pain disorders outside of pain caused by advanced cancer are considered chronic non-cancer pain.
• Have strong endorsement and study participation by relevant patient organizations, professional organizations, and payer or purchaser organizations.
• Take place within typical clinical care and community settings.
• Have a sufficiently large study population to enable precise estimates of effect sizes and to support evaluations of potential differences in intervention effectiveness in patient subgroups. Examples of patient subgroups include those with comorbid mental health disorders or past/present substance abuse; vulnerable populations such as racial and ethnic minorities, children, older adults, people with limited access to health care, or those with limited communication skills; and type of pain—acute versus chronic. 
• Compare the effectiveness[1] of two or more alternatives for improving patient-centered outcomes.

Background

Acute and chronic pain affects a substantial number of Americans and is the leading cause of disability and decreased quality of life. [2]^,[3]^,[4]^,[5] Acute pain, by definition, is of sudden onset and expected to last a short time. [6] It usually can be linked to a specific event, injury, or illness—such as a muscle strain, a severe sunburn, a kidney stone, or pleurisy. [7] Chronic pain, by contrast, lasts more than several months (variously defined as 3–6 months, but longer than “normal healing” time). [8] Chronic pain can be the result of an underlying medical disease or condition, injury, medical treatment, inflammation, or an unknown cause. [9] Estimates of the prevalence of acute and chronic pain range; however, it is evident that the number of people experiencing pain in the United States is considerable. A recent article estimates that approximately 100 million people in the United States suffer from pain annually, and 9 to 12 million of these Americans have chronic to persistent pain, while the remainder suffer from short-term pain due to injuries, illnesses, or medical procedures. [10] The societal costs of pain have been estimated at $560 billion per year from medical expenses, disability, and lost wages and productivity. [11]

Treatment options for pain include over-the-counter and prescription non-opioid analgesics, medications for specific conditions such as neuropathy, and non-pharmacologic options such as cognitive behavioral therapy and physical therapy. In recent decades, using opioids to treat pain has become more prevalent, and their use has increased considerably. Opioids are now the most commonly prescribed class of medications in the United States. [12] In 2012, providers wrote 259 million prescriptions for opioid pain medication, which is enough for every adult in the United States to have a bottle of pills. [13]^,[14] Opioid prescriptions per capita have also increased 7.3 percent from 2007 to 2012, with opioid-prescribing rates increasing faster for internal medicine, general practice, and family practice— compared with other specialties. [15],[16] Most recently, in 2014 U.S. pharmacies distributed 245 million prescriptions for opioid pain relievers, 65 percent of which were for short-term therapy (< 3 weeks); [17] however, 3 to 4 percent of the adult population (9.6 million to 11.5 million) were given a prescription for long-term opioid therapy. [18]^,[19]

Although opioids are widely accepted for treating cancer-related chronic pain, and are also accepted in the palliative care setting, their use for other types of pain remains controversial. [20] The Centers for Disease Control and Prevention (CDC) report that there is evidence to support the short-term efficacy of opioids for reducing pain and improving function in some types of non-cancer pain, but this evidence comes primarily from short-term trials (≤ 12 weeks). [21]^,[22] Few studies exist that address the long-term benefits of opioids for chronic pain. [23]^,[24]

Mounting evidence suggests that opioids may be associated with important harms, including overdose, abuse, addiction, and diversion. [25],[26]^,[27]^,[28]^,[29]^,[30]^,[31] The Drug Abuse Warning Network (DAWN) reports that more than 420,000 emergency department visits were related to the misuse or abuse of narcotic pain relievers in 2011. [32] The widespread use of opioids has also resulted in a national epidemic of opioid overdose deaths and addictions. The CDC reports that in the past decade the death rates for the top leading causes of death—such as heart disease and cancer—have decreased significantly, while the death rate associated with opioid pain medication has increased considerably. [33] More than 165,000 persons in the United States died from opioid-related overdose from 1999 to 2014. [34] During this time, there has also been a corresponding increase in the rate of opioid addiction, affecting approximately 2.5 million adults in 2014. [35]^,[36]

The rising rates of opioid use, abuse, and misuse indicate a need to improve the safe prescribing of opioids in primary care, for both acute and chronic pain patients. Using for acute pain is common and an important predictor of chronic opioid use. [37] Evidence indicates that many patients initiate long-term opioid use by refilling prescriptions that were for short-term pain management (e.g., post-procedure pain or an acute flare-up of an ongoing chronic pain condition). [38] Studies have also shown that having a history of a prescription for an opioid pain medication increases the risk for overdose and opioid use disorder. [39]^,[40]^,[41] In addition, physician descriptions are the major source of diverted opioids. [42]^,[43] It may be difficult for physicians to determine when they are initiating opioids for chronic pain rather than treating acute pain because the line between acute pain and initial chronic pain is not always clear. [44] For these reasons, medical associations have started questioning opioid-prescribing practices. [45]

Recent CDC guidelines recommend that clinicians consider the full range of therapeutic options for treating acute and chronic pain—as well as the risks and harms of opioids—before prescribing opioids. The CDC also recommends using opioids only when alternatives are ineffective. [46] However, access to effective non-opioid alternatives and multimodal therapies is sometimes limited due to lack of insurance reimbursement, and access and cost can be barriers for patients. [47]

Evidence Gap

Although preventing harm from prescription opioids is recognized as an important public health problem, and has been the current target of several federal and statewide initiatives, [48],[49]^,[50]^,[51]^,[52]^,[53] little evidence exists on how to prevent unsafe prescribing of opioids. Notably, most federal initiatives addressing the prescription opioid epidemic call for further research.

Of the multiple recent comprehensive systematic reviews on opioids, [55]^,[56]^,[57]^,[58]^,[59]^,[60] only one addressed strategies for preventing unsafe initiation of opioids. The systematic review conducted to support the American Pain Society (APS) and American Academy of Pain Medicine (AAPM) guidelines on using chronic opioid therapy in chronic non-cancer pain found no systematic reviews, randomized trials, or controlled observational studies addressing the effectiveness of opioid-prescribing policies on clinical outcomes. [61] The lack of evidence is underscored by cross-state comparisons showing that rates of opioid prescribing vary significantly in ways that cannot be explained by the population’s underlying health status, highlighting the lack of consensus among clinicians on how to use opioid pain medication. [62]^,[63]

A number of strategies targeted at providers and patients to promote safe opioid prescribing have been developed, but not rigorously evaluated. [64]^,[65] Pain management decisions are ideally suited for shared decision making, which is a key component of patient-centered health care. [66] Shared decision making is the process in which clinicians and patients work together to make decisions and select treatments and care plans based on clinical evidence that balances risks and expected outcomes with patient preferences and values. [67] It is needed when there is more than one reasonable treatment option, when no one option has a clear advantage, when the possible benefits and harms of each option affect patients differently, or when the evidence is uncertain. [68] This is often the case for decisions about pain. Studies have shown that shared decision making improves patients’ satisfaction with—and involvement in—health care. [69]^,[70] Additional studies have shown that shared decision making with decision aids improved patient satisfaction; improved knowledge; increased risk perception; decreased decisional conflict; and reduced the prevalence of invasive procedures when patients were educated on all the treatment options available to them, along with the risks and benefits of each option. [71]^,[72]^,[73] Strategies that have proven successful in analgesic management of chronic pain patients or in reducing the risk of opioid misuse for chronic pain might be useful to promote safer initiation of opioids, but have not been tested with that aim. [74]^,[75]

Health systems have been experimenting with a wide range of strategies to prevent opioid-related harms. Notably, although the primary aim of these strategies was not the prevention of unsafe prescribing (some aimed to improve pain management, others aimed to reduce dosage in chronic opioid users), their effects suggest that such strategies may be relevant to the goals of this funding announcement. The Department of Veterans Affairs (VA) has implemented its Overdose Education and Naloxone Distribution Program, Opioid Safety Initiative, and other policy changes that have effectively reduced the number of opioid prescriptions and deaths. [76] These programs include components of provider profiling; dissemination of prescribing guidelines; educational programs for providers and patients; and expansion of access to non-opioid alternatives, such as complementary and alternative medicine. Some of these initiatives—including Coordinated-Stepped Care with Protocol-Driven Analgesic Ladder [77] and telecare collaborative management approach [78]—have been rigorously evaluated, but have not been tested outside of the VA. Integrated health systems, such as Kaiser Permanente and Group Health, have adopted similar programs. [79],[80] Robust evaluation is often lacking, as well as understanding the impact of these programs on patient pain management and other relevant patient outcomes. 

Like health systems, payers, often in partnership with states, have been implementing strategies to address the opioid epidemic and improve pain management. In addition to mirroring some of the strategies being used by health systems, payers are using reimbursement policies or incentives to expand access to non-pharmacological interventions for pain and promote guideline concordant care. Washington has launched a particularly comprehensive program. [81] Oregon is expanding benefits to non-pharmacologic alternatives to opioids for patients with lower-back pain, with the goal of slowing the rate of opioid prescribing. [82] Other strategies include changing preferred drug list placement; changing policies about step therapy or prior authorization (such as requiring informed consent for patients considering opioid treatment); introducing clinical criteria for prescriptions; and using quality metrics related to opioid prescribing, among others. [83]^,[84]^,[85]^,[86] A total of 26 states responded to a recent survey about strategies being used to improve safe opioid prescribing and expand access to non-opioid methods for pain management in advance of an April 2016 meeting of the Medicaid Medical Directors Network. The results [87] illustrated wide variation in approaches, presenting opportunities for natural experiments to support CER. Participants noted there was a lack of evidence to help guide them to select the most effective strategies, and that further robust CER would be essential to design more impactful programs.

To ensure that primary-care providers and patients consider safer and more effective treatment; improve outcomes (such as reduced pain and improved function); and reduce the number of persons who develop opioid use disorder, overdose, and other adverse events—additional rigorous CER is needed to identify strategies that prevent unsafe opioid prescribing among patients with acute and chronic non-cancer pain.

Research Topic Prioritization

PCORI relies on input from multiple stakeholders to set its research priorities. On March 7, 2016, PCORI convened a large multi-stakeholder workshop to provide input on whether PCORI-funded research could address specific CER questions on strategies to prevent unsafe opioid use. The aim was to complement an earlier funding announcement that targeted clinical management of people on chronic opioid therapy (longer than three months). This workshop focused on the decisional dilemmas about whether to initiate opioids in the first place or whether to continue opioid use after the initial acute prescription (potential new users or patients on opioids for less than three months). Approximately 66 invited stakeholders attended in person. The meeting was open to the public via teleconference and webinar. Background materials and more information about the workshop, including a meeting summary, can be found on the PCORI website.

Based on the key gaps noted above, input from our large multi-stakeholder workshop, and the PCORI Board of Governors (Board) Scientific Oversight Committee, PCORI is interested in applications that respond to the following comparative effectiveness questions.

The two priority research questions are:

• What is the comparative effectiveness of different payer or health-system strategies that aim to prevent unsafe opioid prescribing while ensuring access to non-opioid methods for pain management with the goal of reducing pain and improving patient function and quality-of-life outcomes, while reducing patient harm?
• What is the comparative effectiveness of different patient- and provider-facing interventions that facilitate improved knowledge, communication, and shared decision making about the relative harms and benefits of opioids and alternative treatments on prevention of unsafe prescribing and improved patient outcomes?

Priority Research Questions

For both questions, applications should target potential new users, of opioids or patients who have used opioids for less than three months with either acute or chronic pain. Applicants need to further define and justify the selected study cohort. The research may focus on either acute or chronic pain or both, if there is adequate attention to subgroups. Patients must be outside of pain due to advanced cancer care. The target setting is primary care, broadly defined to include primary-care practices, emergency departments, dentist offices, or urgent care centers. Applicants should further delineate and justify the setting for the proposed research within this broad area of primary care.

Applications for both questions must include interventions intended to prevent unsafe prescribing, while also ensuring adequate or improved pain management. Unsafe prescribing will need to be described in the context of the proposed study cohort and applicants must provide a strong rationale for how their intervention would avoid potential harms in opioid prescribing. Examples include reducing opioid use as first-line therapy for treating neuropathic pain or fibromyalgia [88], mechanisms to reduce instances of co-prescribing of benzodiazapines with opioids, [89] or using extended-release/long-acting opioids for acute pain. [90] Guideline-concordant care (e.g., the CDC [91] guideline for prescribing opioids to treat chronic pain, the American College of Emergency Physicians guideline for prescribing in emergency departments [92], and the Pennsylvania Guidelines on Use of Opioids in Dental Practice [93] may serve as a benchmark for safe prescribing. Applicants will need to justify their selection of a specific guideline, and then discuss the strength of the evidence underlying their proposed measures for preventing unsafe prescribing.

Strategies selected for comparison for both questions must be either evidence-based or in widespread use (i.e., in use by a number of health systems or providers). Interventions that have evidence for similar situations but have not been used for reducing unsafe opioid prescribing may be used (with adaptation) with sufficient rationale, and if the majority of the time and budget are aimed at establishing comparative effectiveness rather than developing and validating the intervention. PCORI encourages research using active comparators, if feasible. However, given the rapidly changing policy and practice environment for opioid prescribing, a “usual care” arm may be justifiable. “Usual care” must be adequately defined and mechanisms put in place to characterize and measure changes in “usual care” over the course of the study.

What is the comparative effectiveness of different payer or health-system strategies that aim to prevent unsafe opioid prescribing while ensuring access to non-opioid methods for pain management with the goal of reducing pain and improving patient function and quality-of-life outcomes, while reducing patient harm?

Applications should propose pragmatic cluster randomized controlled trials (RCTs) or well-designed, prospective observational studies to address this research question. For randomized trials, the cluster design is likely appropriate because of the anticipated system-wide nature of the organizational strategies likely to be evaluated. Pragmatic trials are designed to maximize applicability of the study’s results in routine clinical practice. They are conducted in routine clinical care settings, and in many cases they must be relatively large to be able to demonstrate differences in comparative effectiveness between different patient subgroups. They should impose fewer constraints on usual practice than traditional RCTs. Investigators are also encouraged to take advantage of natural experiments due to changes in opioid-prescribing policies and pain management strategies with the use of quasi-experimental study designs using concurrent controls or interrupted time-series designs. Proposals should not be solely retrospective, or solely based on cross-sectional studies, or simple pre-post designs without an appropriate comparison. Because the interventions are likely to be complex, multi-modal, multi-level strategies that target both clinicians and patients, PCORI encourages designs that use mixed methods. A mixed-methods approach would include a quantitative and qualitative component, with the qualitative component addressing the contextual factors affecting implementation outcomes.

PCORI also encourages research that spans across more than one healthcare system, and is conducted outside of VA facilities or integrated delivery systems that have already implemented stringent policies to address unsafe opioid prescribing. Comparisons between healthcare systems (e.g., different state Medicaid programs, different healthcare delivery systems, and between the VA and other public or private healthcare systems) are permissible and encouraged. Strategies being compared may include combinations of prescriber monitoring and feedback; case management; structured clinical assessments; physician and patient education; standing orders; electronic health record (EHR) linkages to Prescription Drug Monitoring Programs; payer incentives or policy tools, such as changes in formulary placement, co-payments, or prior authorization; and expanded access to evidence-based non-opioid alternatives. Other strategies may be considered if well-justified. Applicants should provide a convincing explanation for the relevance of the strategies being compared in the proposed study. Applications should include information about the efficacy of the interventions compared, pilot data showing promise, or define what they mean by established programs in widespread practice. Given the dearth of evidence on strategies to reduce unsafe prescribing of opioids, evidence from interventions aiming to improve pain management—but having the effect of improving prescribing practice—or that focused on chronic opioid users—but may also be effective strategies for reducing unsafe prescribing for potential new users—may be acceptable. Evaluation of brand new, untested interventions is not responsive.

PCORI is interested in studies that target or evaluate subgroup populations at high risk for harm or misuse from unsafe opioid prescribing, such as youth, where using opioid medication before high school graduation is associated with more than a 30 percent increase in the likelihood of abuse [94], and misusing prescription opioids in adolescence strongly predicts future heroin use. [95] Trends in dependency and poisoning from narcotics among youth have risen sharply in recent years. [96] Other examples include persons with a history of substance use disorder, or persons with major depression. [97] Other key patient subgroups include groupings based on the source or type of pain.

What is the comparative effectiveness of different patient- and provider-facing interventions that facilitate improved knowledge, communication, or shared decision making about the relative harms and benefits of opioids and alternative treatments on prevention of unsafe prescribing and improved patient outcomes?

To address this research question, applicants should propose a pragmatic RCT or a cluster RCT. Strategies being compared may include combinations of patient-focused and provider-focused interventions such as motivational interviewing; web-based, e-health programs providing education, training and self-management support; physician communication skills and training; and patient coaching. Applicants should provide a convincing explanation for the relevance of the strategies being compared in the proposed study. Applications should include information about the efficacy of the interventions compared. Similar to applications that address the first question, evidence from interventions aiming to improve pain management—but having the effect of improving prescribing behavior—or that focused on chronic opioid users—but may also be effective strategies for new users—may be acceptable. PCORI expects the efficacy or effectiveness of each intervention to be known. Interventions that have documented efficacy or effectiveness in similar situations may be used (with adaptation if necessary) if the efficacy is well-documented (e.g., with multiple trials or with a systematic review) and sufficiently strong rationale for why the intervention would be expected to be efficacious in the proposed new setting(s) or population(s) is provided. If an intervention is to be adapted, PCORI expects the majority of the proposed time and budget to be aimed at establishing comparative effectiveness rather than the adaptation and validation of the interventions. PCORI is interested in studies that target or evaluate subgroup populations at high risk for harm from unsafe opioid prescribing, such as adolescents, persons with a history of substance use disorder, pregnant women, older adults, and persons with low health literacy and numeracy. Other key patient subgroups include groupings based on the pain source or type.

Outcomes

Applicants should consider a broad range of outcomes that are important to patients for both of the priority questions.

Priority question 1: In addition to measuring the reduction in unsafe prescribing (as defined in the particular project), studies should include patient-centered primary outcomes such as pain, quality-of-life, and functional outcomes. Examples of relevant secondary outcomes include opioid initiation; opioid overdose; anxiety or depression; sleep disturbance; disability; harms (e.g., addiction and opioid use disorder, overdose, and death); provider satisfaction; provider self-efficacy; and emergency department use. Applicants should consider a minimum one-year follow-up for main outcome measures.

Priority question 2: Studies should include primary outcomes such as patient and provider knowledge, patient anxiety (from potential health outcomes), quality of life (including pain control), functional outcomes, rate of opioid initiation, reduction in unsafe prescribing, and repeat opioid prescriptions. Examples of relevant secondary outcomes include patient satisfaction; patient decisional regret; patient involvement preference; harms (e.g., tolerance, dependence, addiction or opioid use disorder, overdose, and death); provider satisfaction; length of visit; and provider confidence and self-efficacy. Applicants should consider a minimum one-year follow-up for main outcome measures.

Funds Available

PCORI seeks to fund studies addressing each of the two priority questions, but does not commit to such. PCORI will consider the merit of each submitted application and its responsiveness to each priority question, as well as programmatic requirements and portfolio balance, when making final funding recommendations.

PCORI has devoted up to $20 million in total costs under this PFA to fund high-impact studies related to prevention of unsafe opioid prescribing in primary care among patients with acute or chronic non-cancer pain. The proposed budget for individual studies may range up to $5 million in total costs as appropriate, depending on the specific priority research question or questions the study proposes to address. The maximum project period is three years.

Given the significant treatment costs associated with many opioid and non-opioid therapies (including non-pharmacological options), the applications must specifically address—in the context of the proposed studies—the support from public or private payers, health plans, industry sponsors, or others in covering the study interventions and non-study protocol-related clinical costs and services rendered in the care processes. Please refer to Appendix 2 in the Application Guidelines for details on costs that PCORI will cover. PCORI seeks efficient studies and the supportive involvement of delivery systems or health plans to enhance recruitment, data collection, and coverage of treatment-related costs.

Back to top

II. Guidance for Preparing Applications

Specific Requirements

The proposed study should strive to meet the following requirements:

• Focus on a comparative effectiveness question that is important to patients and other decision makers.
• Address a research gap that has been substantiated by an existing (recent or updated), rigorously conducted systematic review or emphasized by an official professional society’s clinical practice guideline.
• Demonstrate consultation with patients and other stakeholders or their representative groups, or reference previously documented decisional dilemmas to determine if the study is answering a critical question—one that, if adequately answered, would substantially improve decision making.
• Receive endorsement by relevant patient organizations, clinician organizations, payer or purchaser consortia, and life-sciences industry representatives as potentially answering a critical question—one that, if adequately answered, would substantially improve decision making.
• Propose a sample size that is sufficiently large to allow for precise estimation of hypothesized effect sizes or for clear demonstration of non-inferiority. The sample size must also support testing of a priori hypotheses related to potential differences in effectiveness among relevant patient subgroups (Heterogeneity of Treatment Effect, or HTE).
• Examine diverse populations receiving care in real-world settings.
• Have strong interest from and support of host delivery systems and clinical care settings.
• Specify broad and simple eligibility criteria that will allow wide generalization of results, while attending appropriately to ethical concerns of excess risk in some patient subgroups.
• As applicable, compare interventions that are known to be efficacious, effective, or commonly used and that can be implemented in real-world settings.
• Include patient-reported outcomes (PROs) as primary outcomes, when appropriate.
• For RCTs, provide preliminary evidence of the potential for efficient recruitment, high participation rates, and appropriate oversight by local or centralized Institutional Review Boards (IRBs), including plans for streamlining or waiving individual informed consent in cases of low-risk interventions. PCORI believes that the intensity of oversight and the complexity of informed consent procedures should be closely related to the degree of risk from study participation. Applicants must address this issue and present evidence that the study will not encounter significant barriers to recruitment or participation. The relevant IRBs make the final determination of the adequacy of informed consent procedures and participant protections.

For observational studies, employ rigorous designs that can address concerns related to causal inferences about the relative effectiveness of different strategies on patient-centered outcomes. Investigators should carefully consider the appropriate observational design for their question, which may include opportunistic, natural experiments or other types of retrospective and prospective observational research approaches that make use of longitudinal, quasi-experimental study designs using concurrent control conditions, historical trends or interrupted time-series designs. PCORI encourages mixed-methods designs so that interventions can be assessed at multiple levels where appropriate (e.g., system-wide, communities, health organizations, and practice or clinician level). However, applications will need to make a clear conceptual and analytical connection between interventions, comparators, and patient-centered outcomes. Cohort and panel survey data may also be employed, but should permit examination of study interventions that are applied and targeted at large segments of the population by one or more health system(s). Proposals using cross-sectional or simple pre-post designs are discouraged.

Adhere to all applicable PCORI Methodology Standards. The full application will require the applicant to identify the standards appropriate to the proposed study and describe how the study team plans to address each standard.

In the case of RCTs, also adhere to current best practices (standardized inclusion or exclusion criteria; proper randomization; techniques to minimize potential for missing data; and appropriate safety monitoring, including establishing a Data and Safety Monitoring Board [DSMB] or indicating why such a board is unnecessary).

To carry out pragmatic studies, readily adopt the findings in a real-world setting, and maximize the efficient use of resources, applicants must prevent these trials from becoming more complex and onerous than necessary. PCORI encourages the applicant to be creative and consider the following innovative strategies, as appropriate and feasible:

• Be prepared to identify and engage with major patient and stakeholder organizations that would implement study findings, as well as with existing local communities of patients and care providers to refine the research questions and study protocol, help monitor progress, and disseminate the findings.
• Consult with patients and other stakeholders on their decisional dilemma and evidence needs, or reference previously documented decisional dilemmas in preparation for submitting Letters of Intent (LOIs) and the full applications.
• Describe carefully the pertinent evidence gaps and why the project questions represent decisional dilemmas for patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders. Similarly, applicants should document why project outcomes are especially relevant and meaningful endpoints for patients and other stakeholders. Minimize disruption to participants’ daily routines (e.g., minimize participant visits intended for study-assessment purposes; capture PROs during office visits, electronically, or by phone).
• Design the study so that the conduct can integrate with routine clinic or office operations as seamlessly as possible.
• Use efficient methods to obtain participant consent while still meeting ethical and legal requirements.
• Capitalize on the existing EHRs and other computerized information to identify and recruit eligible patients, monitor study conduct and patient safety, and collect study outcomes information. PCORI specifically encourages applications that use the National Patient-Centered Clinical Research Network (PCORNet) infrastructure.
• If data standardization and interoperability across study sites has not already been accomplished, develop methods that will enhance the standardization of data that are accessed from different EHR systems.

Non-responsiveness

Applications will be considered nonresponsive to this PFA if the proposed research:

• Tests efficacy (or comparative efficacy) within a tight, protocol-controlled research setting (as opposed to more real-world and pragmatic CER)
• Conducts a cost-effectiveness analysis in the form of dollar-cost per quality-adjusted life-year to compare two or more alternatives
• Directly compares the care costs between two or more alternative approaches
• Measures the relative care costs of two or more alternative approaches as the primary criteria for choosing the preferred alternative
• Conducts studies of the natural history of disease, instrument development, pharmacodynamics, and fundamental science of biological mechanisms
• Evaluates validity or efficacy of (rather than the comparative effectiveness of) new or existing decision-support tools; this includes the development and efficacy evaluation of decision-support or shared-decision tools or systems for patients, clinicians, or both
• Develops clinical prediction or prognostication tools

Applications that include studies of these issues may measure and report utilization of any or all health services, but may not employ direct measurements of care costs.

PCORI does have an interest, however, in studying conditions that lead to high costs to the individual or to society. Thus, PCORI is also interested in studies that:

• Address cost-related issues, such as the resources needed to replicate or disseminate a successful intervention
• Evaluate interventions to reduce health-system waste or increase health-system efficiency

Applications that include studies of these issues without using cost-effectiveness analyses or comparing the costs of alternatives are considered responsive.

Furthermore, PCORI discourages applications in the following categories and is likely to deem them nonresponsive:

• Study of the natural history of disease
• Instrument development
• Pharmacodynamics
• Fundamental science or study of biological mechanisms
• Establishing efficacy for a new clinical strategy
• Pilot studies intended to inform larger efforts
• Comparisons of patient characteristics rather than clinical strategy options

Features of Patient-Centered Outcomes Research

Patient-centered outcomes research (PCOR) helps people and their caregivers communicate and make informed healthcare decisions, allowing their voices to be heard in assessing the value of healthcare options. This research:

• Assesses the benefits and harms of preventive, diagnostic, therapeutic, and safe opioid prescribing to inform decision making, highlighting the choices that matter to people
• Is inclusive of an individual’s preferences, autonomy and needs, focusing on outcomes that people notice and care about (including survival, functioning, symptoms, and health-related quality of life)
• Incorporates a wide variety of settings and diversity of participants to address individual differences and barriers to implementation and dissemination
• Directly compares clinical interventions that are generally available in the clinical settings
• Obtains stakeholder perspectives to address the burdens to individuals, availability of services, and technology and personnel requirements

Leveraging Existing Resources

PCORI encourages investigators to propose studies that leverage existing resources, such as adding PCOR to an existing large clinical trial or analyzing existing large databases that contain valuable and relevant information that can be used to answer important CER questions. PCORI is interested in studies that leverage existing research networks or consortia that would facilitate the conduct of large, multi-site studies called for in this PFA.

Applicants proposing use of an existing research network infrastructure (e.g., PCORnet); research consortia; or related data resources (e.g., electronic medical record [EMR] data from healthcare delivery systems or administrative claims data from public or commercial insurers) should address the following in the Research Plan (as appropriate), with sufficient specificity:

• Identify and justify all participating research network entities (e.g., health plans, consortia projects and members, etc.) or, in the case of PCORnet, identify the names of the participating Clinical Data Research Networks (CDRNs), Patient-Powered Research Networks (PPRNs), and PCORnet Collaborative Research Groups) that will be collaborating on the project. Also identify the affiliated study performance sites.  
• Demonstrate that the data source can comprehensively capture the study variables needed to assess the interventions, covariates, and outcomes.
• Describe how you will manage data across study sites within the research network or the proposed research consortia, and whether you will use any dedicated data-coordinating functions or facilities.
• As applicable, demonstrate familiarity with the existing network governance policies or data-use restrictions. Provide a study management structure that identifies roles, responsibilities, and decision-making authority across the proposed research consortia.
• As applicable, provide a timeline for establishing data-use agreements.
• As applicable, describe the network infrastructure resource(s) used to conduct the study (i.e., core research-support facilities, streamlined IRBs, contracting, engagement and consenting processes, standardized data resources training, etc.). Indicate the percentage of sites that have previously used centralized versus localized IRBs.
• As applicable, provide documentation supporting the involvement of network leadership throughout the study (e.g., detailed Letters of Support, budgets, and Budget Justifications that cover the costs of the network’s efforts). You can obtain Letters of Support from PCORnet from the Coordinating Center by submitting a request via the PCORnet Front Door.

Preliminary Data and Use of Accepted Measures

PCORI encourages investigators to design their research using valid patient-centered outcomes measures. Include preliminary data that support using the proposed measures. We encourage investigators to consider those measures described in the Patient-Reported Outcomes Measurement Information System (PROMIS). [99]

Methodological Considerations

Regardless of study design, applications must adhere to all relevant PCORI Methodology Standards. These include 48 individual standards that fall into 12 categories. The first five categories are cross-cutting and are relevant to most PCOR studies. Researchers should refer to all of these standards when planning and conducting their research projects. These cross-cutting categories are:

1. Standards for Formulating Research Questions

2. Standards Associated with Patient-Centeredness

3. Standards on Data Integrity and Rigorous Analyses

4. Standards for Preventing and Handling Missing Data

5. Standards for Heterogeneity of Treatment Effect (HTE)

In addition to these five sets of standards, the first standard of “Standards for Causal Inference Methods” -(CI-1)- is cross-cutting and applicable to all PCOR studies.

The seven other standards categories will be applicable to particular study designs and methods. Applicants should use the standards in each of these categories as guidance when they are relevant to a study. These categories are:

1. Standards for Data Registries

2. Standards for Data Networks as Research-Facilitating Infrastructures

3. Standards for Causal Inference Methods

4. Standards for Adaptive and Bayesian Trial Designs

5. Standards for Studies of Medical Tests 

6. Standards for Systematic Reviews

7. Standards for Research Designs Using Clusters

Most of these standards are minimal. The PCORI Methodology Standards reflect practices that applicants should follow in all cases, and all deviations need to be explained and justified. Applicants should address additional best practices—including relevant guidelines for conducting clinical trials developed by other organizations—in the application for PCORI funding. To help reviewers quickly identify adherence to a particular standard, applicants must cite each relevant PCORI Methodology Standard within the Methodology Standards Checklist, following the instruction in the checklist itself and in the Application Guidelines. Program staff use the checklist to evaluate applications.

Applicants should specifically discuss their capacity to measure such factors as differential adherence to chosen treatments (or participation in intervention programs) that could create or explain apparent differences in the effectiveness of the alternative interventions being compared in clinical populations.

Patient and Stakeholder Engagement

PCORI encourages all applicants to outline how patients and other stakeholders will participate as partners in various phases of the proposed research. Before completing this section of the Research Strategy, applicants are encouraged to review PCORI’s Engagement Rubric, which can be found in the PCORI Funding Center. Applicants should also review the PCORI Methodology Standards Associated with Patient-Centeredness, and PCORI’s Sample Engagement Plans. The rubric and Sample Engagement Plans are not intended to be comprehensive or prescriptive; instead, they provide a variety of examples to incorporate engagement, where relevant, into the research process.

Applicants are expected to consult with patients and other stakeholders on their decisional dilemma and evidence needs, or to reference previously documented decisional dilemmas in preparation for the submission of LOIs and applications. To describe the decisional dilemma, state the specific clinical decision(s) or treatment choice(s) confronted by the decision makers and explain how the findings from the proposed research will inform those decisions. State why this decision—such as choosing a specific medication, surgical approach, or care delivery strategy to treat a condition or manage a specific population—is important to patients. Document the uncertainty patients and other stakeholders face in making this decision. Identify the patients and other stakeholders you consulted in determining that the proposed study addresses their evidentiary needs for decision making, and indicate your commitment to continue engaging them actively in the conduct of the study. Similarly, applicants should document how the project outcomes are especially relevant and meaningful endpoints to patients and other stakeholders.

For this funding announcement, applicants are not required to demonstrate that patients and other stakeholders are already engaged as research team members at the time an application is submitted. However, the Engagement Plan should outline how patients and other stakeholders will participate as partners in various phases of the proposed research, once awarded. Applicants should describe their plan to form a Study Advisory Committee (SAC) or other appropriate engagement body, to ensure that a broad spectrum of patients and other stakeholders advise and assist the research team with refining the study questions, outcomes, and protocols. These patients and other stakeholders must include national or regional organizations that represent—at a minimum—patients, caregivers, clinicians, policy makers, and other healthcare system stakeholders. Additional representation may be recommended in collaboration with PCORI, including individual patients with lived experience and other relevant stakeholders, such as scientific and methodological experts. The SAC or other appropriate engagement body should meet in person at least two times per year, and the budget should account for these engagement costs.

PCORI understands that engagement structures and approaches vary widely. Other engagement approaches, such as forming stakeholder groups, panels, task forces, working groups, and other bodies, or involving individual patient and other stakeholder partners in various ways, are also permissible to employ—either in addition to or instead of—the formation of the SAC. For clarification in your application materials and for merit review purposes, please indicate which body or structure is filling the SAC requirements, including the requirements for in-person meetings at least two times per year, and appropriate budgeting.

Populations Studied

PCORI seeks to fund research that includes diverse populations with respect to age, gender, race, ethnicity, geography or clinical status, so that possible differences in CER may be examined (otherwise known as HTE). PCORI recognizes that some proposed studies might represent important PCOR opportunities, even in the absence of a broadly diverse study population. However, the burden is on the applicant in such cases to justify the study’s importance in the absence of diversity. The applicant must also discuss which subgroups are most important and how they will be analyzed, including whether there will be power to examine the question of effectiveness in subgroups. PCORI is particularly interested in including previously understudied populations for whom effectiveness information is especially needed, such as hard-to-reach populations or patients with multiple conditions. Thus, comparisons should examine the impact of the strategies in various subpopulations, with attention to the possibility that the strategy’s effects might differ across subpopulations. PCORI has developed the following list of priority populations to guide our research and engagement efforts:

• Racial and ethnic minority groups
• Low-income groups
• Women
• Children (age 0–17 years)
• Older adults (age 65 years and older)
• Residents of rural areas
• Individuals with special healthcare needs, including individuals with disabilities
• Individuals with multiple chronic diseases
• Individuals with rare diseases
• Individuals whose genetic makeup affects their medical outcomes 
• Patients with low health literacy, numeracy, or limited English proficiency
• Lesbian, gay, bisexual, transgender, and questioning (LGBTQ) persons
• Veterans and members of the Armed Forces and their families

Project Budget and Duration

Applicants may request up to $5 million in total costs for a project period not to exceed three years. Given the public health urgency of generating strong evidence to guide strategies that prevent unsafe opioid prescribing, studies of shorter duration—or those with planned release of interim findings—are encouraged. At the time of contract execution, PCORI sets aside all of the funds associated with an awarded project to be made available throughout the contract’s period of performance. Note that, in general, PCORI will not cover costs for interventions that are being compared in the proposed study. (See Appendix 2 in the Application Guidelines for details.)

Applicants should submit realistic budgets and timelines. For those rare circumstances in which the estimated total costs exceed $5 million, please provide a detailed justification in your LOI that ties the extra expense to the project’s success. Not all requests for additional funds will be approved. Any request for a project period longer than three years will be denied. For further information regarding PCORI’s policies about allowable and unallowable costs, refer to Appendix 2 of the Application Guidelines. Note that although subcontractor indirect costs are included in the prime applicant’s direct-cost budget, subcontractor indirect costs are not factored when determining adherence to the PFA’s direct-cost limit.

The funding mechanism for this program is a contract. Total project funding is contingent upon successful programmatic and budget performance (e.g., meeting recruitment targets). Milestones and targets, as well as possible pilot phases for the sole purpose of assessing recruitment feasibility, should be included in the budget and will be negotiated at the time of the award. Awardees will be expected to provide corroborating evidence to receive continual funding support. Some of the activities that will be considered during negotiations include:

• Developing a study protocol and procedure manual for the intervention
• Assigning roles and responsibilities to study team members for implementing the project 
• Forming an SAC or other appropriate engagement body 
• Obtaining clearances from all institutional and community partners, including IRB approvals
• Establishing a DSMB, or providing a clear description of why a DSMB is not necessary
• Executing all subcontractor agreements
• Agreeing on eligible patient populations for study recruitment
• Identifying barriers to patient recruitment in the study, and addressing these barriers effectively
• Demonstrating successful recruitment during a pilot phase (if indicated)

Refer to the Application Guidelines for a list of additional PFA-specific project milestones.

Collaboration

PCORI is particularly interested in applications that involve community and commercial organizations that can help researchers design, implement, disseminate, and sustain effective interventions. We encourage applications that include novel collaborations with accreditation organizations, credentialing bodies, educational enterprises, patient advocacy groups, industry, professional societies, and subspecialty societies.

Protection of Human Subjects  

This component (up to five pages) is included in the Research Plan Template. Describe the protection of human subjects involved in your proposed research. PCORI follows the Federal Policy for the Protection of Human Subjects (45 CFR part 46), including the Common Rule. For more detailed information, please see Section 5, titled “Human Subjects Research Policy,” in the Supplemental Grant Application Instructions for All Competing Applications and Progress Reports, [103] which is issued by the U.S. Department of Health and Human Services. PCORI does not require that applicants comply with sections of this policy that refer to requirements for federal-wide assurance or that refer to standards for including women, minorities, and children. Awardees must also comply with appropriate state, local, and institutional regulations and guidelines pertaining to the use of human subjects in research.

PCORI requires awardees to ensure that there is a Data and Safety Monitoring Plan (DSMP), which may include the need to appoint a DSMB, as provided in the PCORI Policy on Data and Safety Monitoring Plans for PCORI-Funded Research.

PCORI merit reviewers will examine plans for protection of human subjects in all applications and may provide comments regarding the plans (see How To Evaluate Human Subjects Protections). Reviewers’ comments on human subject research are not reflected in the overall application score, but PCORI staff might use them during potential funding negotiations. Final determinations about the adequacy of human subject protections rest with the IRB or international equivalent that have jurisdiction for the study.

The Awardee Institution, whether domestic or foreign, bears ultimate responsibility for safeguarding the rights and welfare of human subjects in PCORI-supported activities.

Required Education of Key Personnel on the Protection of Human Subject Participants

PCORI requires all applicants to adhere to the National Institutes of Health (NIH) policy on education in the protection of human subject participants in the conduct of research. This applies to all individuals listed as key personnel in the application. The policy and FAQs are available on the NIH website. [106]

Data Management and Data-Sharing Plan

PCORI encourages openness in research and making research data available for purposes of replication and reproducibility. Although not required to be submitted as a component of the research application, if an award is made, the awardee is required to develop and maintain a plan that addresses data management and data sharing of research project data in a manner that is appropriate for the nature of the research project and the types of research project data, and that is consistent with applicable privacy, confidentiality, and other legal requirements.

Peer Review and Release of Research Findings

PCORI has a legislative mandate to ensure the scientific integrity of the primary research it supports and to make study findings widely available and useful to patients, clinicians, and the general public within a specific timeframe. Accordingly, the Board adopted the Process for Peer Review of Primary Research and Public Release of Research Findings.

In summary, Awardee Institutions are required to submit to PCORI for peer review a draft final research report that provides the methodological details, describes the main study results, and interprets the findings in clinical or other decisional contexts. Subject matter experts; individuals with expertise in research methodology or biostatistics; and patients, caregivers, and other healthcare stakeholders will review the draft final research report. After Awardee Institutions have responded to reviewers’ comments to PCORI’s satisfaction, the report will be accepted and considered final. PCORI will then prepare a 500-word abstract summarizing the study results for patients and the general public, which the Awardee Institution will review and approve.

PCORI will post the following materials on its website no later than 90 days after the draft final research report is accepted: (1) a 500-word abstract for medical professionals; (2) a standardized summary of the study results for patients and the general public; (3) a link to the study record on ClinicalTrials.gov (as applicable); and (4) ancillary information, including conflict of interest disclosures. The final research report, along with anonymized reviewer comments, will be made publicly available on the PCORI website no later than 12 months after its acceptance, except by prior mutual agreement with the Awardee Institution.

Back to top

III. How To Submit an Application

Letter of Intent

Applicants should download the Cycle 2 2017 Strategies to Prevent Unsafe Opioid Prescribing LOI Template from the PCORI Funding Center. They must complete the document and convert it to a PDF with a three-page limit, excluding references. PCORI suggests including all references as in-text citations using American Medical Association citation style, but other citation styles are accepted. Do not upload additional documents as part of your LOI, such as Letters of Endorsement or Support, because they are not requested at this stage. Their inclusion will result in LOI rejection without review. Please visit the PCORI Funding Center for additional applicant resources, including the PFA and required templates.

Please answer all of the questions in the LOI Template. This includes the question on brief justification for the proposed cost of the study. Providing the answer, “costs not to exceed $5 million” is not sufficient. Upload your document to PCORI Online. The deadline for LOI submission is October 2, 2017, by 5 p.m. (ET).

Letter of Intent Review

LOIs are evaluated based on the following criteria:

• Whether the proposed topic addresses the priority research question identified in this funding announcement
• Importance of the specific research question (comparison), as evidenced by critical gaps identified by clinical guidelines developers or recent relevant systematic reviews
• A size or scope sufficient enough to have a significant impact on patient outcomes or healthcare practices
• Clarity and credibility of the applicants’ responses to the LOI questions, as well as their justification of the proposed study size, citing published estimates, including effect sizes, standard deviations and the need for rigorous comparative analysis of important subgroups
• Prior relevant experience
• Programmatic fit and balance, considering whether the research study question and study design are compliant with requirements in this funding announcement
• Adherence to the administrative and formatting requirements listed in the Application Guidelines, specifically the four-page limit for the LOI

Only applicants whose LOIs are deemed most responsive to this PFA will be invited to submit a full application. Notification of denial or approval to submit an application will occur no later than October 30, 2017. Please refer to the Application Guidelines for information on how to submit your LOI via PCORI Online.

You are invited to submit an application based on the information provided in the LOI. Any changes to the following require PCORI’s approval:

• Research question(s)
• Specific aims
• Study design
• Comparators
• Principal Investigator (PI)
• Institution

If you need to change any of this information or have any questions, please email [email protected].

All applicants, including those resubmitting from the previous Strategies to Prevent Opioid Prescribing PFA cycle, are required to submit an LOI for PCORI staff to review. This allows PCORI to determine whether proposed revisions and changes made to specific aims or methodological approaches from the original applications align with our evolving strategic priorities.

Note: A PI can only submit one LOI per PFA. However, an individual listed as a PI on one LOI may be listed as and serve in another non-PI role (e.g., co-investigator or consultant) on other LOIs within the same PFA during the same cycle. A PI may submit multiple LOIs to different program PFAs in a cycle, but the PI must ensure that the research topics and projects are not similar. If a PI submits an LOI to multiple program PFAs, LOIs that exhibit scientific overlap or that appear to be duplicate submissions will be disqualified. PCORI will contact the PI and provide him or her with an opportunity to choose which PFA he or she would like to apply to. This applies to single and dual-PI submissions.

Submission Dates

LOIs and applications must be submitted in accordance with the published dates and times listed in the Overview section of this document and in the PCORI Funding Center.

PCORI Online System

To submit an application, you must register with PCORI Online and submit an LOI and an application for each cycle in which you are applying.

Back to top

IV. Applicant Resources

• Unsafe Opioid Prescribing PFA (Cycle 2 2017)
• Unsafe Opioid Prescribing PFA Application Guidelines (Cycle 2 2017)
• Unsafe Opioid Prescribing FAQs (Cycle 2 2017)

Back to top 

V. Merit Review

PCORI’s merit review process is designed to support the following goals:

• Identify applications that have the strongest potential to help patients, caregivers, clinicians, policymakers and other healthcare system stakeholders make informed decisions to improve patient outcomes.
• Implement a transparent, fair, objective, and consistent process to identify these applications.
• Elicit high-quality feedback that reflects a diversity of perspectives to ensure that the PCORI-funded research reflects the interests and views of patients and other stakeholders and those who care for them, and that it meets the criteria for scientific rigor.
• Fund projects that fill important evidence gaps and have strong implementation potential.
• Regularly evaluate and continually improve the merit review process and policies in support of PCORI's mission.

PCORI merit review is a multiphase process that includes PFA development; staff evaluation of LOIs; the review panel’s preliminary review of full applications; an in-person panel discussion of a subset of full applications (identified by PCORI’s Research Priority Area Program staff and based on the preliminary review and program priorities); the Selection Committee’s recommendation of applications for funding; and, finally, Board award approval.

Preliminary Review

PCORI conducts rigorous merit review of the full applications it receives. Note that PCORI may eliminate applications from the review process for administrative or scientific reasons (e.g., non-responsiveness). An application may be administratively withdrawn if it is incomplete; submitted past the stated due date and time; or does not meet the formatting criteria outlined in the Application Guidelines, in the PCORI templates, and in PCORI Online. An application can be scientifically withdrawn if it is not responsive to the guidelines described in this PFA, describes research that is not comparative, includes a cost-effectiveness analysis, or otherwise does not meet PCORI programmatic requirements.

PCORI Merit Review Officers (MROs) recruit each panel based on the number of and topic areas represented by invited LOIs. MROs recruit the panel chair, scientist reviewers who are subject matter experts, patient representatives, and representatives of other stakeholder groups. All panel members receive training during the review cycle to ensure that they understand the programmatic and organizational goals of review.

The table below is designed to help applicants understand how the PCORI merit review criteria align with criteria from other funding organizations with which applicants might be familiar (e.g., NIH). Though PCORI’s criteria do map to most NIH criteria, there are areas where we ask for different information (i.e., PCORI does not include a criterion that tracks to NIH’s innovation criterion, but does include criteria evaluating patient-centeredness and engagement) reflecting PCORI’s unique approach.

Crosswalk of PCORI Merit Review Criteria with NIH Criteria
SIGNIFICANCE	Potential for the study to fill critical gaps in evidence Potential for the study findings to be adopted into clinical practice and improve delivery of care
APPROACH	Scientific merit (research design, analysis, and outcomes) Investigator(s) and environment
PCORI-Only Merit Review Criteria
PATIENT-CENTEREDNESS/ENGAGEMENT	Patient-centeredness Patient and stakeholder engagement

Below are PCORI’s merit review criteria. PCORI’s merit review panels use these criteria during the preliminary and in-person review phases to evaluate and score all submitted applications, and to ensure consistency and fairness in how applications are evaluated.

Criterion 1. Potential for the study to fill critical gaps in evidence:

The application should address the following questions:

• Does the application convincingly describe the clinical burden?
• Does the application identify a critical gap in current knowledge as noted in systematic reviews, guideline development efforts, or previous research prioritizations?
• Does the application identify a critical gap in current knowledge, evidenced by inconsistency in clinical practice and decision making?
• Would research findings from the study have the potential to fill these evidence gaps?

Criterion 2. Potential for the study findings to be adopted into clinical practice and improve delivery of care

The application should describe how evidence generated from this study could be adopted into clinical practice and delivery of care by others. The application should also address the following questions:

• Does the application identify who will make the decision (i.e., the decision maker) or use (i.e., the end-user) the study findings (not the intervention) this study produces, such as local and national stakeholders?
• Does the application identify potential end-users of study findings—such as local and national stakeholders—and describe strategies to engage these end-users?
• Does the application provide information that supports a demand for this kind of a study from end-users?
• Would this study’s research findings have the potential to inform decision making for key stakeholders? If so, provide an example. How likely is it that positive findings could be reproduced by others, resulting in improvements in practice and patient outcomes? Identify the potential barriers that could hinder adoption of the intervention by others.
• Does the application describe a plan for how study findings will be disseminated beyond publication in peer-review journals and at national conferences?

Criterion 3. Scientific merit (research design, analysis, and outcomes)

The application should show sufficient technical merit in the research design to ensure that the study goals will be met. The application should also address the following questions:

• Does the application describe a clear conceptual framework anchored in background literature which informs the design, key variables, and relationship between interventions and outcomes being tested?
• Does the Research Plan describe rigorous methods that demonstrate adherence to the PCORI Methodology Standards?
• Is the overall study design justified?
• Are the patient population and study setting appropriate for the proposed research question?
• Does the application provide justification that the outcome measures are validated and appropriate for the population?
• Are each of the comparators (e.g., active intervention arm and comparator arm) described clearly and well-justified? If “usual care” is one of the arms, is it adequately justified and will it be sufficiently measured?
• Are the sample sizes and power estimates appropriate? Is the study design (e.g., cluster randomized design, RCT, observational study) accounted for and anticipated effect size adequately justified?
• Is the study plan feasible? Is the project timeline realistic, including specific scientific and engagement milestones? Is the strategy for recruiting participants feasible? Are assumptions about participant attrition realistic, and are plans to address patient or site attrition adequate?

Criterion 4. Investigator(s) and environment

This criterion should assess the appropriateness (e.g., qualifications and experience) of the investigator(s)/team and the environment’s capacity (e.g., resources, facilities, and equipment) to support the proposed project. It should not be an assessment of the institution’s quality.

The application should also address the following questions:

• How well-qualified are the PIs, collaborators, and other researchers to conduct the proposed activities? Is there evidence of sufficient clinical or statistical expertise (if applicable)?
• Does the investigator or co-investigator have demonstrated experience conducting projects of a similar size, scope, and complexity?
• If the project is collaborative or dual-PI, do the investigators have complementary and integrated expertise? Are the leadership, governance, and organizational structures appropriate for the project?
  • (Dual-PI Option Only) Does the Leadership Plan adequately describe and justify PI roles and areas of responsibility?
• Is the level of effort for each team member appropriate for successfully conducting the proposed work?
• Does the application describe adequate availability of and access to facilities and resources (including patient populations, samples, and collaborative arrangements) to carry out the proposed research?
• Is the institutional support appropriate for the proposed research?

Criterion 5. Patient-centeredness

The application should demonstrate that the study focuses on improving patient-centered outcomes and employs a patient-centered research design (i.e., a design informed or endorsed by patients). (Note: The study can be patient-centered even if the end-user is not the patient, as long as patients will benefit from the information.)

The application should also address the following questions:

• Does the application include a thorough description about which outcomes (both benefits and harms) are important to patients, and are those outcomes included in the study plan?
• Does the application provide information that indicates that closing the evidence gap is important to patients and other stakeholders?
• Are the interventions being compared in the study available to patients now, and are they the best options for comparison (including whether they would be chosen by patients and their healthcare providers for managing the condition being studied)?

Criterion 6. Patient and stakeholder engagement

The application should demonstrate the engagement of relevant patients and other stakeholders (e.g., patients, caregivers, clinicians, policy makers, hospitals and health systems, payers [insurance], purchasers [business], industry, researchers, and training institutions) in the conduct of the study. Quality of engagement should be evaluated based on scope, form, and frequency of patient and stakeholder involvement throughout the research process.

The application should also address the following questions:

• Does the application provide a well-justified description of how the research team incorporates stakeholder involvement? Does the study include the right individuals (e.g., researchers, patients, caregivers, clinicians, policy makers and other healthcare system stakeholders) to ensure that the projects will be carried out successfully?
• Does the application show evidence of active engagement among scientists, patients, and other stakeholders throughout the research process (e.g., formulating questions, identifying outcomes, monitoring the study, disseminating, and implementing)? Is the frequency and level of patient and stakeholder involvement sufficient to support the study goals?
• Is the proposed Engagement Plan appropriate and tailored to the study?
• Are the roles and the decision-making authority of all study partners described clearly?
• Are the organizational structure and resources appropriate to engage patients and stakeholders throughout the project?

In-Person Review

During preliminary review, all administratively and scientifically compliant applications are evaluated and scored based on PCORI’s merit review criteria, including evaluation of adherence to the PCORI Methodology Standards. After PCORI completes the preliminary review, PCORI program staff members evaluate panel scores and critiques to identify a subset of applications for merit reviewers to discuss at the in-person review meeting. Not all submitted applications move forward to in-person review.

During the in-person review, merit reviewers meet to discuss applications and to clarify further the merits of the proposed research. They also identify areas for improvement. Each application is re-scored based on the content of discussion. The Panel Chair and PCORI MRO lead the in-person panel meeting and ensure that all applications receive a fair and thorough review according to the standards outlined in the PFA.

Post-Panel Review

After the in-person meeting, PCORI program staff evaluate final merit review panel scores and comments, identify duplication or synergy among funded projects, and consider the fit of applications within the programmatic vision. Program staff members then recommend projects to a Selection Committee, which includes members of the Board. The Selection Committee considers recommendations and works with staff to identify a slate of applications for possible funding based on merit review scores, programmatic balance and fit, and PCORI’s strategic priorities. This slate is then proposed to the Board for consideration and approval.

In addition, PCORI evaluates applicant risk before issuing a PCORI award. Factors considered include financial stability, quality of management systems, audit findings, and past performance on PCORI awards (e.g., compliance with PCORI reporting requirements, conformance to PCORI terms and conditions on previous awards, and timely achievement of milestones). Based on the risk assessment, PCORI may impose special terms and conditions on awardees or withhold contract issuance until such business risks are mitigated. PCORI will not award new contracts to current awardees with overdue reports (progress, interim, final, etc.) until the overdue reports have been submitted to PCORI. 

Summary Statements and Funding Recommendations

Summary statements are provided to applicants approximately two weeks before funding decisions are announced. If an application progresses to in-person discussion, the applicant will receive a summary statement inclusive of:

• In-person panel discussion notes
• Final average overall score
• Preliminary reviewer critiques
• Application quartile, which provides information for applicants to understand how they did relative to other discussed applications

Summary statements for applications that do not progress to in-person discussion include only the preliminary reviewer critiques.

Funding recommendations are made by identifying meritorious applications that fit the programmatic needs and that satisfactorily address the merit review criteria while adhering to the PCORI Methodology Standards. Programs also consider the funds allotted for the current funding announcement when deciding which applications to recommend to the Board for approval. Applicants to this current cycle’s PFA will receive summary statements in August 2018 and notification of the funding status of their application no later than September 2018.

Back to top