From our early days, we’ve worked to evaluate whether our efforts are making a difference. A direct comparison of PCORI-funded research to similar studies funded by others would be the most straightforward way to know whether our approach not only is different, but in fact leads to different results. Now that we have funded more than 400 CER studies, a substantial number of which are completed or nearly so, we can take a clear look at how our research portfolio is unique and complements that of other funders.
Comparing Two Portfolios
Fortunately, the National Institutes of Health, our nation’s premier funder of biomedical research, has worked to make information on its research portfolio transparent, so we can compare our CER work to theirs. The NIH’s primary focus is biologic discovery and developing new treatments based on those efforts, but it does classify a small number of the studies it funds as CER. So we’ve now launched a series of projects comparing key aspects of CER studies funded by our two institutions, including important study characteristics, investigator characteristics, markers of study efficiency, and indicators of dissemination (see box).
PCORI’s analysis comparing our research portfolio with other funders’ CER portfolios is just beginning. It will include comparisons such as the following:
We are committed to ensuring that our findings are both trustworthy and useful. Thus, we are working closely with NIH leadership and staff experts to be sure we are asking the right questions and that our interpretations of the NIH portfolio are justifiable. We’ll aim to apply the highest standards for methodological rigor. We’re also committed to transparency and reproducibility of our findings; a draft of our project protocol will soon appear on PCORI’s Evaluating Our Research Portfolio web page. As the project unfolds, we’ll post updates and study materials, so others can explore further.
Our first step involves using publicly available data from the PCORI and NIH websites and ClinicalTrials.gov to assess similarities and differences in our two organizations’ CER study characteristics. We plan to look at such things as the clinical conditions under study, types of interventions compared, and the primary study purpose, such as prevention, diagnosis, or treatment. We’ll also examine study design, sample size, duration, and cost. We’ll pay particular attention to possible differences in the numbers and types of outcomes that those studies measure, including patient-reported outcomes.
Through this first phase, we’ll gain a better understanding of how each organization is contributing to comparative effectiveness research. We hope both organizations—and the research community as a whole—will find valuable insights in the results. We will then use what we’ve learned to inform how we go about looking at study efficiency and impact, to make our comparisons of other elements of our portfolios appropriate and robust. At the end of this process, we will be able to tell our stakeholders more about the return on investment that results from the “PCORI Way.”
These comparisons will help us ensure we’re on the path to the ultimate goals of speeding the uptake of research in clinical decision making and improving healthcare outcomes. While we don’t yet know what we will learn from this project, PCORI is committed to reporting on the investment in the organization, evaluating ourselves against established goals, and improving future research.