Project Summary

Patients, providers, and other stakeholders need to know when they can trust observational analyses (i.e., analyses that do not rely on random experimental treatment assignment) to learn about the effects of different treatments. To determine when observational analyses can be trusted, one commonly used strategy is to compare the results of observational analyses designed to closely emulate existing randomized trials against the results of those trials. We refer to such comparisons as benchmarking the observational analyses against the trial.

Benchmarking comparisons can have important practical implications: if observational analyses can produce estimates with the same interpretation as those of trials for research questions where trials are indeed available, then patients and providers may be able to trust observational analyses enough to rely on them when trials are infeasible or inadequate. Furthermore, when benchmarking comparisons suggest that observational analyses (for a particular clinical topic) can be trusted, it may be desirable to jointly analyze the trial data with the observational data to improve efficiency. For example, when examining heterogeneity of treatment effects for critical, patient-relevant outcomes, for which trials are typically under-powered, joint analysis may be beneficial.

However, there is no good framework for conducting and interpreting benchmarking comparisons. Furthermore, methods for joint analysis of trial and observational data are not well-developed and have not been carefully tested. Our long-term goal is to support patient-centered outcomes research and care by developing observational analysis methods that can be trusted when trials are infeasible or inadequate. To move toward this goal, we seek to accomplish four Specific Aims:

(1) Develop a framework for benchmarking observational analyses against trials.

(2) Develop statistical methods for joint analysis of data from trials and observational analyses.

(3) Evaluate the methods developed under Aims 1 & 2 in simulation studies.

(4) Implement the methods developed for Aims 1 & 2 in in two use cases involving comparative effectiveness studies of treatments for myocardial infarction.

To achieve these Aims, with input from 2 stakeholder panels (a Technical Expert Panel and a Patient & Provider panel), we propose to develop a comprehensive benchmarking framework by combining state-of-the-science causal inference and statistical methods. The framework will allow investigators to conduct and interpret comparisons between trials and observational analyses designed to emulate the trials.

Building on this framework, we will develop novel methods for the joint analysis of trial and observational data. The methods will be appropriate for outcomes that patients care about, including outcomes assessed at the end of the study (e.g., status at 30 days after treatment), outcomes assessed over time (e.g., repeat measurements of functional status), and time-to-event outcomes (e.g., overall survival or time free-from-recurrence).

Next, we will evaluate the benchmarking and joint analysis methods in large-scale, simulation studies that will resemble real-world data structures. Last, we will implement the methods to analyze data from two large, registry-based trials of treatments for myocardial infarction and their corresponding observational analyses (conducted with registry data, including individuals who were eligible for but did not participate in the trials). These applied analyses have the potential to directly influence clinical practice and will also aid the dissemination of the methods to practitioners.

Project Information

Issa Dahabreh, MD, ScD, MS
President and Fellows of Harvard College
$1,009,437 *

Key Dates

36 months *
March 2022

*All proposed projects, including requested budgets and project periods, are approved subject to a programmatic and budget review by PCORI staff and the negotiation of a formal award contract.


Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: April 27, 2022