Modeling Strategies for Observational Comparative Effectiveness Research - What Works Best When?
Comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) both focus on what works best under which circumstances for a given patient. Observational data, which include data sources such as patient registries, health plan data, and data collected from large observational studies (where patients are followed for outcomes without studying specific interventions), represent an important and increasing available tool for CER and PCOR. Observational data have many strengths, such as using generalizable populations and assessing effectiveness of treatments based on exactly how they are utilized in practice. However, there are also many concerns, such as self-selection of treatment status by choice (vs. randomization), which can also involve factors associated with the outcome and thus inappropriately influence our conclusion about treatment effectiveness. There are many methods to address this issue, including regression, many variations of propensity scores (to account for one's propensity to receive treatment), and instrumental variables. However, there are still no clear answers to the most fundamental questions, such as "How do I know if standard regression models are sufficient?" or "Which method should I trust more if instrumental variables and propensity scores lead to different results for my data?"; the existing literature is generally inadequate or not easily interpretable in terms of guiding CER and PCOR. We seek to develop an observational analysis methodology decision tree (OAMDT) for recommending optimal analysis method(s) for a given data set.
We will accomplish this overall goal through three specific aims. First, we will conduct a systematic review of the existing literature to assess and organize results about what is currently known about these methods. Although there are a number of existing reviews, they are too limited in scope to address our concerns and adequately inform development of the OAMDT. Next, although much is already known, other guidance on what to do when will need to be determined by simulating data from known associations, i.e., where we already know the true answer and we assess which method works best. Those first two aims of our proposed project will inform development of a draft OAMDT, to be reviewed by stakeholders who conduct CER and PCOR in practice. They will assess interpretability and usefulness of the tool, and we will subsequently pilot the tool through the University of Pittsburgh's Institute for Clinical Research Education (ICRE) CER courses. The revised OAMDT will be reviewed again by SCIs and an expert panel. This version will then be disseminated through
- subsequent ICRE CER courses,
- the ICRE website, and
- presentations and publications; all steps will target applied or nonstatisticians doing observational CER.
The final impact will be providing clinicians, and ultimately patients, with the best possible information on which treatment works best for them.
VIDEO (below): Including Stakeholders in Research
Although skeptical at first, this statistician included some unusual stakeholders in his methods research.