Results Summary
What was the project about?
A randomized controlled trial, or RCT, is the best way to compare how well different treatments work to improve patients’ health. In RCTs, researchers assign patients to treatment groups by chance. But RCTs aren’t always an option due to high costs or ethical concerns. In these cases, researchers use other types of study designs such as
- Cohort studies, which look at patients’ data over time to see how a treatment affects the risk of a certain health event, such as a heart attack
- Case-control studies, which compare data from patients who did and didn’t have a certain health event
These designs often use data from health records to compare treatment results. In these studies, researchers use statistical methods to make results more like results from RCTs. Current methods work well for cohort studies but not for case-control studies.
In this study, the research team created and tested new methods and a guide to analyze case-control studies so that results would be more like results from an RCT.
What did the research team do?
The research team talked with researchers who do case-control studies. The team used what they learned to create the new methods and a guide. The guide explained how to use the new methods for case-control studies. The team then asked the researchers for feedback to improve the guide.
To test the guide and new methods, the research team used health records from patients receiving care at health systems in Washington State. The team did a case-control and cohort study to look at whether statins, a medicine to lower cholesterol, helped prevent heart attacks. Next, the team compared the results from previous RCTs of statins to results from
- The new methods for case-control studies
- Current methods for case-control studies
- Current methods for cohort studies
What were the results?
The guide had information on how to mimic an RCT with case-control data and how to set up data for analysis. The research team also created a computer program to analyze case-control data.
Compared with current methods for case-control and cohort studies, the new methods for case-control studies got more accurate results.
What were the limits of the project?
The new methods only work when researchers collect data at multiple time points. The research team tested the new methods using data for one treatment for heart attacks.
Future research could test if the new methods work for other health problems or treatments.
How can people use the results?
Researchers can use the guide and computer program when designing case-control studies.
Professional Abstract
Background
Randomized controlled trials (RCTs) are the optimal study design for testing treatment efficacy but sometimes, they are not viable due to cost or ethical considerations. In these situations, researchers use observational designs, such as cohort or case-control studies; these designs often use electronic medical record (EMR) data.
In case-control studies, researchers can validate data through record reviews and patient interviews with a small number of cases and controls, but this process is expensive for large cohort studies. Causal methods, or methods for estimating causal effects of treatments, do not exist for case-control studies, and conventional case-control analysis methods can lead to biased results. Researchers need to choose between using causal methods with potentially lower-quality data from a cohort design or using conventional methods with higher-quality data from a case-control design.
Objective
To reduce bias in comparative effectiveness research in case-control studies by developing causal methods that emulate the design and analysis of a hypothetical RCT, or a target trial
Study Design
| Design Element | Description |
|---|---|
| Design | Methods development; qualitative analysis; empirical analysis |
| Data Sources and Data Sets |
2 sets of semistructured interviews with 20 investigators and data analysts identified by the University of Washington, Kaiser Permanente Washington, and Harvard T. H. Chan School of Public Health EMR data from Kaiser Permanente Washington enrollees from January 1, 1993, through December 31, 2014
|
| Analytic Approach |
|
| Outcomes | Hazard ratios and odds ratios of fatal and nonfatal myocardial infarction |
Methods
The research team interviewed 20 investigators and data analysts with experience conducting case-control studies. Questions focused on replicability of a target trial in an observational study and preferred statistical programs and causal methods for case-control studies. Based on thematic analyses, the team developed a guideline for a new method of causal analysis that emulates a target trial using a case-control design. The team then interviewed the investigators and data analysts again to get feedback on the guideline.
Applying the guideline, the research team emulated a target trial of statin therapy for fatal or nonfatal myocardial infarction using a cohort design and a case-control design. Using EMR data from Kaiser Permanente Washington patients between 1993 and 2014, the team compared relative risks in terms of odds ratios and hazard ratios from conventional case-control analysis, causal cohort analysis, and the new causal case-control analysis to a previously published meta-analysis of RCTs of statin therapy.
Results
The guideline provided details on how to conceptualize the target trial, prepare the analytic data sets, and conduct the analyses. The research team also created a computer program for causal inference analysis in SAS, a type of statistical software.
Odds ratios from causal case-control methods were consistent with estimates from a meta-analysis of RCTs. However, hazard ratios from the cohort and odds ratios from a conventional case-control analysis were inconsistent with the estimates from the meta-analysis.
Limitations
The methods require time-varying data on eligibility, treatment, and factors affecting treatment choice and health outcomes. The methods do not apply to case-control studies that have measured such factors only at or just prior to the date of the health outcome. The research team tested the methods using data for one treatment for heart attacks.
Conclusions and Relevance
Emulating the design and analysis of a target trial using a case-control design reduced bias compared with conventional case-control analysis.
Future Research Needs
Future research could apply the methods to target trials of other medications or to evaluate other health outcomes.
Final Research Report
View this project's final research report.
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers lauded the strengths of this report and requested minor changes, focusing largely on the investigators providing more information on the similarities and differences of cohort and case-control observational research designs. The researchers added language to describe the similarities and differences in these two approaches, as well as provided previously published explanations of the theoretical background behind the approaches the researchers evaluated in this study.
- The reviewers noted that there were demographic differences in patient characteristics between the cohort and case-control samples and wondered if these differences could lead to bias that would make the results of the cohort and case-control comparison different from results of similar analyses between either of these designs and a randomized trial design. The researchers acknowledged that there were differences in prevalence of hypertension and use of antihypertensives between the cohort and case-control study design results. However, the researchers stated that they did not expect the same demographic factors to affect the comparison of these two study designs because previous meta-analyses have not demonstrated that these factors led to differences in other studies of antihypertensives and myocardial infarction.
Conflict of Interest Disclosures
Project Information
Key Dates
Study Registration Information
| This project's final research report is expected to be available by December 2021. |