Results Summary
What was the project about?
One way to see if a treatment works is to compare data from people who received the treatment with data from those who didn’t or who received a different treatment. But sometimes the ways that people differ, such as their age or other health problems, can bias results. For example, if the people who didn’t get the treatment are older or sicker than people who did get the treatment, results could suggest that the treatment works better than it really does.
One way to avoid this type of bias is to use case-only study designs. Case-only studies compare each patient’s health before and after treatment. But case-only studies often report the relative risk of a health event, such as stroke, among two groups of patients, instead of the absolute risk. For example, relative risk can show how the risk of stroke differs between patients who smoke and those who do not. Absolute risk would give the percentage of patients having a stroke among all patients. Absolute risk can help inform treatment decisions. But methods to measure absolute risk in case-only studies are limited. Also, clear guidance is lacking on how to best design and analyze a case-only study.
In this study, the research team created a guide and new methods for designing and analyzing case-only studies.
What did the research team do?
To create the guide for choosing the right study design, the research team compared six types of case-only studies. First, the team used a computer program to create test data to look like real patient health data. Using the test data, the team compared the accuracy of results from each of the six types of case-only studies to learn which study design works best for different types of data. The team also looked at the risk of bias under different scenarios. For example, in one scenario, patients’ health habits, like smoking, changed before and after treatment.
The research team then created new methods to measure absolute risk in case-only studies. The methods show:
- How many people need to receive treatment for one of them to have a benefit or harm
- Whether treatments work better or worse for different groups of people
Doctors and patients provided input on the study design.
What were the results?
The research team created a guide to help researchers choose the right study design for case-only studies. The guide describes which case-only design to use based on the type of data and the purpose of the study. The new methods help measure absolute risk in case-only studies.
What were the limits of the project?
The new methods can only be used when patients’ treatment status is a category, such as yes versus no or daily versus weekly. The methods don’t work, for example, when a patient’s treatment dose changes over time.
How can people use the results?
Researchers can use the guide when designing and analyzing case-only studies.
Professional Abstract
Background
Comparative effectiveness research studies often use secondary data sources, such as electronic health records. Secondary data include detailed clinical information but may be missing data on potential confounders, which are variables that affect both treatment choices and outcomes. Not accounting for confounders in an analysis can lead to biased results about the effectiveness of a treatment.
In such situations, researchers can use case-only studies, which compare each patient’s outcomes before and after treatment. Case-only studies eliminate the influence of confounders that remain stable over time. However, case-only studies do not identify heterogeneity of treatment effects. Also, they typically report results in terms of relative risk rather than absolute risk, as current methods for estimating absolute risk in case-only studies are limited. Information about absolute risk, like the number of patients who need to be treated for at least one patient to have a benefit or harm from treatment, can help in clinical decision making.
Objective
To improve methods and provide clear guidance for designing and analyzing case-only studies
Study Design
| Design Element | Description |
|---|---|
| Design | Theoretical development; simulation studies |
| Data Sources and Data Sets | Simulated data based on data from Beth Israel Deaconess Medical Center (N=120,000) |
| Analytic Approach |
Develop guidance about choosing and implementing the appropriate case-only design to address the research question of interest Simulation analysis to compare findings from 6 case-only designs:
|
| Outcomes |
Validity (bias) and efficiency (mean squared error; mean-variance ratio for the estimator) |
Methods and Results
To develop guidance for selecting and analyzing a case-only design, the research team analyzed simulated data and compared the validity and efficiency of findings from six case-only designs. The team examined the potential bias when statistical assumptions of case-only designs were and were not met. For example, case-only designs assume that confounders and the probability of receiving treatment do not change over time. Based on the simulation results, the team described data and design considerations for case-only studies that worked when different statistical assumptions were relaxed.
To estimate absolute risk and heterogeneity of treatment effects, the research team developed new methods that account for differences in changes over time in the probability of treatment and health outcomes.
The research team provided guidance about how to select an appropriate case-only design to best address the research question of interest, corresponding statistical assumptions, and data considerations.
Doctors and patients provided input that helped in designing the study.
Limitations
The new methods are only appropriate for categorical treatment status, such as a binary indicator for receiving a particular treatment.
Conclusions and Relevance
The new methods calculate absolute risk, which can aid in clinical decision making. The methods guidance can help researchers select an optimal design and develop an analytic plan for case-only studies.
Future Research Needs
Future research could focus on developing methods for continuous treatment status, such as varying the treatment dosage.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. This final research report was well received by the reviewers. They asked for clarification on some of the concepts the researchers described but they did not consider these clarifications to be major concerns with the report overall.