• Download Summary

What was the project about?

One way to see if a treatment works is to compare data from people who received the treatment with data from those who didn’t or who received a different treatment. But sometimes the ways that people differ, such as their age or other health problems, can bias results. For example, if the people who didn’t get the treatment are older or sicker than people who did get the treatment, results could suggest that the treatment works better than it really does.

One way to avoid this type of bias is to use case-only study designs. Case-only studies compare each patient’s health before and after treatment. But case-only studies often report the relative risk of a health event, such as stroke, among two groups of patients, instead of the absolute risk. For example, relative risk can show how the risk of stroke differs between patients who smoke and those who do not. Absolute risk would give the percentage of patients having a stroke among all patients. Absolute risk can help inform treatment decisions. But methods to measure absolute risk in case-only studies are limited. Also, clear guidance is lacking on how to best design and analyze a case-only study.

In this study, the research team created a guide and new methods for designing and analyzing case-only studies.

What did the research team do?

To create the guide for choosing the right study design, the research team compared six types of case-only studies. First, the team used a computer program to create test data to look like real patient health data. Using the test data, the team compared the accuracy of results from each of the six types of case-only studies to learn which study design works best for different types of data. The team also looked at the risk of bias under different scenarios. For example, in one scenario, patients’ health habits, like smoking, changed before and after treatment.

The research team then created new methods to measure absolute risk in case-only studies. The methods show:

• How many people need to receive treatment for one of them to have a benefit or harm
• Whether treatments work better or worse for different groups of people

Doctors and patients provided input on the study design.

What were the results?

The research team created a guide to help researchers choose the right study design for case-only studies. The guide describes which case-only design to use based on the type of data and the purpose of the study. The new methods help measure absolute risk in case-only studies.

What were the limits of the project?

The new methods can only be used when patients’ treatment status is a category, such as yes versus no or daily versus weekly. The methods don’t work, for example, when a patient’s treatment dose changes over time.

How can people use the results?

Researchers can use the guide when designing and analyzing case-only studies.

Background

Comparative effectiveness research studies often use secondary data sources, such as electronic health records. Secondary data include detailed clinical information but may be missing data on potential confounders, which are variables that affect both treatment choices and outcomes. Not accounting for confounders in an analysis can lead to biased results about the effectiveness of a treatment.

In such situations, researchers can use case-only studies, which compare each patient’s outcomes before and after treatment. Case-only studies eliminate the influence of confounders that remain stable over time. However, case-only studies do not identify heterogeneity of treatment effects. Also, they typically report results in terms of relative risk rather than absolute risk, as current methods for estimating absolute risk in case-only studies are limited. Information about absolute risk, like the number of patients who need to be treated for at least one patient to have a benefit or harm from treatment, can help in clinical decision making.

Objective

To improve methods and provide clear guidance for designing and analyzing case-only studies

Study Design

Methods and Results

To develop guidance for selecting and analyzing a case-only design, the research team analyzed simulated data and compared the validity and efficiency of findings from six case-only designs. The team examined the potential bias when statistical assumptions of case-only designs were and were not met. For example, case-only designs assume that confounders and the probability of receiving treatment do not change over time. Based on the simulation results, the team described data and design considerations for case-only studies that worked when different statistical assumptions were relaxed.

To estimate absolute risk and heterogeneity of treatment effects, the research team developed new methods that account for differences in changes over time in the probability of treatment and health outcomes.

The research team provided guidance about how to select an appropriate case-only design to best address the research question of interest, corresponding statistical assumptions, and data considerations.

Doctors and patients provided input that helped in designing the study.

Limitations

The new methods are only appropriate for categorical treatment status, such as a binary indicator for receiving a particular treatment.

Conclusions and Relevance

The new methods calculate absolute risk, which can aid in clinical decision making. The methods guidance can help researchers select an optimal design and develop an analytic plan for case-only studies.

Future Research Needs

Future research could focus on developing methods for continuous treatment status, such as varying the treatment dosage.