Developing and Testing New Methods for Estimating Treatment Effectiveness in Observational Studies Using High-Dimensional Data

• Download Summary

What was the project about?

A randomized controlled trial, or RCT, is often the best way to learn if one treatment works better than another. RCTs assign patients to different treatments by chance. But RCTs are not always feasible. In such cases, researchers can use observational studies. In observational studies, researchers look at what happens when patients and their doctors choose the treatments. Traits such as age, gender, or health status may affect treatment choices. These traits may also affect patients’ health, making it hard to know if changes in patients’ health are due to treatment or to patient traits.

To figure out whether changes in patients’ health result from treatment or something else, researchers use statistical methods. Two of these methods are:

• Propensity score, or PS. PS methods compare the health of patients who have similar measured traits but received different treatments. These traits are in patient health records.
• Instrumental variable, or IV. IV methods account for things that may affect treatment choice and patients’ health but aren’t in the patients’ health records, such as personal preference about treatment.

But existing PS and IV methods don’t work well when data sets include a lot of traits and health conditions for each patient. Such data sets are called high-dimensional data.

In this study, the research team created and tested one PS method and one IV method for use with high-dimensional data.

What did the research team do?

First, the research team created the two new methods for use with high-dimensional data. The team then used a computer program to create test data that look like real patient data. The team applied the new methods to the test data.

Next the research team applied the new methods to real data from previous studies. They applied the PS method to data from a study that looked at a medical test for measuring how well a person’s heart is pumping. The team applied the IV method to data from a study that looked at the effect of education on personal earnings.

Using both test and real data, the research team compared findings from the new methods with those from existing PS and IV methods. They checked to see if findings from the new methods were accurate when they included different patient traits and health conditions in the analysis.

What were the results?

Compared with existing methods, the new methods led to more accurate results, even when including a variety of patient traits or health conditions in the analysis. The research team also created a computer program called RCAL that applies the methods in the R statistical software.

What were the limits of the project?

The research team created and tested the new methods for studies that look at patients’ health at one point in time. The methods may not apply to studies that look at patients’ health over time.

How can people use the results?

Researchers can use the new methods to measure treatment effects when analyzing high-dimensional data in observational studies.

Background

Randomized controlled trials are often the best way to determine whether differences between patient health outcomes are due to treatments. However, random assignment is not always feasible or ethical. In observational studies, researchers use statistical methods to mimic random treatment assignment. Two such methods are:

• Propensity scores (PS). Researchers use statistical methods to create two groups of patients with similar characteristics who received or did not receive the treatment. Researchers then compare health outcomes between the two groups.
• Instrumental variables (IV). Researchers divide the study sample by whether a patient has a characteristic that affects the treatment choice but does not directly affect the outcome, much like randomizing treatment.

PS and IV methods account for confounders that could affect both health outcomes and treatment choices. However, they have limitations, especially when using high-dimensional data. High-dimensional data have numerous variables or many nonlinear and interaction terms for a moderate number of variables.

To use PS and IV methods with high-dimensional data, researchers make ad hoc choices about which variables and nonlinear and interaction terms to include. These choices may lead to model misspecification. Model misspecification can occur when statistical methods do not account for all confounders, which results in biased or imprecise estimates. PS and IV methods that account for model misspecification may estimate treatment effects more accurately and reliably when using high-dimensional data.

Objective

To develop and test a new set of PS and IV methods that account for model misspecification when estimating causal effects of treatments using high-dimensional data

Study Design

Methods

The research team developed a PS method and an IV method for use with high-dimensional data that account for model misspecification. The PS method estimates treatment effects in the absence of unmeasured confounders. The IV method estimates treatment effects when the data do not include all confounders.

The research team compared the new and existing methods using simulation and empirical analyses with varying degrees of model misspecification. To empirically test the new PS method, the team used data from a medical study about the effects of right heart catheterization. The team tested the IV method with survey data to estimate the causal effect of education on earnings.

Results

In simulation analysis, the new methods led to lower bias and more accurate coverage proportions in confidence intervals than existing methods when statistical models were misspecified. In empirical analysis, the magnitudes of treatment effect estimates varied across the new and existing PS and IV methods, but estimates from the new methods had lower standard errors than those from existing methods.

The research team developed RCAL, a computer program, to implement the new methods in R statistical software.

Limitations

The new methods are for cross-sectional observational studies and may not apply to longitudinal or survival studies that examine patient outcomes over time.

Conclusions and Relevance

Researchers can use the new methods to estimate treatment effects when implementing PS or IV analysis with high-dimensional data.

Future Research Needs

Future research could extend the methods to analyze longitudinal and survival data.