Pilot Project: Comparing Two Research Methods to Measure How Well Dynamic Treatment Works

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

People who have the same chronic condition often receive different treatments. The choice of treatment sometimes depends on things like the patient’s age, sex, and treatment history. Also, doctors often change treatment over time depending on how the patient responds. When treatment varies in these ways, it is called dynamic. Researchers find it difficult to analyze data about the effectiveness of dynamic treatments because the way the treatments are used changes over time.

For example, how well the medicine being studied works might depend on how well a medicine the patient took in the past worked. Usual research methods don’t handle this difficulty well.

The research team proposed using an analysis method called the “parametric g-formula” to address this difficulty. This method has not yet been used with existing data from medical records or health insurance companies.

Project Purpose

The researchers compared the g-formula method with a more commonly used method of analysis called “inverse probability weighting” (IPW) to learn the advantages and disadvantages of using these methods to study dynamic treatments.

Methods

The research team compared the g-formula and IPW methods using medical record data stored in a registry of patients with chronic kidney disease called the United States Renal Data System (USRDS). The study included records for 156,945 patients who met all of the following criteria:

• Had both kidney failure and heart disease
• Got hemodialysis (using a machine to clean the blood when the kidneys don’t work right) in an outpatient clinic each June from 2006 to 2010
• Took a medicine called epoetin alfa (EPO), which is used to help patients make more red blood cells

Patients got different amounts of EPO depending on whether the doctor wanted the patient to maintain low, medium, or high levels of red blood cells.

The research team used the g-formula and IPW methods to see how often patients who got these different treatments died or had heart attacks in an 18-month period. They also compared the results from both methods to results from a randomized controlled trial that asked a similar question.

Findings

The research team found that patients who got EPO doses intended to give them low levels of red blood cells had less chance of dying than patients who got EPO doses intended to give them medium or high levels of red blood cells. The g-formula and IPW methods produced the same results. The results using patient registry data were the same as the results from the large randomized trial.

Limitations

Both the g-formula and IPW methods are limited by the data that are available at the time a study is conducted. For example, if something influences health and affects the patient outcome (in this case, mortality) but was not recorded in the data, then researchers can’t take it into account. In this study, researchers don’t know the exact EPO dose and red blood cell response from each treatment. Instead, researchers only have monthly data on EPO doses and patient red blood cell levels.

Conclusions

The research team found that the g-formula and IPW methods worked equally well for this case. The research team encourages using the g-formula and IPW together whenever possible to confirm the results.

Findings from this study support current US Food and Drug Administration guidelines that recommend keeping levels of red blood cells in the blood low (less than 33 percent) for patients with kidney and heart disease who get hemodialysis.

Sharing the Results

The researchers shared results of their work through a webinar and are submitting articles to medical journals (see below).

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

The most common treatment strategies for the care of persons with chronic medical conditions are dynamic. The g-formula method, a powerful tool to compare dynamic treatment strategies, has never been applied to large administrative databases. Therefore, the feasibility and relative advantages and disadvantages of this method compared to other more commonly used analytical methods such as inverse-probability weighting (IPW) are unknown.

Project Purpose

To provide a step-by-step description of how to implement the parametric g-formula and IPW methods based on causal inference and to compare g-formula results to those of IPW. Unlike standard regression analysis, these methods can appropriately adjust for time-varying confounders themselves affected by past treatment. As a case study, researchers compared dynamic epoetin alfa (EPO) treatment strategies to correct anemia in hemodialysis patients covered by Medicare using the g-formula method versus IPW.

Study Design

This study used a retrospective cohort study design.

Participants, Interventions, Settings, and Outcomes

Eligible patients were individuals with end-stage renal disease and congestive heart failure or ischemic heart disease who were undergoing hemodialysis in outpatient dialysis facilities in the United States in June of each year between 2006 and 2010.

Researchers examined three dynamic EPO treatment strategies that are consistent with current and historical clinical practice but that have never been compared in randomized trials: (1) low hematocrit (adjust dose of EPO to try and maintain hematocrit at 30–33%); (2) mid hematocrit (adjust dose of EPO to try and maintain hematocrit at 33–36%); and (3) high hematocrit (adjust dose of EPO to try and maintain hematocrit at 36–39%).

Similar to existing EPO trials, the primary endpoint is a composite outcome including death and hospitalization for myocardial infarction, and the secondary endpoint is all-cause mortality. Study subjects are followed from study baseline until the outcome, censoring due to loss to follow-up (defined as data anomalies, switch to darbepoetin use, and a 30-day gap in outpatient dialysis or inpatient claims), or administrative end of study (18 months after baseline), whichever occurs earlier.

Data Sources

This study used data collected and maintained by the United States Renal Data System (USRDS). This database incorporates demographics and a detailed longitudinal record of utilization, diagnoses, and procedures for all dialysis patients covered by Medicare. The Researcher's Guide to the USRDS Database, available from http://www.usrds.org, describes variables, data sources, collection methods, and validation studies.

Data Analysis

An analysis to estimate the effect of following different dynamic treatment strategies over time (i.e., “per-protocol effects’’) requires adjustment for both baseline and post-baseline prognostic factors that affect adherence to the protocol. Conventional regression analysis cannot appropriately adjust for prognostic factors such as hematocrit values (which are affected by past treatment). For this reason, g-methods—including IPW, g-estimation of structural equations models, and the g-formula—are often needed. In this study, researchers applied the parametric g-formula and IPW to estimate the 18-month risk ratios comparing risks under three treatment strategies. Under the assumptions of no unmeasured confounding and no model misspecification, estimates from the observational analyses conducted in this study using the parametric g-formula and IPW can be interpreted as if they were obtained from a randomized clinical trial.

Findings

Using data from USRDS, estimates indicated that among renal failure patients with cardiac disease undergoing hemodialysis, targeting low hematocrit of 30–33% reduces the risk of death when compared to higher hematocrit targets. The table below delineates results based on the g-formula and IPW analysis separately.

These findings are consistent with existing empirical evidence provided by the Normal Hematocrit Study (NHS), the largest randomized trial that included patients with cardiac disease who were undergoing hemodialysis and receiving EPO between 1993 and 1996. The patients were assigned to one of two anemia management strategies: one group aimed to achieve a normal hematocrit of 42%, and another aimed to achieve a low hematocrit of 30%. The trial was terminated early (at 29 months) because patients who were randomized to the normal-hematocrit strategy had increased mortality and myocardial infarction. Specifically, 29-month risk ratio for the primary endpoint (death or a first nonfatal MI) for the normal-hematocrit group as compared with the low-hematocrit group was 1.3 (95% CI, 0.9 to 1.9). The 18-month risk ratio was found to be similar (1.2, CI not reported). The research described herein supports current Food and Drug Administration advisories recommending a hematocrit target of <33% when treating hemodialysis patients, including those with serious comorbidities.

Limitations

In contrast to IPW, the parametric g-formula is more flexible because it can be used to compare many treatment strategies that are difficult or impossible to estimate using IPW. However, one trade-off is heavier reliance on parametric modeling (i.e., time-varying covariates, exposures, and outcomes are all predicted using parametric models, resulting in increased concerns of model misspecification). Both the g-formula method and IPW cannot adjust for unmeasured confounding (e.g., unmeasured risk factors for the outcome that affect compliance with the pre-specified study protocol). Generally, methods for causal inference from observational data—no matter how sophisticated and including those described in this analysis—will yield biased estimates unless the assumption of no unmeasured confounders is approximately true given the variables measured by the investigators. Researchers considering using these methods should carefully evaluate the assumptions and use more than one method for robustness.

Conclusions

Researchers provided an approach to compare dynamic dosing strategies based on Medicare claims data via the parametric g-formula analysis. They illustrated how to implement this powerful tool by estimating the observational analogue of the “per-protocol” effect and using the publicly available GFORMULA SAS macro. Future implementations of the g-formula can extend this example to perform multiple comparisons of real-world dynamic strategies from various observational databases. Compared to the most commonly used IPW approach, the parametric g-formula might be more advantageous or suitable in addressing certain types of dynamic treatment questions (although both are encouraged as confirmatory analysis whenever possible). Specifically, in addition to increased statistical efficiency, the g-formula allows you to define and compare a wide range of hypothetical interventions, especially those that are dynamic and continuous as EPO strategies. Given the recent availability of the SAS g-formula macro, the parametric g-formula should be routinely used for analysis of dynamic treatment regimens together with other causal inference methods such as IPW.