PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

Treatments for many health problems don’t work the same way in all people. A person’s age, sex, health problems, genes, and other factors can affect how well a treatment will work. For example, a treatment might work well in most people with a heart problem but not very well for people who also have diabetes. Researchers usually do not look at how well a treatment works for different groups of individuals (e.g., men and women, those with or without diabetes). This makes it harder for doctors to use research study results when treating their patients.

Project Purpose

The researchers used a statistical approach called subgroup analysis to determine whether the results of a study differ for specific groups of people. Then they developed a computer program to help determine where individual patients fit into the different groups; this program is intended to be used by doctors and patients when making treatment decisions.

The researchers asked whether this approach would be useful for helping clinicians and patients take study results into account when making treatment decisions. They tested this approach with data from a wide variety of medical research studies.

Methods

The researchers looked at results from 32 studies involving treatments for different medical conditions.

Each study

• Had 1,000 or more participants
• Split participants by chance into two or more groups to compare treatments
• Had a specific goal for treatment

The researchers compared the following information in each study:

• Patient details, such as age, gender, and other health conditions
• How well the treatment worked

The researchers made sure the patient information was complete for each study they examined. If some information was missing, then they checked to make sure that the missing information wouldn’t change the results.

The researchers used the information from these comparisons to determine how well each treatment would be expected to work, on average, for most patients. Then they compared this with what happened for different groups of patients in each study.

Findings

The researchers found that, how well treatments worked varied a lot for different groups of people. Often, results from some groups of patients made it look like the treatment was effective for everyone in the study. But many other patients in the study didn’t get much benefit from the treatment. For example, in one diabetes study, most of the benefit was limited to the quarter of participants at highest risk from the disease.

In a few cases, only a small group of patients in the study actually benefited from the treatment at all, but because the benefit for these small groups was so large, it appeared that the treatment worked well for everyone.

Limitations

The researchers only had information about the treatment and the patients that was provided in the publication describing each study. Some studies provided more information than others. If researchers had more information across all studies, they might have had different results. Also, the researchers could have used other proven statistical techniques for studying differences between groups. These techniques might have produced different results.

Importance of Findings

In studies of medical treatments, it is common for treatments to work better for some people than others. If researchers only look at how well a treatment works overall, they might miss important information about who is more likely to benefit from the treatment and who is likely to receive unnecessary treatment.

If researchers used information about patients (such as age, gender, and other health conditions) when looking at the results of studies, which the researchers in this study found is possible to do, they could tell doctors which groups of people are most likely to benefit from a treatment. This information could help doctors and patients make better decisions about whether a treatment is right for an individual patient.

Sharing the Results

The researchers has published journal articles about their research and held a meeting to present the findings of this study to doctors and researchers. The presentation was recorded and is available at http://ilearn.tuftsctsi.org/courses/courseCatalog.aspx.

PCORI has funded the research team to undertake a project to advance the use of these research results to improve health care in real-world settings.

PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Project Purpose

Determining the best treatment for an individual is a fundamentally different problem from determining the best treatment, on average, in a population because individuals vary from each other in important ways that influence the risks and benefits of treatment. This idea is known as heterogeneity of treatment effect (HTE). Although enormous efforts have gone into conducting randomized controlled trials (RCTs) for medical interventions, there has been remarkably little research into how best to use trial results to ensure that as many individual patients as possible receive the most appropriate treatment given their personal characteristics.

Research Objective

Researchers previously proposed a framework for the analysis of HTE. This framework prioritizes the analysis and reporting of multivariate risk-based HTE in the form of predictive risk models (e.g., risk scores, risk calculators). In addition, the researchers proposed that other subgroup analyses should be explicitly labeled either as primary (confirmatory) subgroup analyses or secondary (exploratory) subgroup analyses. The overall objective of this pilot project was to evaluate the feasibility and value of this framework across a large set of RCTs.

Study Design

Reanalysis of RCTs

Participants, Interventions, Settings, and Outcomes

Not applicable because the RCTs included a wide range of participants, interventions, settings, and outcomes.

Data Sources

Researchers used publicly available RCTs from the National Heart, Lung, and Blood Institute, National Institute of Diabetes and Digestive and Kidney Diseases, and other sources. Researchers included data from large RCTs that enrolled at least 1,000 participants, randomized to at least two treatment groups, and had an event or time-to-event clinical outcome.

Data Analysis

Using a set of 32 large trials, researchers developed risk models for the primary outcome of each trial. This method was informed by a set of risk models identified in the literature and matched to the trial. These models were then used to stratify trial participants by their risk of the primary outcome.

To describe and quantify risk heterogeneity, researchers used several novel but intuitive measures. Extreme quartile risk ratio (EQRR) describes the ratio of the outcome rates between the highest risk quartile of patients and the lowest risk quartile. Median-to-mean risk ratio (MMRR) is a measure of skewness calculated as the ratio of the median predicted outcome probability to the mean predicted outcome probability. As the MMRR deviates from one, it reflects the degree to which the summary (average) result may not reflect the effects in the “typical” patient in the trial.

Researchers then analyzed the effect of risk on treatment effect, estimated on relative and absolute scales (i.e., relative and absolute risk reduction). HTE was evaluated by testing for an interaction between predicted risk and treatment assignment, a statistical test for the contrast of the treatment effect on the relative risk (or odds ratio) scale. Researchers also compared relative and absolute risk reduction (ARR) in the extreme risk quartiles.

To verify that the use of internal models would not bias estimates of HTE across risk groups, researchers performed a series of simulations. Across a range of scenarios, analyses based on internal risk models developed on trial participants yield results similar to analyses based on external risk models developed on nonparticipants sampled from the same super-population. To minimize missing data bias, multiple imputation was used when restricting to patients without missing data (i.e., a complete case analysis) would exclude more than 5% of trial participants. To prevent model overfitting, researchers limited the number of variables included in models to maintain an events-per-variable ratio of >20 and used bootstrapping to correct for optimism.

Findings

Researchers identified 32 RCTs that met inclusion criteria, most in the field of cardiovascular disease. Although the typical (median) predicted EQRR was approximately 4, more than a quarter of all trials had an EQRR above 5 and the range exceeded 30. Similarly, although the median MMRR was 0.86 (indicating that the typical patient exhibited an outcome risk 14% lower than the summary [average] risk), this figure ranged from 0.4 to 1. These metrics may demonstrate that risks—defined on the basis of easily obtainable clinical characteristics—often vary tremendously across trial participants, and that the typical patient is usually at lower risk than the average, because most outcomes are accounted for by a group of high risk patients that skew the data.

In the 13 trials with statistically significant results, 18 unique treatment comparisons were analyzed. Overall, there was no apparent relationship between baseline predicted risk and the hazard (or odds) ratios of treatment across trials. Only one of 18 analyses had a significant interaction between treatment and the predicted risk on the proportional scale. ARR varied substantially and was generally higher in high-risk strata. The difference in the ARR between the extreme risk quartiles ranged from 3.2% to 28.3% (median = 5.1%; interquartile range = 0.3–10.9). Thus, in a quarter of analyses, patients in the fourth quartile had an ARR that was at least 10% higher than in the first quartile.

The results of several analyses (Medical Therapy of Prostatic Symptoms, Diabetes Prevention Program, and the Digitalis Investigation Group trial) were so striking that separate clinical manuscripts were published to report the HTE. Each showed that all or most of the benefits of the tested interventions could be obtained by targeting only the high-risk patients (top quartile), without the costs/harms that come with population-wide therapy across all quartiles of predicted risk.

Limitations

Heterogeneity may be slightly overestimated from model overfitting or underestimated from underfitting (i.e., omitting clinically important predictors); more careful model building—exploring nonlinearity and interactions—may have increased the estimation of risk heterogeneity. Researchers did not explore nonlinear relationships between predicted risk and treatment effects, which may have revealed additional HTE.

Conclusions

Clinically significant risk heterogeneity is common in trials. The typical patient may generally be at lower risk than that reflected by the trial summary results. Clinically significant differences in absolute treatment effects within trials appear common and would otherwise be obscured if trials were analyzed in the conventional way. A risk-stratified approach to trial analysis is feasible and may be most clinically informative when the outcome is predictable and uncommon.