Results Summary
What was the project about?
Benchmarks are measures that are used to compare the quality of care patients receive across clinics. Doctors can also use benchmarks to see how well a treatment is working for patients.
In diabetes care, one benchmark is whether patients at a clinic reach their HbA1c goals. HbA1c is a blood test that measures how well a patient’s blood sugar is controlled over three months. Current HbA1c benchmarks show average results across patients. But they don’t account for patient traits, such as age or number of health problems, which may affect how well treatments work.
In this study, the research team used statistical methods to look at how patient traits affect diabetes treatment. The team used these results to create HBA1c benchmarks for looking at quality of care for patients with different traits. These benchmarks may also help doctors and patients with treatment planning.
What did the research team do?
The research team used two data sets. Both had information about patient traits, treatments, and results from HbA1c tests for patients with type 1 or type 2 diabetes. One data set had information from 19 research studies with 6,954 patients. The second had electronic health record, or EHR, data for 8,107 patients in Massachusetts. Using statistical methods, the team predicted how likely patients were to reach HbA1c goals based on patient traits and the treatment received.
From these results, the research team created two benchmarks that took patient traits into account. The benchmarks show how well a treatment will work for a patient with certain traits to reach their HbA1c goal.
A 12-member advisory group of patients with diabetes, family caregivers, and healthcare providers gave input during the study.
What were the results?
Many traits affected how well patients responded to diabetes treatments, including
- Personal traits, such as age, race, ethnicity, education, and income
- Physical traits, such as blood pressure or HbA1c results before treatment
- Patients’ quality of life, such as mental stress or how much symptoms affect the patient
- How satisfied patients were with their treatment
The research team created online benchmark calculators for doctors and patients to compare quality of care across clinics. To help doctors and patients with treatment planning, the calculators show which patient traits can affect how well a treatment works.
What were the limits of the project?
The study used data from research studies and EHR data from one state. Using other data may have led to different results. The research studies took place from 1999 to 2010 and didn’t include newer diabetes treatments.
Future research could include newer research study data and EHR data from other states.
How can people use the results?
Doctors can use the results to compare quality of care and inform treatment decisions for diabetes care.
Professional Abstract
Background
Creating benchmarks that compare quality of care across clinics for the typical patient can help clinicians recognize performance gaps and improve diabetes care. Professional medical organizations recommend using glycemic target goals based on average glycosylated hemoglobin (HbA1c) levels to guide clinical care. Although these goals serve as a benchmark for the average patient, they do not account for patient-specific clinical, demographic, and socioeconomic factors that may affect treatment response.
Objective
(1) To describe patient-specific factors that affect response to diabetes treatments; (2) To develop performance benchmarks related to HbA1c goals that account for patient-specific factors to improve quality of care and clinical decision making for diabetes treatments
Study Design
| Design Element | Description |
|---|---|
| Design | Empirical analysis |
| Data Sources and Data Sets |
|
| Analytic Approach |
|
| Outcomes | Change in HbA1c, odds of achieving target HbA1c, and time to target HbA1c |
Methods and Results
Researchers used pooled data from 19 randomized controlled trials (RCTs) and electronic health record (EHR) data from patients with diabetes. They estimated regression coefficients to predict the effects of patient-specific factors, including sociodemographic and clinical characteristics, patient-reported quality of life, and treatment satisfaction on achieving two target glycemic control outcomes of HbA1c (HbA1c <8% and HbA1c <7%).
Across RCT and EHR data, several patient-specific factors contributed to heterogeneity of treatment response: age, sex, race, ethnicity, baseline HbA1c, body mass index, income, education, patient-reported quality of life, and treatment satisfaction.
The estimated probabilities from the regression provided benchmarks that accounted for patient-specific factors for HbA1c goals. Researchers specified two benchmarks:
- Best practice, based on estimates from the RCT data, which allow clinics to compare performance to the best possible outcomes in a controlled trial environment
- Usual practice, based on estimates from the EHR data, which allow clinics to compare performance with similar clinics under real-world conditions
Researchers used results from the analysis to develop online benchmark calculators that help clinicians examine the effects of changes in patient-specific factors on achieving HbA1c goals.
A 12-member advisory group, including patients with diabetes, family caregivers, healthcare providers, and other professionals provided input on the study.
Limitations
The study used data from 19 RCTs and a single EHR system; neither the RCTs nor the EHR system were nationally representative samples. The RCT data were from studies conducted from 1999 to 2010; these studies did not include newer medications.
Conclusions and Relevance
Patient-specific factors influence diabetes treatment response. This study created two benchmarks for achieving HbA1c goals that account for patient-specific factors.
Future Research Needs
Future research could validate the benchmarks using newer and additional data to improve generalizability of the results.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers commented that the report seemed to present a predictive model for hemoglobin A1c (HbA1c) improvements based on analyses from electronic health records. The reviewers expressed concern because electronic health record data would be prone to potential bias because of unmeasured confounders and missing data for predictors and outcomes. The researchers revised their description of the electronic health records analysis to clarify that their probability models were meant to be explanatory rather than predictive. In other words, they were answering the question, “What happened?” rather than, “What will happen?”
- The reviewers asked whether the researchers tested their probability model for external validity, or the application of the model in a new population to see how well it predicts the same outcome. The researchers initially aimed to test external validity by applying the model they built using data from the randomized controlled trial to community data from patients’ electronic health records. The reviewers pointed out that the two data sets were not equivalent: the trial participants were more homogenous because of strict trial inclusion criteria compared to the community participants. In addition, the follow-up periods for the randomized controlled trial were more consistent, with shorter intervals, than the follow-up periods for the data collected from community electronic health records. The researchers therefore built a separate probability model for reaching the desired HbA1c levels for community patients using data from the electronic health records cohort of patients. The researchers tested neither probability model with a new population, and they identified the lack of testing as a limitation of the study.
- The reviewers noted that the researchers did not follow current standards for missing data when they used the most recent preceding available observation of HbA1c, or the Last Observation Carried Forward, to replace missing HbA1c data at the study’s endpoint. The researchers contended that this technique was valid in this situation because HbA1c is a summary measure of glycemic control over a three-month period and that this technique was conservative because HbA1c was likely to have improved if the study endpoint measure was collected. The reviewers disagreed with this contention and requested the researchers to acknowledge that their methods to address missing data were potentially biased, which was a limitation of the study.