Results Summary

What was the project about?

Benchmarks are measures that are used to compare the quality of care patients receive across clinics. Doctors can also use benchmarks to see how well a treatment is working for patients.

In diabetes care, one benchmark is whether patients at a clinic reach their HbA1c goals. HbA1c is a blood test that measures how well a patient’s blood sugar is controlled over three months. Current HbA1c benchmarks show average results across patients. But they don’t account for patient traits, such as age or number of health problems, which may affect how well treatments work.

In this study, the research team used statistical methods to look at how patient traits affect diabetes treatment. The team used these results to create HBA1c benchmarks for looking at quality of care for patients with different traits. These benchmarks may also help doctors and patients with treatment planning.

What did the research team do?

The research team used two data sets. Both had information about patient traits, treatments, and results from HbA1c tests for patients with type 1 or type 2 diabetes. One data set had information from 19 research studies with 6,954 patients. The second had electronic health record, or EHR, data for 8,107 patients in Massachusetts. Using statistical methods, the team predicted how likely patients were to reach HbA1c goals based on patient traits and the treatment received.

From these results, the research team created two benchmarks that took patient traits into account. The benchmarks show how well a treatment will work for a patient with certain traits to reach their HbA1c goal.

A 12-member advisory group of patients with diabetes, family caregivers, and healthcare providers gave input during the study.

What were the results?

Many traits affected how well patients responded to diabetes treatments, including

  • Personal traits, such as age, race, ethnicity, education, and income
  • Physical traits, such as blood pressure or HbA1c results before treatment
  • Patients’ quality of life, such as mental stress or how much symptoms affect the patient
  • How satisfied patients were with their treatment

The research team created online benchmark calculators for doctors and patients to compare quality of care across clinics. To help doctors and patients with treatment planning, the calculators show which patient traits can affect how well a treatment works.

What were the limits of the project?

The study used data from research studies and EHR data from one state. Using other data may have led to different results. The research studies took place from 1999 to 2010 and didn’t include newer diabetes treatments.

Future research could include newer research study data and EHR data from other states.

How can people use the results?

Doctors can use the results to compare quality of care and inform treatment decisions for diabetes care. 

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers commented that the report seemed to present a predictive model for hemoglobin A1c (HbA1c) improvements based on analyses from electronic health records. The reviewers expressed concern because electronic health record data would be prone to potential bias because of unmeasured confounders and missing data for predictors and outcomes. The researchers revised their description of the electronic health records analysis to clarify that their probability models were meant to be explanatory rather than predictive. In other words, they were answering the question, “What happened?” rather than, “What will happen?”
  • The reviewers asked whether the researchers tested their probability model for external validity, or the application of the model in a new population to see how well it predicts the same outcome. The researchers initially aimed to test external validity by applying the model they built using data from the randomized controlled trial to community data from patients’ electronic health records. The reviewers pointed out that the two data sets were not equivalent: the trial participants were more homogenous because of strict trial inclusion criteria compared to the community participants. In addition, the follow-up periods for the randomized controlled trial were more consistent, with shorter intervals, than the follow-up periods for the data collected from community electronic health records. The researchers therefore built a separate probability model for reaching the desired HbA1c levels for community patients using data from the electronic health records cohort of patients. The researchers tested neither probability model with a new population, and they identified the lack of testing as a limitation of the study.
  • The reviewers noted that the researchers did not follow current standards for missing data when they used the most recent preceding available observation of HbA1c, or the Last Observation Carried Forward, to replace missing HbA1c data at the study’s endpoint. The researchers contended that this technique was valid in this situation because HbA1c is a summary measure of glycemic control over a three-month period and that this technique was conservative because HbA1c was likely to have improved if the study endpoint measure was collected. The reviewers disagreed with this contention and requested the researchers to acknowledge that their methods to address missing data were potentially biased, which was a limitation of the study.

Conflict of Interest Disclosures

Project Information

Marcia Anne Testa, BA, MPH, MPHIL, PhD
Harvard University School of Public Health
$2,025,244
10.25302/03.2021.CE.13046756
Benchmarking the Comparative Effectiveness of Diabetes Treatments Using Patient-Reported Outcomes and Socio-Demographic Factors

Key Dates

September 2013
February 2021
2013
2021

Study Registration Information

Tags

Has Results
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
Populations Populations PCORI is interested in research that seeks to better understand how different clinical and health system options work for different people. These populations are frequently studied in our portfolio or identified as being of interest by our stakeholders. View Glossary
Intervention Strategy Intervention Strategies PCORI funds comparative clinical effectiveness research (CER) studies that compare two or more options or approaches to health care, or that compare different ways of delivering or receiving care. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 25, 2023