Results Summary

What was the project about?

Clinical prediction models, or CPMs, are statistical models that can predict a patient’s risk for a specific event, such as a health problem, adverse effect, or even death. To create a CPM, researchers use a single data set, such as data from a clinical trial. To find out whether the CPM accurately predicts risks for patients who weren’t part of the original data, researchers can test the CPM with other data sets. This testing can help researchers know if the CPM is accurate for patients from diverse backgrounds and whether it can be used to make health decisions. But few CPMs have been tested with other data sets.

In this study, the research team used other data sets to look at how well CPMs for heart disease predict patients’ risks. They also looked at how to improve CPMs.

What did the research team do?

First, the research team reviewed existing studies that tested CPMs with other data sets. They found that 58 percent of CPMs had never been tested with other data sets. Also, CPMs varied in how well they predicted patients’ risks when tested with other data sets.

Next, the research team identified 36 other data sets to test 108 heart disease CPMs. To see whether the CPMs accurately predicted patients’ risks, the team tested each CPM with these other data sets. They also looked at whether decisions about health care based on the CPMs would do more harm than good.

Finally, the research team tested three statistical methods to improve CPMs so that they predict risks more accurately.

What were the results?

When tested using other data sets, the selected heart disease CPMs often didn’t accurately predict patients’ risks. For over 80 percent of the CPMs, health decisions based on the CPM would have done more harm than good.

The three statistical methods helped improve the accuracy of the selected CPMs so that fewer CPMs led to decisions that would have done more harm than good.

Working with researchers and doctors, the research team created a website to share information about heart disease CPMs. The website shows how many times a CPM has been tested with other data sets and how well it predicted patients’ risks.

What were the limits of the project?

The data sets used to create the CPMs, and the data sets used to test the CPMs, didn’t always have consistent information. As a result, the research team could only test 108 CPMs. Findings may have differed if the team had tested other CPMs.

Future research could look at why CPM results vary across different patient data sets.

How can people use the results?

Researchers can use the results to test and improve CPMs. Doctors can use the website to learn about heart disease CPMs.

Final Research Report

View this project's final research report.

Journal Citations

Related Journal Citations

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers were generally laudatory in their comments about this study and the potential for the prediction modeling methods.
  • The reviewers asked the researchers to explain their use of the term harm in relation to clinical prediction models, because harm in clinical research typically refers to adverse patient outcomes. The researchers defined harm as a negative result from their decision curve analyses, indicating that using the clinical prediction model to make treatment decisions had net harm rather than net benefit.

Conflict of Interest Disclosures

Project Information

David M. Kent, MD, MS
Tufts Medical Center
$1,472,857
10.25302/09.2021.ME.160635555
How Well Do Clinical Prediction Models (CPMs) Validate? A Large-Scale Evaluation of Cardiovascular Clinical Prediction Models

Key Dates

December 2016
October 2022
2016
2021

Study Registration Information

Tags

Has Results
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: October 18, 2023