Jump to Section
Results Summary and Professional Abstract Final Research Report Peer-Review Summary Conflict of Interest Disclosures Project Information Key Dates Study Registration Information
Research
Completed; PCORI Public and Professional Abstracts, and Final Research Report Posted
Has Results

How Accurate Are PCORnet® Data for Measures of Children's Healthcare Quality? -- A PCORnet® Study

Sign Up for Updates

Results Summary

What was the research about?

Health systems use quality measures to assess how well they deliver care to patients. These measures require accurate data. Health systems often do a manual medical chart review to get accurate data. To do a manual review, staff combine data from a number of patient health records. This process is expensive and takes time.

PCORI funded PCORnet® to make it easier to conduct research. PCORnet is made up of networks of health systems. These networks gather data from electronic health records and transform the data into a common format, called the Common Data Model, or CDM.

In this study, the research team wanted to learn if CDM data are as accurate as data from a manual medical chart review. The team compared how often data from manual chart reviews agreed with CDM data for three quality measures for health problems in children:

  • The percentage of patients with sickle cell anemia who received screenings to prevent stroke
  • The percentage of patients with two or more prescriptions for antipsychotics who received blood sugar and cholesterol tests
  • The proportion of visits for ear infections where children received antibiotic prescriptions that matched recommended care

What were the results?

Overall, CDM data varied in agreement with data from manual chart reviews:

  • Sickle cell anemia. Manual chart review data agreed with CDM data an average of 98 percent of the time when patients received screening and 89 percent of the time when patients didn’t.
  • Antipsychotics. Manual chart review data agreed with CDM data almost always when patients received blood sugar screening and 81 percent of the time when patients didn’t. Chart data also agreed with CDM data almost always when patients received cholesterol screening and 88 percent of the time when patients didn’t.
  • Ear infections. Manual chart review data agreed with CDM data almost always for patients who received antibiotics and 98 percent of the time for patients who didn’t.

Agreement between manual chart reviews and CDM data varied across health systems. For example, when patients received screening for sickle cell anemia, agreement ranged from 90 to 100 percent across health systems. When patients didn’t receive this screening, agreement ranged from 56 to 100 percent across systems.

What did the research team do?

The research team reviewed data from 13 health systems in two PCORnet networks. The networks represent more than 6 million patients. The team then looked at data in medical charts for each health problem. The team compared the CDM data with the medical chart data to see if the data differed.

Parents of patients; health system staff; and clinicians, like doctors and nurses, helped design the study and analyze results.

What were the limits of the study?

The study included data from two networks. Results may differ with data from other patients and settings.

Future research could look at why medical chart and CDM data differ.

How can people use the results?

Health systems can use the results when looking at data for quality measures.

How this project fits under PCORI’s Research Priorities
The PCORnet® Study reported in this results summary was conducted using PCORnet®, the National Patient-Centered Clinical Research Network. PCORnet® is intended to improve the nation’s capacity to conduct health research, particularly comparative effectiveness research (CER), efficiently by creating a large, highly representative network for conducting clinical outcomes research. PCORnet® has been developed with funding from the Patient-Centered Outcomes Research Institute® (PCORI®).

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers asked for more information about the conditions that the researchers considered examining in this project, and how the researchers decided to focus on type 1 diabetes in children as the model condition used in the research. The researchers added information to the report describing the process they underwent to narrow the original eight conditions under consideration to three top priority conditions, and how they used health system leader input to decide to prioritize type 1 diabetes.
  • Reviewers questioned the researchers’ decision to omit outcome data from the National Quality Forum measure on cases of acute otitis media in children and suggested that the researchers describe the results they had on this measure for transparency. The researchers disagreed, explaining in the text the issues that made the validity of the measure questionable. The researchers responded to reviewers that they did not feel they should report data that were inaccurate and probably invalid.
  • The reviewers advised the researchers to remove a section of the report related to US News & World Report rankings of hospitals and hospital departments. This information was initially included because such rankings and benchmarking of hospitals would typically significantly affect referrals and revenue. The reviewers felt that this information heavily involved marketing rather than research and therefore did not belong in the report. The researchers moved this information to an appendix instead.

Conflict of Interest Disclosures

View the COI Disclosure Form

Project Information

Kathleen Walsh, MD
Cincinnati Children's Hospital Medical Center
$1,220,083
10.25302/09.2021.HSD.160435160
Automating Quality and Safety Benchmarking for Children: Meeting the Needs of Health Systems and Patients

Key Dates

June 2016
July 2021
2016
2021

Study Registration Information

HSRP20164074

Tags

Has Results
Project Status
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
Populations Populations PCORI is interested in research that seeks to better understand how different clinical and health system options work for different people. These populations are frequently studied in our portfolio or identified as being of interest by our stakeholders. View Glossary
Funding Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 20, 2023