Results Summary

What was the project about?

This project aims to improve the methods that researchers use to compare how treatments affect different patients. When researchers use data from patients’ health records to compare treatments, it’s often hard to know whether changes in a patient’s health are from the treatment or something else. Factors other than the treatment may affect the patient’s health, including

  • A patient’s traits, such as age, gender, or other health problems
  • Group-level factors, such as where patients get care or where they live

To address this problem, researchers rely on statistical methods. Existing methods use data from patients who have similar traits but received different treatments. But they may not work well if some group-level factors affect both the treatment and patients’ health. In this study, the research team created two new ways of including group-level factors in the methods they use to find similar patients.

What did the research team do?

The research team made test data to look like patient health records. The test data included information on patient traits, group-level factors, and treatments. The team picked one group-level factor, such as hospital size, that can affect treatment results. Then the team created two new ways of including the group-level factor in the methods they use to find similar patients. To see which ways worked best, the team compared findings from the two new ways to those from an existing way of including group-level factors. Finally, the team compared findings from all three ways of including the group-level factor with findings from not including the group-level factor.

One parent, one clinician, and one researcher helped design the study.

What were the results?

Including group-level factors in the analysis got more accurate results than not including them. Among the three ways of including group-level factors, the existing way and one of the new ways provided more accurate findings about how treatments work.

What were the limits of the project?

The test data set had only one group-level factor. Results might have been different with real health record data with more than one group-level factor.

Future research could create and test ways of including more than one group-level factor.

How can people use the results?

Researchers can use methods that include group-level factors to get more accurate findings about how treatments work.

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers asked for a statistical description of the overlap of estimated propensity scores within each cluster group. The researchers added figures showing the distribution of estimated propensity scores to illustrate the overlaps in the simulations and data examples.
  • The reviewers asked that the limitations sections in the abstract and discussion sections be expanded to include some of the statistical limitations they had described. The researchers did so, and added a description of the usefulness of also examining full matching to the limitations section.
  • The reviewers asked why particular previously published methods, including preferential matching, were not used in simulations. The researchers explained that one previously published method was not used in simulations here because of differences in the data types used. Regarding preferential matching, the researchers added text to the background section explaining how preferential matching could be integrated with the hierarchical-matching method that performed best in this study. The researchers said that in future work they will compare preferential matching with preferential implementation of the hierarchical-matching method they used in this report.

Conflict of Interest Disclosures

Project Information

Mi-Ok Kim, PhD
University of California, San Francisco^
Propensity Score-Based Methods for CER Using Multilevel Data: What Works Best When

Key Dates

September 2014
September 2019

Study Registration Information

^Mi-Ok Kim, PhD, was affiliated with Cincinnati Children's Hospital Medical Center in Ohio when this study was initially awarded.


Has Results
Award Type
Health Conditions Health Conditions These are the broad terms we use to categorize our funded research studies; specific diseases or conditions are included within the appropriate larger category. Note: not all of our funded projects focus on a single disease or condition; some touch on multiple diseases or conditions, research methods, or broader health system interventions. Such projects won’t be listed by a primary disease/condition and so won’t appear if you use this filter tool to find them. View Glossary
Intervention Strategy Intervention Strategies PCORI funds comparative clinical effectiveness research (CER) studies that compare two or more options or approaches to health care, or that compare different ways of delivering or receiving care. View Glossary
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: April 11, 2024