Skip to main content
Patient-Centered Outcomes Research Institute
Patient-Centered Outcomes Research Institute
  • Blog
  • Newsroom
  • Find It Fast
  • Help Center
  • Subscribe
  • Careers
  • Contact Us

PCORI

Patient-Centered Outcomes Research Institute

Search form

  • About Us
    Close mega-menu

    About Us

    • Our Programs
    • Governance
    • Financials and Reports
    • Procurement Opportunities
    • Our Staff
    • Our Vision & Mission
    • Contact Us

    Fact Sheets: Learn More About PCORI

    Download fact sheets about out work, the research we fund, and our programs and initiatives.

    Find It Fast

    Browse through an alphabetical list of frequently accessed and searched terms for information and resources.

    Subscribe to PCORI Email Alerts

    Sign up for weekly emails to stay current on the latest results of our funded projects, and more.

  • Research & Results
    Close mega-menu

    Research & Results

    • Explore Our Portfolio
    • Evaluating Our Work
    • Research Results Highlights
    • Putting Evidence to Work
    • Peer Review
    • Evidence Synthesis
    • About Our Research

    Evidence Updates from PCORI-Funded Studies

    These updates capture highlights of findings from systematic reviews and our funded research studies.

    Journal Articles About Our Funded Research

    Browse through a collection of journal publications that provides insights into PCORI-funded work.

    Explore Our Portfolio of Funded Projects

    Find out about projects based on the health conditions they focus on, the state they are in, and if they have results.

  • Topics
    Close mega-menu

    Topics

    • Addressing Disparities
    • Arthritis
    • Asthma
    • Cancer
    • Cardiovascular Disease
    • Children's Health
    • Community Health Workers
    • COVID-19
    • Dementia and Cognitive Impairment
    • Diabetes
    • Intellectual and/or Developmental Disabilities
    • Kidney Disease
    • Maternal Morbidity and Mortality
    • Medicaid
    • Men's Health
    • Mental and Behavioral Health
    • Minority Mental Health
    • Multiple Chronic Conditions
    • Multiple Sclerosis
    • Obesity
    • Older Adults' Health
    • Pain Care and Opioids
    • Rare Diseases
    • Rural Health
    • Shared Decision Making
    • Telehealth
    • Transitional Care
    • Veterans Health
    • Women's Health

    Featured Topic: Women's Health

    Learn more about the projects we support on conditions that specifically or more often affect women.

  • Engagement
    Close mega-menu

    Engagement

    • The Value of Engagement
    • Engagement in Health Research Literature Explorer
    • Influencing the Culture of Research
    • Engagement Awards
    • Engagement Resources
    • Engage with Us

    Engagement Tools and Resources for Research

    This searchable peer-to-peer repository includes resources that can inform future work in patient-centered outcomes research.

    Engagement Awards

    Learn about our Engagement Awards program and view the announcements of all our open funding opportunities.

    Research Fundamentals: A New On-Demand Training

    It enables those new to health research or patient-centered research to learn more about the research process.

  • Funding Opportunities
    Close mega-menu

    Funding Opportunities

    • What & Who We Fund
    • What You Need to Know to Apply
    • Applicant Training
    • Merit Review
    • Awardee Resources
    • Help Center

    PCORI Funding Opportunities

    View and learn about the newly opened funding announcements and the upcoming PFAs in 2021.

    Tips for Submitting a Responsive LOI

    Find out what PCORI looks for in a letter of intent (LOI) along with other helpful tips.

    PCORI Awardee Resources

    These resources can help awardees in complying with the terms and conditions of their contract.

  • Meetings & Events
    Close mega-menu

    Meetings & Events

    • Upcoming
    • Past Events

    PCORI 2021 and Beyond

    During this webinar, PCORI leaders shared ways to get involved in PCOR, improvements to our funding opportunities, and more.

    2020 PCORI Annual Meeting

    Watch recordings of sessions and view titles and descriptions of posters presented at the virtual meeting.

    Board Approves Future PFA Topics at April Meeting

    The more than a dozen high-priority research topics will be considered for PCORI Funding Announcements (PFAs) that will be released this fall and in 2022.

You are here

  • Research & Results
  • Explore Our Portfolio
  • Statistical Methods for Handling Miss...

This project has results

Statistical Methods for Handling Missing Data in Long-Term Clinical Research

Sign Up for Updates to This Study  

Results Summary and Professional Abstract

Results Summary
Download Summary

Results Summary

What was the research about?

Researchers can use data from health registries or electronic health records to compare two or more treatments. Registries store data about patients with a specific health problem. These data include how well those patients respond to treatments and information about patient traits, such as age, weight, or blood pressure. But sometimes data about patient traits are missing.

Missing data about patient traits can lead to incorrect study results, especially when traits change over time. For example, weight can change over time, and the patient may not report their weight at some points along the way. Researchers use statistical methods to fill in these missing data.

In this study, the research team compared a new statistical method to fill in missing data with traditional methods. Traditional methods remove patients with missing data or fill in each missing number with a single estimate. The new method creates multiple possible estimates to fill in each missing number.

What were the results?

The new method worked better than traditional methods. Patients with certain traits were more likely to have missing data. Accounting for such traits in the analysis made the new method work even better.

What did the research team do?

The research team first developed computer software, which created test data that mimicked a study about patients living with HIV. The data included patient traits that change over time and are often missing, such as body mass index, a measure of weight and height. The team used the test data to compare the new method of filling in missing data with traditional methods.

What were the limits of the study?

When the research team compared the methods, they looked at the test data only in certain situations. For example, because patients with certain traits were more likely to have missing data, they compared results with and without those patients. Findings may differ if the team looked at the test data in other situations.

Future research could compare methods for handling missing data in other situations.

How can people use the results?

Researchers can use the new method to get results that are more accurate in studies with missing data.

Professional Abstract

Professional Abstract

Objective

To evaluate statistical approaches for handling missing data in longitudinal studies of comparative effectiveness research by (1) developing software packages to generate realistically complex data and (2) comparing multiple imputation (MI)-based strategies with commonly used approaches to analyzing data with missing time-varying covariates

Study Design

Design Elements Description
Design Simulation studies
Data Sources and Data Sets Simulated data that resemble the complexity of an empirical study to identify antiretroviral therapies associated with increased risk of cardiovascular disease among patients with HIV
Analytic Approach Simulations were conducted using software packages developed by the research team to examine the following approaches: MI, complete-case analysis, and 3 single imputation methods including mean imputation, last value carried forward, and imputation using missing indicators
Outcomes Bias, average standard error, coverage probability, relative mean squared error

In longitudinal observational data sets, missing data can compromise analyses and bias results. They also pose a unique challenge in that longitudinal studies induce a correlated or clustered data structure. Traditional methods for handling missing data include complete-case analysis, which excludes patients with missing data from the analysis, and single imputation methods, which substitute each missing value with a single value. The MI method generates multiple data sets with imputed values and combines analysis results from these data sets.

Few studies have examined how well MI performs compared with traditional methods in longitudinal studies with missing time-varying covariates and a right-censored outcome, where records are correlated within subjects over time. Time-varying covariates are variables that may change over time, such as a patient’s medical regimens. A longitudinal study outcome is right-censored when the outcome event, such as death, does not occur for all individuals within the study period.

The research team developed two software packages in the R software, called SimTimeVar and SimulateCER, to simulate comparative effectiveness studies with time-varying covariates. They used a software package called PermAlgo to simulate the right-censored outcome. In addition, the team generated auxiliary variables that can help impute the missing data in the MI process but are not part of the main analysis. The software packages generated data that mimic an empirical HIV study. Then, based on the simulated data, the team compared the performance of the MI-based methods with traditional methods, including complete-case analysis and single imputation methods. The team examined the methods under different scenarios, including with and without auxiliary variables, with and without missing values for the covariate of interest, and across missing data mechanisms with different complexity levels.

Results

Software development. The software packages successfully created data sets with right-censored outcomes as well as time-varying and static covariates.

Comparison of methods. For time-varying covariates where the covariate of interest has no missing values, MI outperformed complete-case analysis and single imputation methods. By including auxiliary variables, MI methods performed well even in complex missing-at-random and not-missing-at-random conditions.

For time-varying covariates where the covariate of interest has missing values, the last-value-carried-forward method outperformed other methods if the outcome did not have a strong increasing or decreasing pattern over time. As the missing data mechanism became more complex, the bias increased across all methods. In general, MI outperformed other methods, especially if it included an auxiliary variable strongly related to the missing data mechanism.

Limitations

The research team used simulated data sets to explore a variety of realistic scenarios. However, results may not generalize to scenarios not examined in this study. The team did not evaluate MI methods that accounted for clustered data structures because of computational limitations.

Conclusions and Relevance

The MI method was advantageous under most scenarios and could enable more robust and valid results in analyses of longitudinal studies with missing time-varying covariates. Including auxiliary variables, especially those related to the missing data mechanism, could greatly improve the performance of MI.

Future Research Needs

Future research could evaluate the MI method under other simulation scenarios. Additional research could also develop software to implement MI methods that account for clustered data structures in complex comparative effectiveness studies.

Final Research Report

View this project's final research report.

Journal Articles

Results of This Project

Kidney International

Reduced risk of myocardial infarct and revascularization following coronary artery bypass grafting compared with percutaneous coronary intervention in patients with chronic kidney disease

Related Articles

Statistics in Medicine

Missing data strategies for time-varying confounders in comparative effectiveness studies of non-missing time-varying exposures and right-censored outcomes

Journal of Modern Applied Statistical Methods

Guidelines for generating rightcensored outcomes from a Cox model extended to accommodate time-varying covariates

Journal of Modern Applied Statistical Methods

Multiple Imputation When Rate of Change Is The Outcome of Interest

Statistical Methodology

Navigating choices when applying multiple imputation in the presence of multi-level categorical interaction effects

Clinical Infectious Diseases

Risk of cardiovascular events associated with current exposure to HIV antiretroviral therapies in a US veteran population

Clinical Journal of the American Society of Nephrology

Addressing missing data in clinical studies of kidney diseases

More on this Project  

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented, and the researchers made changes or provided responses. The comments and responses included the following:

  • Reviewers requested additional information about the missingness models, simulations, and model assumptions. The reviewers noted that it was hard to keep track of the specifics of each simulation study and the results obtained. The researchers added an appendix including a table that summarizes the three main simulation studies, their assumptions, the methods evaluated, and the findings.
  • Reviewers asked for additional details about the scenarios used to test the model simulations, to help readers better understand the limitations to the generalizability of these models. The researchers responded by adding discussion of the proportion of missing records and the missing data mechanism. However, they disagreed with the reviewers’ contention that the discrepancies between simulated and observed data limited the generalizability of their findings regarding performance of missing data methods.
  • Reviewers noted that the simulations of time-dependent categorical variables did not appear to perform well and asked for additional discussion of the impact of this poor performance on conclusions that could be made from these simulations. The researchers acknowledged that the performance of the simulation tool was not optimal in generating covariates but did not consider this an important issue because the simulation of survival times was excellent. They also noted excellent recovery of true parameter estimates of the relationship between the time-dependent covariates and outcome, which are critical to gauging how well missing data methods handle time-varying covariates.

Conflict of Interest Disclosures

View the COI disclosure form.

 

Project Details

Principal Investigator
Manisha Desai, PhD
Project Status
Completed; PCORI Public and Professional Abstracts, and Final Research Report Posted
Project Title
The Handling of Missing Data Induced by Time-Varying Covariates in Comparative Effectiveness Research Involving HIV Patients
Board Approval Date
September 2013
Project End Date
January 2019
Organization
Stanford University
Year Awarded
2013
State
California
Year Completed
2018
Project Type
Research Project
Health Conditions  
Cardiovascular Diseases
Coronary or Ischemic Heart Disease
Infectious Diseases
HIV/AIDS
Multiple/Comorbid Chronic Conditions
Funding Announcement
Improving Methods for Conducting Patient-Centered Outcomes Research
Project Budget
$802,122
Study Registration Information
HSRP20143589
Page Last Updated: 
August 9, 2020

About Us

  • Our Programs
  • Governance
  • Financials and Reports
  • Procurement Opportunities
  • Our Staff
  • Our Vision & Mission
  • Contact Us

Research & Results

  • Explore Our Portfolio
  • Evaluating Our Work
  • Research Results Highlights
  • Putting Evidence to Work
  • Peer Review
  • Evidence Synthesis
  • About Our Research

Engagement

  • The Value of Engagement
  • Engagement in Health Research Literature Explorer
  • Influencing the Culture of Research
  • Engagement Awards
  • Engagement Resources
  • Engage with Us

Funding Opportunities

  • What & Who We Fund
  • What You Need to Know to Apply
  • Applicant Training
  • Merit Review
  • Awardee Resources
  • Help Center

Meetings & Events

April 27
Priorities on the Health Horizon: Informing PCORI's Strategic Plan (Webinar #2)
May 6
Advisory Panel on Healthcare Delivery and Disparities Research Spring 2021 Meeting
May 10
Cycle 2 2021 Nonsurgical Options for Women with Urinary Incontinence -- Applicant Town Hall

PCORI

Footer contact address

Patient-Centered Outcomes
Research Institute

1828 L Street, NW, Suite 900
Washington, DC 20036
Phone: (202) 827-7700 | Fax: (202) 355-9558
info@pcori.org

Subscribe to Newsletter

Twitter Facebook LinkedIn Vimeo

© 2011-2021 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Privacy Policy | Terms of Use | Trademark Usage Guidelines | Credits | Help Center