Results Summary

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for certain patients. Researchers often use data from patients’ electronic health records to compare different treatments. This study addresses some problems that can arise from this practice. In some long-term research studies, researchers use data collected when patients in the studies see their doctors. Regularly scheduled doctor visits, called well visits, include yearly checkups or periodic blood pressure checks. Other doctor visits, called sick visits, occur when a patient feels sick or needs special care.

Well and sick visits can produce different types of health record data. In addition, test results at sick visits may be different from results at well visits. Using data from sick visits may inappropriately influence, or bias, a study’s results. Also, patients may go to the doctor more often when they have symptoms or chronic health problems. Researchers may then collect more data from these patients than they collect from the healthier patients. Unequal amounts of data per patient make it harder to compare treatment results.

For this study, the research team created three tests to find if data from sick visits lead to bias in a study’s findings. The team also compared standard and newer statistical methods for analyzing data that include sick visits. Researchers designed the newer methods to reduce bias from data obtained at sick visits. With less biased results, doctors can be more certain about which treatment worked better for certain patients.

What were the results?

  • Two of the three tests the research team created were useful for finding out if sick visits were biasing results.
  • For analyzing data, the newer statistical methods did not work as well. The standard methods worked better to reduce bias from sick visit data.

What did the research team do?

First, the research team talked with four doctors who oversee clinical databases. The doctors answered questions about how they define well and sick visits with patients. The research team gathered data for 50 to 100 of each doctor’s patients. From these data, the team learned that many sick visits are unplanned. Also, patients may miss regular visits for reasons related to their health conditions. For some treatments, patients may need to see their doctors more often than for other treatments.

Next, the research team created simulated data. These data had known biases, including sick bias, built in. Using these data, the team

  • Looked to see if their three new tests detected sick visits that were biasing results
  • Compared six statistical methods for looking at data that may include data from sick doctor visits
  • Compared standard methods with newer methods that were developed specifically to reduce bias from sick visit data

What were the limits of the study?

The research team may not have captured all reasons why patients see their doctors for sick visits. Also, the simulated data may differ in important ways from data in actual studies.

Future research can apply these methods to data from patients with different health conditions in various kinds of studies.

How can people use the results?

Researchers may use these results to reduce bias in comparative effectiveness studies that include patient data from sick visits. They may use the new tests to identify data that can lead to bias. These methods may help researchers get more accurate findings from patient health records. The more accurate findings may help patients and their doctors compare and choose treatment options.

Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.

Peer review identified the following strengths and limitations in the report:

  • The reviewers noted that the results of the simulation studies performed as part of this methods research project were affected by the possibility of incorrect identification of the outcome, as well as by missing data. The researcher disagreed, noting that in sensitivity analyses, they tested violations of their assumptions about outcome tracking and missing data, and found that results did not change substantively.
  • The reviewers expressed concern that the comparison methods for measuring outcome-dependent visits had assumed missing data were missing at random. This might lead to overestimating the value of the researchers’ approach to this problem. The authors defended their chosen comparisons by explaining the steps they took to assure that these comparison methods were representative and not overly biased.
  • The authors added a limitation to their discussion noting that although they included generalized linear mixed models—which are used widely in comparative effectiveness research—in their investigations, they did not consider time-to-event models in their comparisons. The researchers acknowledged that time-dependent models were important for the field, but were beyond the scope of this study.
  • The reviewers requested more information on the simulation studies the researchers performed as well as the proposed method for handling outcome-dependent visit analyses, and the authors added extensive appendices to report the simulations.

Conflict of Interest Disclosures

Project Information

Charles Elliott McCulloch, PhD
University of California San Francisco
$863,910 *
Methods for Analysis and Interpretation of Data Subject to Informative Visit Times

Key Dates

December 2013
July 2018

Study Registration Information

Final Research Report

View this project's final research report.

Journal Articles


Has Results
Award Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: January 20, 2023