• Download Summary

What was the research about?

Comparative effectiveness research compares two or more treatments to see which one works better for certain patients. Researchers often use data from patients’ electronic health records to compare different treatments. This study addresses some problems that can arise from this practice. In some long-term research studies, researchers use data collected when patients in the studies see their doctors. Regularly scheduled doctor visits, called well visits, include yearly checkups or periodic blood pressure checks. Other doctor visits, called sick visits, occur when a patient feels sick or needs special care.

Well and sick visits can produce different types of health record data. In addition, test results at sick visits may be different from results at well visits. Using data from sick visits may inappropriately influence, or bias, a study’s results. Also, patients may go to the doctor more often when they have symptoms or chronic health problems. Researchers may then collect more data from these patients than they collect from the healthier patients. Unequal amounts of data per patient make it harder to compare treatment results.

For this study, the research team created three tests to find if data from sick visits lead to bias in a study’s findings. The team also compared standard and newer statistical methods for analyzing data that include sick visits. Researchers designed the newer methods to reduce bias from data obtained at sick visits. With less biased results, doctors can be more certain about which treatment worked better for certain patients.

What were the results?

• Two of the three tests the research team created were useful for finding out if sick visits were biasing results.
• For analyzing data, the newer statistical methods did not work as well. The standard methods worked better to reduce bias from sick visit data.

What did the research team do?

First, the research team talked with four doctors who oversee clinical databases. The doctors answered questions about how they define well and sick visits with patients. The research team gathered data for 50 to 100 of each doctor’s patients. From these data, the team learned that many sick visits are unplanned. Also, patients may miss regular visits for reasons related to their health conditions. For some treatments, patients may need to see their doctors more often than for other treatments.

Next, the research team created simulated data. These data had known biases, including sick bias, built in. Using these data, the team

• Looked to see if their three new tests detected sick visits that were biasing results
• Compared six statistical methods for looking at data that may include data from sick doctor visits
• Compared standard methods with newer methods that were developed specifically to reduce bias from sick visit data

What were the limits of the study?

The research team may not have captured all reasons why patients see their doctors for sick visits. Also, the simulated data may differ in important ways from data in actual studies.

Future research can apply these methods to data from patients with different health conditions in various kinds of studies.

How can people use the results?

Researchers may use these results to reduce bias in comparative effectiveness studies that include patient data from sick visits. They may use the new tests to identify data that can lead to bias. These methods may help researchers get more accurate findings from patient health records. The more accurate findings may help patients and their doctors compare and choose treatment options.

Objective

To advance statistical methodologies for analyzing data collected during regular health care by (1) comparing test strategies for identifying outcome-dependent visit data and (2) assessing bias in longitudinal statistical estimates arising from clinic data generated at outcome-dependent visits

Study Design

Clinical research uses information collected during patients’ regular health care to support large-scale comparative effectiveness research (CER) and observational studies. Patients with certain conditions often have, in addition to scheduled regular visits, symptom-based clinic visits in which patient characteristics or symptoms relate to the study outcome. For example, a patient who notices condition-related symptoms may schedule a clinic visit outside the regular schedule that yields data related to the CER outcome being measured; a patient without symptoms on a regular visit schedule will not have comparable data. Also, patients with advanced disease may experience higher likelihoods of visits. Researchers call visits, where frequency or content may be symptom driven, outcome-dependent visits.

Ordinary clinic data analyses are subject to systematic error because clinic visit frequency and timing may be related to the longitudinal outcome being studied. In this study, the research team created simulated data to assess bias introduced from outcome-dependent visit data in estimating treatment effects; the team used three standard statistical models and three newer models specifically designed to minimize this bias.

The research team proposed three tests for identifying data that may be affected by the number and timing of outcome-dependent visits. The test statistics incorporated the following patient-level data characteristics: inter-visit time, observed visit time, and total number of visits.

To develop realistic simulations, the research team interviewed a stakeholder panel of four clinician-scientists who oversee clinical databases. The team wanted to determine how regular clinic visits and outcome-dependent clinic visits are defined and to document reasons for outcome-dependent visits. A wide range of simulated conditions included variations in outcome distributions, cluster and per cluster sample sizes, degree of outcome dependence in visit data, and extent to which statistical assumptions were met or violated. Outcome-dependence simulation variations included visit probability based on an underlying health problem or outcome, subgroups with more frequent visits, and varied visit probabilities related to regularly or randomly scheduled visits.

Results

Identifying outcome-dependent data. When the research team analyzed simulated data with a known level of outcome dependence, two of the three test statistics performed well in identifying outcome-dependent visit processes affecting the data: a test based on the total number of visits for each patient and a test based on the random effects portion of the linear predictor.

Bias from outcome-dependent visits. Compared to standard statistical models, the newer models designed to reduce bias from outcome-dependent visits did not perform better across all conditions. Bias primarily emerged in relation to random effects in the model (i.e., intercept, time), not to fixed effects (i.e., group, group by time). The models based on maximum likelihood (ML) exhibited the least susceptibility to bias. Also, results became less biased if the data set included at least a small number of regular visits, rather than outcome-dependent data only.

Limitations

The research team used simulation rather than actual clinical data. Other studies could include different drivers of visit frequency or timing of data availability, and these statistical methods might not apply to those situations. These methods did not consider other models, such as time-to-event models, which could produce different results. The models developed for this study may not capture unknown outcome-dependent visit features that could occur when the model omits important random effects.

Conclusions and Relevance

Because longitudinal data from irregular, outcome-dependent clinic visits can introduce bias, researchers should undertake these analyses with care. Also, researchers should interpret covariates modeled as random effects with caution. These findings indicate that standard ML-based methods are less susceptible to bias than methods purporting to adjust for outcome dependence. The proposed test statistics can screen data for potential outcome-dependent visit processes that may bias analytic results. By reducing bias from outcome-dependent visits in conducting CER, clinicians and patients can ultimately make better informed treatment decisions.

Future Research Needs

Future research into the practical application of these models should assess their validity when assumptions underlying the statistical models are not met. Researchers can apply these models to clinical databases of other diseases.