4: Standards for Preventing and Handling Missing Data

MD-1: Describe methods to prevent and monitor missing data

Investigators should explicitly state potential reasons that study data may be missing. Missing data can occur from patient dropout, nonresponse, data collection problems, incomplete data sources, and/or administrative issues. As relevant, the protocol should include the anticipated amount of and reasons for missing data, plans to prevent missing data, and plans to follow up with study participants. The study protocol should contain a section that addresses steps taken in study design and conduct to monitor and limit the impact of missing data. This standard applies to all study designs for any type of research question.

Public comments

No comments.

MD-2: Use valid statistical methods to deal with missing data that properly account for statistical uncertainty due to missingness

Valid statistical methods for handling missing data should be pre-specified in study protocols. The reasons for missing data should be considered in the analysis. A discussion of the potential ramifications of the statistical approach to missing data on the results should be provided. The plausibility of the assumptions associated with the approach should be assessed. Statistical inference of intervention effects or measures of association should account for statistical uncertainty attributable to missing data. This means that methods used for imputing missing data should produce valid confidence intervals and should permit unbiased inferences based on statistical hypothesis tests. Bayesian methods, multiple imputation, and various likelihood-based methods are valid statistical methods to deal with missing data. Single imputation methods like last observation carried forward, baseline observation carried forward, and mean value imputation are discouraged as the primary approach for handling missing data in the analysis. If investigators do use single-based imputation methods, they must provide a compelling scientific rationale as to why the method is appropriate. This standard applies to all study designs for any type of research question.

Public comments

Along with the addition of “mean value imputation” as one of the examples of handling missing data, AcademyHealth also recommends including ‘hot deck imputation,’ in which each missing value is replaced with an observed response from a “similar” unit. For additional clarification within this standard, PCORI should consider distinguishing between imputation of outcomes versus control variables. Finally, and more generally, we encourage PCORI to push the research community to understand and report the underlying processes of data generation. The application of missing data techniques draws from a solid understanding of these underlying processes, so that the methods employed align with the mechanisms by which the data are missing.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:36pm

Why the mention of "valid" confidence intervals? What would constitute an invalid CI? Also, should we be "discouraging" the use of single imputation methods as the primary approach to handling missing data? Sometimes such methods are indeed preferred. {EBort}

{EBort} Merck & Co Inc, Industry, 02/23/2016 - 2:35pm

MD-3: Record and report all reasons for dropout and missing data, and account for all patients in reports

Whenever a participant drops out of a research study, the investigator should document the following:

1. the specific reason for dropout, in as much detail as possible;
2. who decided that the participant would drop out; and
3. whether the dropout involves some or all types of participation.

Investigators should attempt to continue to collect information on key outcomes for participants unless consent is withdrawn. All participants included in the study should be accounted for in the report, whether or not they are included in the analyses. Describe and justify any planned reasons for excluding participants from analysis. In addition, missing data due to other mechanisms (such as nonresponse and data entry/collection) should also be well documented and handled appropriately in the analyses.

Public comments

AcademyHealth encourages PCORI to make its Methodology Standards as clear as possible, so they are of greatest benefit to the researcher. PCORI’s request that missing data due to other mechanisms be “well documented and handled appropriately” is exceedingly vague and may not elicit the response PCORI seeks. What does PCORI consider well documented? Handled appropriately? This Standard would be enhanced with additional clarification surrounding these points, or removal of qualifiers to simply require documentation of the reasons for missingness. As a final point, we would reiterate the point made on Standard MD-2, that pushing the research community to understand and report the underlying processes of data generation is more important than focusing on making definitive statements about the processes used in the analysis, which depend on this external content.

Lisa Simpson, AcademyHealth, Stakeholder - Other, 04/11/2016 - 4:36pm

MD-4: Examine sensitivity of inferences to missing data methods and assumptions, and incorporate into interpretation

Examining sensitivity to the assumptions about the missing data mechanism (i.e., sensitivity analysis) should be a mandatory component of the study protocol, analysis, and reporting. This standard applies to all study designs for any type of research question. Statistical summaries should be used to describe missing data in studies, including a comparison of baseline characteristics of units (e.g., patients, questions, or clinics) with and without missing data. These quantitative results should be incorporated into the interpretation of the study and reflected in the discussion section and possibly the abstract.

Public comments

No comments.

General feedback on the Standards for Preventing and Handling Missing Data

Public comments

• We recommend adding additional language in these standards regarding data from secondary (claims, EMR) databases. Suggested text: In some cases, there is so much missing data that it may be better to search for another database that is more complete. • In addition, it would be beneficial if the standards contained citations for: -Little RJ et al. The Prevention and Treatment of Missing Data in Clinical Trials. N Engl J Med 2012; 367; 14 1355-1360. -National Research Council. The prevention and treatment of missing data in clinical trials. Washington, DC: National Academies Press, 2010.

Eli Lilly and Company, Industry, 03/30/2016 - 2:17pm