Objective
To explore the effect on meta-analyses of using multiple data sources for each clinical trial and to develop guidance on conducting meta-analyses using multiple data sources
Study Design
| Design Element |
Description |
| Design |
Cross-sectional analysis of multiple data sources for clinical trials in two case studies |
| Data Sources and Data Sets |
Public and nonpublic data sources for clinical trials used in meta-analyses |
| Analytic Approach |
- Assess completeness and consistency of information in different data sources for the same clinical trial by comparing meta-analysis outcomes and reviewing methodological quality using the Cochrane Risk of Bias Tool
- Random-effects model meta-analyses
|
| Outcomes |
Completeness and consistency of data sources, and meta-analytic findings using information obtained from public and nonpublic data sources of clinical trial results
|
The research team examined the effect of adding and replacing public and nonpublic data sources for each clinical trial included in meta-analyses for two case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. The research team conducted searches to identify all sources of information about randomized clinical trials for these two interventions and determined whether these trials included patient-centered outcomes. The research team included patient co-investigators and clinical experts.
The research team searched for public sources of clinical trial results, such as study registries, reviews from the US Food and Drug Administration (FDA), conference abstracts, and journal articles. The team also searched for nonpublic sources of study results, such as clinical study reports and individual patient data. The team assessed the completeness and consistency of the methodological information in each data source, including information about interventions, comparators, outcomes, and results. To examine the effect of using multiple data sources for each clinical trial, the research team conducted a series of meta-analyses in which the team added or replaced data from each of the sources for one domain, measure, or time point.
The team included outcomes in the meta-analyses if the outcomes included a point estimate and a measure of precision, such as a standard error or confidence interval.
Results
Number of sources found. The research team identified 80 data sources for 21 trials of gabapentin (from 1997 to 2013) and 51 data sources for 7 trials of quetiapine (from 2003 to 2014). The team found more than one data source for 71 percent of the gabapentin trials and 100 percent of the quetiapine trials. The most commonly available data sources were journal articles and conference abstracts. The team did not include 9 of the 21 gabapentin trials and 4 of the 7 quetiapine trials in their meta-analyses because no data sources for those trials had sufficient statistical information.
Consistency of information. Data sources varied considerably in the amount of information they included about methods or results. In some cases, different data sources had different information about the same trial. Clinical study reports were consistently the most complete source of information about methods and patient-centered outcomes. Among public data sources, journal articles typically contained the most information about methods and patient-centered outcomes, but these articles often lacked sufficient statistical information to include results in meta-analyses.
Impact on meta-analysis findings. For gabapentin, adding nonpublic data reduced the magnitude of the effect size from -0.46 for a meta-analysis drawing only on peer-reviewed journal articles to -0.31 for a meta-analysis using all available sources. However, including nonpublic data did not affect the statistical significance of the intervention effect. For quetiapine, there were no meaningful differences when adding and replacing data from public and nonpublic sources.
Methods guidance. The research team developed guidance for researchers who conduct meta-analyses. The guidance is intended to help researchers deal with two issues. First, accessing, obtaining, and analyzing public and nonpublic data sources require significant resources and skills. Second, information varies considerably across sources.
Limitations
The study looked at using public and nonpublic data sources for only two case studies. The study did not examine intervention harms; therefore, researchers don’t know whether differences between public and nonpublic data sources would affect a meta-analysis of harms.
Conclusions and Relevance
Using public and nonpublic data sources in meta-analyses may give researchers more information about trial methods and results than using a single source of information does. Using both public and nonpublic data may also affect the magnitude of findings in meta-analyses. However, including nonpublic data sources in meta-analyses may require researchers to expend greater effort to extract relevant data. In addition, public and nonpublic data sources may provide inconsistent information for the same trial.
Future Research Needs
Future research could explore using public and nonpublic data sources in additional case studies to determine the generalizability of results. Future research could also examine the effects of using nonpublic data sources in meta-analyses of potential harms.
