Results Summary
What was the research about?
Meta-analyses combine the results of many studies to find out how well a treatment or other healthcare intervention works. Most meta-analyses use public sources of data, such as published journal articles, as the main sources of information for study results. But journal articles are not the only sources of study results. Some results appear in other places, such as clinical study reports. Clinical study reports are documents that describe what researchers did and found in much more detail than journal articles. However, these reports may not be available to the public. As a result, meta-analyses may not include all available information about a treatment.
The research team wanted to learn whether adding or replacing public and nonpublic data sources changed the results of meta-analyses. To find out, the research team added and replaced data as they conducted two meta-analyses. The first looked at adult use of a nerve-pain medicine. The second meta-analysis looked at adult use of a medicine to treat bipolar depression.
What were the results?
The research team found that adding nonpublic data sources to public ones provided new details about study methods and results. Using all available data sources changed the size of the effect in the nerve-pain meta-analysis but doing so didn’t change the results for the bipolar-depression meta-analysis.
Nonpublic data sources, such as clinical study reports, often had information that medical journal articles lacked. Based on these results, the research team developed guidance for researchers who want to use different kinds of data sources in meta-analyses.
What did the research team do?
The research team asked patients and doctors to make a list of the health outcomes that matter to patients who take the medicines for nerve pain or bipolar depression. Then, the research team searched for studies about the two medicines. The research team looked at whether changing the sources of data on the outcomes that matter to patients changed the results of the meta-analyses.
The team looked at public and nonpublic data sources for the two meta-analyses. The research team found 80 sources for 21 studies about the nerve-pain medicine. The team found 51 sources for 7 studies about the bipolar-depression medicine.
What were the limits of the study?
The study looked at the effect of adding and replacing data sources in only two meta-analyses. Future research could look at whether adding and replacing data sources changes the results for meta-analyses focusing on other treatments or health problems. Also, future research could look at the effect of using nonpublic data sources on meta-analysis findings about treatment harms.
How can people use the results?
Researchers may want to use the results of this study when they look for data sources for meta-analyses.
Professional Abstract
Objective
To explore the effect on meta-analyses of using multiple data sources for each clinical trial and to develop guidance on conducting meta-analyses using multiple data sources
Study Design
| Design Element | Description |
|---|---|
| Design | Cross-sectional analysis of multiple data sources for clinical trials in two case studies |
| Data Sources and Data Sets | Public and nonpublic data sources for clinical trials used in meta-analyses |
| Analytic Approach |
|
| Outcomes |
Completeness and consistency of data sources, and meta-analytic findings using information obtained from public and nonpublic data sources of clinical trial results |
The research team examined the effect of adding and replacing public and nonpublic data sources for each clinical trial included in meta-analyses for two case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. The research team conducted searches to identify all sources of information about randomized clinical trials for these two interventions and determined whether these trials included patient-centered outcomes. The research team included patient co-investigators and clinical experts.
The research team searched for public sources of clinical trial results, such as study registries, reviews from the US Food and Drug Administration (FDA), conference abstracts, and journal articles. The team also searched for nonpublic sources of study results, such as clinical study reports and individual patient data. The team assessed the completeness and consistency of the methodological information in each data source, including information about interventions, comparators, outcomes, and results. To examine the effect of using multiple data sources for each clinical trial, the research team conducted a series of meta-analyses in which the team added or replaced data from each of the sources for one domain, measure, or time point.
The team included outcomes in the meta-analyses if the outcomes included a point estimate and a measure of precision, such as a standard error or confidence interval.
Results
Number of sources found. The research team identified 80 data sources for 21 trials of gabapentin (from 1997 to 2013) and 51 data sources for 7 trials of quetiapine (from 2003 to 2014). The team found more than one data source for 71 percent of the gabapentin trials and 100 percent of the quetiapine trials. The most commonly available data sources were journal articles and conference abstracts. The team did not include 9 of the 21 gabapentin trials and 4 of the 7 quetiapine trials in their meta-analyses because no data sources for those trials had sufficient statistical information.
Consistency of information. Data sources varied considerably in the amount of information they included about methods or results. In some cases, different data sources had different information about the same trial. Clinical study reports were consistently the most complete source of information about methods and patient-centered outcomes. Among public data sources, journal articles typically contained the most information about methods and patient-centered outcomes, but these articles often lacked sufficient statistical information to include results in meta-analyses.
Impact on meta-analysis findings. For gabapentin, adding nonpublic data reduced the magnitude of the effect size from -0.46 for a meta-analysis drawing only on peer-reviewed journal articles to -0.31 for a meta-analysis using all available sources. However, including nonpublic data did not affect the statistical significance of the intervention effect. For quetiapine, there were no meaningful differences when adding and replacing data from public and nonpublic sources.
Methods guidance. The research team developed guidance for researchers who conduct meta-analyses. The guidance is intended to help researchers deal with two issues. First, accessing, obtaining, and analyzing public and nonpublic data sources require significant resources and skills. Second, information varies considerably across sources.
Limitations
The study looked at using public and nonpublic data sources for only two case studies. The study did not examine intervention harms; therefore, researchers don’t know whether differences between public and nonpublic data sources would affect a meta-analysis of harms.
Conclusions and Relevance
Using public and nonpublic data sources in meta-analyses may give researchers more information about trial methods and results than using a single source of information does. Using both public and nonpublic data may also affect the magnitude of findings in meta-analyses. However, including nonpublic data sources in meta-analyses may require researchers to expend greater effort to extract relevant data. In addition, public and nonpublic data sources may provide inconsistent information for the same trial.
Future Research Needs
Future research could explore using public and nonpublic data sources in additional case studies to determine the generalizability of results. Future research could also examine the effects of using nonpublic data sources in meta-analyses of potential harms.
Final Research Report
View this project's final research report.
Journal Citations
Related Journal Citations
Peer-Review Summary
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
The awardee made the following revisions in response to peer review:
- The awardee explained to reviewers that the project was a methods study and the clinical issues described in the methods section were not the focus and should not appear in the background section. The awardee reasoned that the background section should focus on the methodological gaps the study aimed to fill.
- The awardee shortened the description of the methods used to conduct meta-analyses and generally reduced the length of the methods section by replacing text with tables. The awardee increased the section’s readability by using shorter paragraphs.
- In the discussion section, the awardee strengthened the argument for the study’s generalizability.
- The awardee added a study limitation related to a lack of assessment of potential harms of the interventions. The awardee also noted the need for future research on the reporting of harms in different data sources.