Skip to main content
Patient-Centered Outcomes Research Institute
Patient-Centered Outcomes Research Institute
  • Blog
  • Newsroom
  • Find It Fast
  • Help Center
  • Subscribe
  • Careers
  • Contact Us

PCORI

Patient-Centered Outcomes Research Institute

Search form

  • About Us
    Close mega-menu

    About Us

    • Our Programs
    • Governance
    • Financials and Reports
    • Procurement Opportunities
    • Our Staff
    • Our Vision & Mission
    • Contact Us

    Fact Sheets: Learn More About PCORI

    Download fact sheets about out work, the research we fund, and our programs and initiatives.

    Find It Fast

    Browse through an alphabetical list of frequently accessed and searched terms for information and resources.

    Subscribe to PCORI Email Alerts

    Sign up for weekly emails to stay current on the latest results of our funded projects, and more.

  • Research & Results
    Close mega-menu

    Research & Results

    • Explore Our Portfolio
    • Research Fundamentals
    • Research Results Highlights
    • Putting Evidence to Work
    • Peer Review
    • Evidence Synthesis
    • About Our Research

    Evidence Updates from PCORI-Funded Studies

    These updates capture highlights of findings from systematic reviews and our funded research studies.

    Journal Articles About Our Funded Research

    Browse through a collection of journal publications that provides insights into PCORI-funded work.

    Explore Our Portfolio of Funded Projects

    Find out about projects based on the health conditions they focus on, the state they are in, and if they have results.

  • Topics
    Close mega-menu

    Topics

    • Addressing Disparities
    • Arthritis
    • Asthma
    • Cancer
    • Cardiovascular Disease
    • Children's Health
    • Community Health Workers
    • COVID-19
    • Dementia and Cognitive Impairment
    • Diabetes
    • Intellectual and/or Developmental Disabilities
    • Kidney Disease
    • Maternal Morbidity and Mortality
    • Medicaid
    • Men's Health
    • Mental and Behavioral Health
    • Minority Mental Health
    • Multiple Chronic Conditions
    • Multiple Sclerosis
    • Obesity
    • Older Adults' Health
    • Pain Care and Opioids
    • Rare Diseases
    • Rural Health
    • Shared Decision Making
    • Telehealth
    • Transitional Care
    • Veterans Health
    • Women's Health

    Featured Topic: Women's Health

    Learn more about the projects we support on conditions that specifically or more often affect women.

  • Engagement
    Close mega-menu

    Engagement

    • The Value of Engagement
    • Engagement in Health Research Literature Explorer
    • Influencing the Culture of Research
    • Engagement Awards
    • Engagement Resources
    • Engage with Us

    Engagement Tools and Resources for Research

    This searchable peer-to-peer repository includes resources that can inform future work in patient-centered outcomes research.

    Engagement Awards

    Learn about our Engagement Awards program and view the announcements of all our open funding opportunities.

    Research Fundamentals: A New On-Demand Training

    It enables those new to health research or patient-centered research to learn more about the research process.

  • Funding Opportunities
    Close mega-menu

    Funding Opportunities

    • What & Who We Fund
    • What You Need to Know to Apply
    • Applicant Training
    • Merit Review
    • Awardee Resources
    • Help Center

    PCORI Funding Opportunities

    View and learn about the newly opened funding announcements and the upcoming PFAs in 2021.

    Tips for Submitting a Responsive LOI

    Find out what PCORI looks for in a letter of intent (LOI) along with other helpful tips.

    PCORI Awardee Resources

    These resources can help awardees in complying with the terms and conditions of their contract.

  • Meetings & Events
    Close mega-menu

    Meetings & Events

    • Upcoming
    • Past Events

    PCORI 2021 and Beyond

    During this webinar, PCORI leaders shared ways to get involved in PCOR, improvements to our funding opportunities, and more.

    2020 PCORI Annual Meeting

    Watch recordings of sessions and view titles and descriptions of posters presented at the virtual meeting.

    PCORI Board of Governors Meeting: April 13, 1:00pm ET

    Tune in as the Board discusses possible topics in the development of upcoming PCORI Funding Announcements, among other business. Register now.

You are here

  • Research & Results
  • Explore Our Portfolio
  • The Benefits and Challenges of Using ...

This project has results

The Benefits and Challenges of Using Multiple Sources of Information about Clinical Trials

Sign Up for Updates to This Study  

Results Summary and Professional Abstract

Results Summary
Download Summary Español (pdf) Audio Recording (mp3)

Results Summary

What was the research about?

Meta-analyses combine the results of many studies to find out how well a treatment or other healthcare intervention works. Most meta-analyses use public sources of data, such as published journal articles, as the main sources of information for study results. But journal articles are not the only sources of study results. Some results appear in other places, such as clinical study reports. Clinical study reports are documents that describe what researchers did and found in much more detail than journal articles. However, these reports may not be available to the public. As a result, meta-analyses may not include all available information about a treatment.

The research team wanted to learn whether adding or replacing public and nonpublic data sources changed the results of meta-analyses. To find out, the research team added and replaced data as they conducted two meta-analyses. The first looked at adult use of a nerve-pain medicine. The second meta-analysis looked at adult use of a medicine to treat bipolar depression.

What were the results?

The research team found that adding nonpublic data sources to public ones provided new details about study methods and results. Using all available data sources changed the size of the effect in the nerve-pain meta-analysis but doing so didn’t change the results for the bipolar-depression meta-analysis.

Nonpublic data sources, such as clinical study reports, often had information that medical journal articles lacked. Based on these results, the research team developed guidance for researchers who want to use different kinds of data sources in meta-analyses.

What did the research team do?

The research team asked patients and doctors to make a list of the health outcomes that matter to patients who take the medicines for nerve pain or bipolar depression. Then, the research team searched for studies about the two medicines. The research team looked at whether changing the sources of data on the outcomes that matter to patients changed the results of the meta-analyses.

The team looked at public and nonpublic data sources for the two meta-analyses. The research team found 80 sources for 21 studies about the nerve-pain medicine. The team found 51 sources for 7 studies about the bipolar-depression medicine.

What were the limits of the study?

The study looked at the effect of adding and replacing data sources in only two meta-analyses. Future research could look at whether adding and replacing data sources changes the results for meta-analyses focusing on other treatments or health problems. Also, future research could look at the effect of using nonpublic data sources on meta-analysis findings about treatment harms.

How can people use the results?

Researchers may want to use the results of this study when they look for data sources for meta-analyses.

Professional Abstract

Professional Abstract

Objective

To explore the effect on meta-analyses of using multiple data sources for each clinical trial and to develop guidance on conducting meta-analyses using multiple data sources

Study Design

Design Element Description
Design Cross-sectional analysis of multiple data sources for clinical trials in two case studies
Data Sources and Data Sets Public and nonpublic data sources for clinical trials used in meta-analyses
Analytic Approach
  • Assess completeness and consistency of information in different data sources for the same clinical trial by comparing meta-analysis outcomes and reviewing methodological quality using the Cochrane Risk of Bias Tool
  • Random-effects model meta-analyses
Outcomes

Completeness and consistency of data sources, and meta-analytic findings using information obtained from public and nonpublic data sources of clinical trial results

The research team examined the effect of adding and replacing public and nonpublic data sources for each clinical trial included in meta-analyses for two case studies: (1) gabapentin for neuropathic pain and (2) quetiapine for bipolar depression. The research team conducted searches to identify all sources of information about randomized clinical trials for these two interventions and determined whether these trials included patient-centered outcomes. The research team included patient co-investigators and clinical experts.

The research team searched for public sources of clinical trial results, such as study registries, reviews from the US Food and Drug Administration (FDA), conference abstracts, and journal articles. The team also searched for nonpublic sources of study results, such as clinical study reports and individual patient data. The team assessed the completeness and consistency of the methodological information in each data source, including information about interventions, comparators, outcomes, and results. To examine the effect of using multiple data sources for each clinical trial, the research team conducted a series of meta-analyses in which the team added or replaced data from each of the sources for one domain, measure, or time point.

The team included outcomes in the meta-analyses if the outcomes included a point estimate and a measure of precision, such as a standard error or confidence interval.

Results

Number of sources found. The research team identified 80 data sources for 21 trials of gabapentin (from 1997 to 2013) and 51 data sources for 7 trials of quetiapine (from 2003 to 2014). The team found more than one data source for 71 percent of the gabapentin trials and 100 percent of the quetiapine trials. The most commonly available data sources were journal articles and conference abstracts. The team did not include 9 of the 21 gabapentin trials and 4 of the 7 quetiapine trials in their meta-analyses because no data sources for those trials had sufficient statistical information.

Consistency of information. Data sources varied considerably in the amount of information they included about methods or results. In some cases, different data sources had different information about the same trial. Clinical study reports were consistently the most complete source of information about methods and patient-centered outcomes. Among public data sources, journal articles typically contained the most information about methods and patient-centered outcomes, but these articles often lacked sufficient statistical information to include results in meta-analyses.

Impact on meta-analysis findings. For gabapentin, adding nonpublic data reduced the magnitude of the effect size from -0.46 for a meta-analysis drawing only on peer-reviewed journal articles to -0.31 for a meta-analysis using all available sources. However, including nonpublic data did not affect the statistical significance of the intervention effect. For quetiapine, there were no meaningful differences when adding and replacing data from public and nonpublic sources.

Methods guidance. The research team developed guidance for researchers who conduct meta-analyses. The guidance is intended to help researchers deal with two issues. First, accessing, obtaining, and analyzing public and nonpublic data sources require significant resources and skills. Second, information varies considerably across sources.

Limitations

The study looked at using public and nonpublic data sources for only two case studies. The study did not examine intervention harms; therefore, researchers don’t know whether differences between public and nonpublic data sources would affect a meta-analysis of harms.

Conclusions and Relevance

Using public and nonpublic data sources in meta-analyses may give researchers more information about trial methods and results than using a single source of information does. Using both public and nonpublic data may also affect the magnitude of findings in meta-analyses. However, including nonpublic data sources in meta-analyses may require researchers to expend greater effort to extract relevant data. In addition, public and nonpublic data sources may provide inconsistent information for the same trial.

Future Research Needs

Future research could explore using public and nonpublic data sources in additional case studies to determine the generalizability of results. Future research could also examine the effects of using nonpublic data sources in meta-analyses of potential harms.

Final Research Report

View this project's final research report.

Journal Articles

Related Articles

Trials

Opportunities for selective reporting of harms in randomized clinical trials: Selection criteria for non-systematic adverse events

Journal of Clinical Epidemiology

Harms are assessed inconsistently and reported inadequately part 1: systematic adverse events

Journal of Clinical Epidemiology

Harms are assessed inconsistently and reported inadequately Part 2: nonsystematic adverse events

Trials

Caveat emptor: the combined effects of multiplicity and selective reporting

Pilot and Feasibility Studies

Methods to identify and prioritize patient-centered outcomes for use in comparative effectiveness research

Research Synthesis Methods

Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study)

Journal of Clinical Epidemiology

Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis

Systematic Reviews

Integrating multiple data sources (MUDS) for meta-analysis to improve patient-centered outcomes research: a protocol for a systematic review

The BMJ

Are manufacturers sharing data as promised?

More on this Project  

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

The awardee made the following revisions in response to peer review:

  • The awardee explained to reviewers that the project was a methods study and the clinical issues described in the methods section were not the focus and should not appear in the background section. The awardee reasoned that the background section should focus on the methodological gaps the study aimed to fill.
  •  The awardee shortened the description of the methods used to conduct meta-analyses and generally reduced the length of the methods section by replacing text with tables. The awardee increased the section’s readability by using shorter paragraphs.
  • In the discussion section, the awardee strengthened the argument for the study’s generalizability.
  • The awardee added a study limitation related to a lack of assessment of potential harms of the interventions. The awardee also noted the need for future research on the reporting of harms in different data sources.

Conflict of Interest Disclosures

View the COI disclosure form.

Project Details

Principal Investigator
Kay Dickersin, PhD, MA
Project Status
Completed; PCORI Public and Professional Abstracts, and Final Research Report Posted
Project Title
Integrating Multiple Data Sources for Meta-analysis to Improve Patient-Centered Outcomes Research
Board Approval Date
September 2013
Project End Date
October 2016
Organization
Johns Hopkins University
Year Awarded
2013
State
Maryland
Year Completed
2017
Project Type
Research Project
Health Conditions  
Mental/Behavioral Health
Bipolar Disorder
Neurological Disorders
Chronic Pain
Nutritional and Metabolic Disorders
Diabetes
Intervention Strategies
Drug Interventions
Funding Announcement
Improving Methods for Conducting Patient-Centered Outcomes Research
Project Budget
$761,279
DOI - Digital Object Identifier
10.25302/3.2018.ME.13035785
Study Registration Information
HSRP20143578
Page Last Updated: 
September 16, 2019

About Us

  • Our Programs
  • Governance
  • Financials and Reports
  • Procurement Opportunities
  • Our Staff
  • Our Vision & Mission
  • Contact Us

Research & Results

  • Explore Our Portfolio
  • Research Fundamentals
  • Research Results Highlights
  • Putting Evidence to Work
  • Peer Review
  • Evidence Synthesis
  • About Our Research

Engagement

  • The Value of Engagement
  • Engagement in Health Research Literature Explorer
  • Influencing the Culture of Research
  • Engagement Awards
  • Engagement Resources
  • Engage with Us

Funding Opportunities

  • What & Who We Fund
  • What You Need to Know to Apply
  • Applicant Training
  • Merit Review
  • Awardee Resources
  • Help Center

Meetings & Events

April 12
Engagement Awards 2021 Special Cycle -- Applicant Office Hours (Three)
April 13
Board of Governors Meeting: April 13, 2021
April 19
Increasing Vaccine Confidence among Long-Term Care Workers: Expedited COVID-19 PFA -- Applicant Town Hall

PCORI

Footer contact address

Patient-Centered Outcomes
Research Institute

1828 L Street, NW, Suite 900
Washington, DC 20036
Phone: (202) 827-7700 | Fax: (202) 355-9558
info@pcori.org

Subscribe to Newsletter

Twitter Facebook LinkedIn Vimeo

© 2011-2021 Patient-Centered Outcomes Research Institute. All Rights Reserved.

Privacy Policy | Terms of Use | Trademark Usage Guidelines | Credits | Help Center