Final Research Report

View this project's final research report.

Peer-Review Summary

Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.

The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments. 

Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:

  • The reviewers brought attention to several past reports that they suggested were relevant to this work and should be cited, particularly related to the validity of using observational studies to make causal inferences. The researchers noted that the idea of using observational data for experimental studies goes as far back as the 1940s at least and has been especially widely discussed in fields where randomizing study participants was not feasible for practical or ethical reasons. They added language to the report acknowledging the potential use of observational studies, particularly when there were ethical or methodological barriers to randomized controlled trials. However, the researchers said that they would not want to frame their views as recommendations because providing recommendations was not within the scope of this project.
  • The reviewers questioned how the researchers estimated standard errors in the models they developed given the importance of estimating the standard error correctly when aiming to compare weighted with nonweighted study results. The researchers revised the report to describe the nonparametric bootstrap techniques they used to estimate standard errors. However, in their response to reviewers the researchers also noted that it would be useful to compare different methods to estimate standard errors to test the variance and validity of the estimates. The reviewers chose not to report all of the alternate approaches to standard errors because the number would be fairly large.
  • The reviewers suggested the report address additional issues, for example, the implications of varying sample sizes, including using small samples in observational studies. The researchers added some discussion to the report in response to the various suggestions but on the issue of sample size, the researchers said that they believe that inference from observational data generally requires large amounts of data, especially to assess heterogeneity of treatment effects.

Conflict of Interest Disclosures

Project Information

Issa J. Dahabreh, MD, MS
Brown University
Evaluating Observational Data Analyses: Confounding Control and Treatment Effect Heterogeneity

Key Dates

December 2013
September 2019

Study Registration Information


Has Results
Award Type
State State The state where the project originates, or where the primary institution or organization is located. View Glossary
Last updated: March 4, 2022