Results Summary and Professional Abstract
Peer review of PCORI-funded research helps make sure the report presents complete, balanced, and useful information about the research. It also assesses how the project addressed PCORI’s Methodology Standards. During peer review, experts read a draft report of the research and provide comments about the report. These experts may include a scientist focused on the research topic, a specialist in research methods, a patient or caregiver, and a healthcare professional. These reviewers cannot have conflicts of interest with the study.
The peer reviewers point out where the draft report may need revision. For example, they may suggest ways to improve descriptions of the conduct of the study or to clarify the connection between results and conclusions. Sometimes, awardees revise their draft reports twice or more to address all of the reviewers’ comments.
Peer reviewers commented and the researchers made changes or provided responses. Those comments and responses included the following:
- The reviewers commented that the report seemed to present a predictive model for hemoglobin A1c (HbA1c) improvements based on analyses from electronic health records. The reviewers expressed concern because electronic health record data would be prone to potential bias because of unmeasured confounders and missing data for predictors and outcomes. The researchers revised their description of the electronic health records analysis to clarify that their probability models were meant to be explanatory rather than predictive. In other words, they were answering the question, “What happened?” rather than, “What will happen?”
- The reviewers asked whether the researchers tested their probability model for external validity, or the application of the model in a new population to see how well it predicts the same outcome. The researchers initially aimed to test external validity by applying the model they built using data from the randomized controlled trial to community data from patients’ electronic health records. The reviewers pointed out that the two data sets were not equivalent: the trial participants were more homogenous because of strict trial inclusion criteria compared to the community participants. In addition, the follow-up periods for the randomized controlled trial were more consistent, with shorter intervals, than the follow-up periods for the data collected from community electronic health records. The researchers therefore built a separate probability model for reaching the desired HbA1c levels for community patients using data from the electronic health records cohort of patients. The researchers tested neither probability model with a new population, and they identified the lack of testing as a limitation of the study.
- The reviewers noted that the researchers did not follow current standards for missing data when they used the most recent preceding available observation of HbA1c, or the Last Observation Carried Forward, to replace missing HbA1c data at the study’s endpoint. The researchers contended that this technique was valid in this situation because HbA1c is a summary measure of glycemic control over a three-month period and that this technique was conservative because HbA1c was likely to have improved if the study endpoint measure was collected. The reviewers disagreed with this contention and requested the researchers to acknowledge that their methods to address missing data were potentially biased, which was a limitation of the study.
Conflict of Interest Disclosures
View the COI disclosure form.