The PCORI Board of Governors approved this proposed category of Methodology Standards for a public comment period in July 2018. The public comment period took place between July 24 and September 21, 2018, and the final standards were adopted by the Board in February 2019.

View the Final Standards

View the Draft Proposed Standards for the Public Comment Period

 

Submitted Comments

Submitted

7/24/18 13:26

General feedback on the Standards for the Individual Participant-Level Data Meta-Analysis (IPD-MA)

The largest clear source of bias in IPDMA are when there are studies that exist, but for which the data isn't provided. I think all IPDMA should have a summary table showing, within their criteria, all studies that they included compared to the published values of the studies that were eligible, but not available for IPDMA.

Name and Organization

Jeremy Sussman, University of Michigan

Community

Stakeholder

Patient Community

Patient

Stakeholder Community

Health Researcher

---

Submitted

8/23/18 12:54

IPD-4: Specify the design and planned analyses of the IPD-MA in a protocol, document any changes, and report significant amendments and modifications.

Consider whether specifying sensitivity analyses to test robustness of assumptions about models, or across dimensions such as study-level risk of bias, etc needed? How will missing data be handled?

Name and Organization

Anonymous

Community

Stakeholder

Patient Community

Patient

Stakeholder Community

Health Researcher

---

Submitted

8/26/18 21:18

IPD-4: Specify the design and planned analyses of the IPD-MA in a protocol, document any changes, and report significant amendments and modifications.

There is growing concern about the transparency and reproducibility of meta-analyses using individual patient health information that must be kept protected. Our community of researchers will not have the opportunity to attempt reproducing results, as is possible with study-level meta-analyses. Can the Methods Committee consider requesting that all PCORI-funded meta-analyses of individual patient data make available the analysis code (but not the data) as a supplement with the primary manuscript publication? This would allow other researchers to walk through each step of methods as they were conducted. There are often small and unexpected decisions that a researcher must make when carrying out an analysis plan and this can sometimes have a large impact on the results. Open source code could provide a critical level of detail than is often presented in a statistical analysis plan to make the methods completely transparent. There are many free websites, such as GitHub, where code to support the manuscript can be easily shared. If there is push-back from researchers, consider developing language so that PCORI-funded researchers would have to make a strong case as to whey their project should not have to make their code publicly available.

Name and Organization

Blythe Adamson, Flatiron Health and University of Washington

Community

Stakeholder

Patient Community

Patient

Stakeholder Community

Health Researcher

---

Submitted

9/21/18 10:17

IPD-1: Specify the research question(s) that will be addressed through the IPD-MA and describe the specific information the IPD-MA will provide that other approaches would not.

We recommend adding “or algorithms” after approaches, so the sentence would read: “….the IPD-MA will address the limitations of other potential approaches or algorithms….”

IPD-2: Describe the proposed governance structure for the IPD-MA in the protocol and study reports.

AcademyHealth recommends adding that the protocol and study reports should also describe the level and frequency of individual trial investigator involvement. For example, does the work necessitate in-person meeting and/or virtual meetings? If so, how often? Additionally, AcademyHealth recommends that in addition to data use agreements, information be reported about IRB and protection of human subjects. Finally, AcademyHealth recommends adding “clinical” in the first bullet, so the sentence reads: “Roles, relationships, and decision-making authority of the research team leading the IPD-MA, the clinical trial investigators…”

IPD-3: Use systematic, reproducible methods to identify studies for inclusion in the IPD-MA.

AcademyHealth feels this standards is generally appropriate, but notes that the standard is not unique to IPD-MA. Rather, the standard is appropriate for most any systematic review and meta-analysis, regardless of whether it is an IPD-MA or a study-level aggregate MA. Is there anything more specific that could be added to this standard for IPD-MA meta-analysis? AcademyHealth also suggests that PCORI consider including in the standard the creation of a PRISMA flow diagram to account for each step. Please see this site for further information. http://www.prisma-statement.org/ More specifically, AcademyHealth recommends adding “presented” and “excluded” in the text, so the sentences read: “Describe the approach to ensuring that all relevant published, presented, and unpublished studies are considered for inclusion.” “Record the number of studies and participants identified and screened, assessed for eligibility, and included or excluded in the IPD-MA.”

IPD-4: Specify the design and planned analyses of the IPD-MA in a protocol, document any changes, and report significant amendments and modifications.

AcademyHealth recommends being less specific about the place of public registration (i.e., PROSPERO), and allow for other mechanisms of publishing the protocol (e.g., government sponsored websites, or other publicly available sources). And, similar to our previous comments, many of these items are not particularly unique to an IPD-MA, they also apply to aggregate MAs. Is there anything more specific about IPD-MAs that PCORI might consider highlighting or emphasizing? More specifically, AcademyHealth recommends the following additions or edits to the language shown in italics below: • In the first bullet, last sentence, “Data should be requested on all randomized participants eligible for the IPD-MA, even if they were not included in the final analyses of an original clinical trial.” • In the second bullet, “Document the processes used to check accuracy of data and to correct and harmonize data, including conferring with the original trial investigators.” • In the third bullet, “Describe the approach to assess the quality of the data, including assessing the risks or potential risks of biases in individual studies.” • In the fourth bullet, “Describe the statistical analysis plan, which should include pre-specification and justification of the hypotheses within different types of participant subgroups, for example including whether these will be analyzed at the participant or study level…” • In the closing sentence, “All amendments and modifications to the protocol should be documented, and any significant changes (e.g., outcome definitions, analytic approaches, additional analyses) should be reported and amended in the publicly available protocol or statistical analysis plan.”

General feedback on the Standards for the Individual Participant-Level Data Meta-Analysis (IPD-MA)

AcademyHealth appreciates PCORI development of draft standards for individual participant-level meta-analysis (IPD-MA) and the opportunity to comment. To strengthen and clarify the standard, AcademyHealth suggest that PCORI begins with a definition of individual participant-level data and specific examples of types of data they believe this includes. For example, is it IPD-MA clinical trial data only or could it include data from electronic health records, patient-reported outcome data, qualitative interview data, etc.? This definition and clear examples will ensure that all potential users of the standard are on the same page. If primarily focused on clinical trial data, PCORI should consider referencing and encouraging use of the PICOTS framework developed by AHRQ’s EPC program to specify the research question. Please see: https://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDRH/CDRHInnovation/UCM587380.pdf

Name and Organization

AcademyHealth

Community

Stakeholder

Stakeholder Community

Stakeholder - Other

---