Draft Methodology Report Public Comments (2012)

Public Comment Submissions

Author

Informed Medical Decisions Foundation

Date

Sunday, September 9, 2012

Zip Code

2108

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

They are consistent, but we have quite a few thoughts about how they could be strengthened

2. What methodological gaps should be addressed in future reports?

See attached memo. However, briefly, we would like to see: 1. Encouragement of focus groups and other strategies for getting patient input into what to measure2. Value of routine measurement of patients' goals, concerns and preferences3. Limits of studies using medical records unless they include input from patients4. Better framing of issues related to internal and external validity, with particular emphasis on importance of external validity concerns5. Addition of importance of standards for survey question design and good quality evaluation of questions in addition to psychometric evaluation6. Emphasis on the importance of studies of long-term natural history of conditions, especially those that are defined by symptoms, like knee pain

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

2. What methodological gaps should be addressed in future reports?

Community engagement

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

In a general sense. The central challenge is that methods will vary enormously depending on study question. While establishing guidelines for the field is likely to prove beneficial over the long run, it could make for a more onerous review by study section members - at least until they have committed criteria to memory.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

5. Please enter any additional comments or concerns here.

My first question is whether applicants will be expected to explicitly reference the methods report including citing each section number in their application? Given that the report may not be finalized until November, will applicant during the November cycle be expected to adhere to these criteria?

Author

Leif Solberg

Date

Sunday, August 12, 2012

Zip Code

55419

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

1 Comments

As stated in the PPACA, "The purpose of the Institute is to assist patients, cliniicians, purchasers, and policy makers in making informed health decision by advancing the quality and relevance of evidence,” yet one searches in vain for evidence that the standards are meant to accomodate anything but a patient perspective. I realize this report was meant to focus on patients, but this seems to be the one that will identify standards for research, so how will they be expanded to reflect a broader group of stakeholders?

2. What methodological gaps should be addressed in future reports?

Valuing and obtaining the views of clinicians and care systems about which knowledge gaps are important to them and building research in a way that incorporates their perspective

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

3 Comments

Not if it continues to present these guidelines as standards (see attachment)

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

5. Please enter any additional comments or concerns here.

see attachment

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments-on-PCORI-Draft-Methodology-Report.doc

Author

MTPPI

Date

Tuesday, July 31, 2012

Zip Code

20814

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

1) In the ‘causal inference’ section of the report, MSM models are not mentioned and should be in addition to propensity scores instrumental variable models. 2) Conducting a per protocol analysis (versus an intent to treat) is not mentioned although it is applicable to observational data.3) Interrupted time series is also not mentioned -- nor is any kind of pre/post analysis – although these are also considered to be ‘causal’ models.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

Patient

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

5. Please enter any additional comments or concerns here.

Personal Health Records (PHRs) concept should be part of your “Patient-centered Outcomes Research”. As your report says, “patient-reported outcomes is essential to patient-centeredness”.Furthermore, an “anonymous PHRs” or “confidential PHRs” will give a patient more privacy and help more people to adopt a PHRs program sooner. An anonymous PHRs system is able to collect more unbiased patient report and help government obtain certain information that will be hard to collect otherwise.As the inventor of “anonymous PHRs” concept, we can provide more research result that is based on our pioneer anonymous PHRs online service.

Author

Prof. F. Chiappelli

Date

Monday, July 23, 2012

Zip Code

90095

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes - but incomplete Three critical and timely issues of PCOR methodology that are understated or altogether under-discussed in the Report are outlined in (2) below

2. What methodological gaps should be addressed in future reports?

The reports requires a more substantial depth in the domains of:A. development, standardization & validation of instruments for the assessment of "quality" of the evidence ("how well the research was done) (beyond the "level" of the evidence, which simply tell "what research was done") - that is, and specifically content & criterion validity, inter-rater & internal consistency reliability, coefficient of agreement - such as, for example, Kung et al. From Systematic Reviews to Clinical Recommendations for Evidence-Based Health Care: Validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for Grading of Clinical Relevance. The Open Dentistry Journal 2010, 4:84-91. Phi et al. Expanding the Grading of Recommendations Assessment, Development, and Evaluation (Ex-GRADE) for Evidence-Based Clinical Recommendations: Validation Study. The Open Dentistry Journal 2012 6:31-40. Chiappelli et al. Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT. Bioinformation, 2012 8: 691-4.B. development, standardization & validation of instruments for the assessment of clinical relevance, such as, for example, Phi et al. Expanding the Grading of Recommendations Assessment, Development, and Evaluation (Ex-GRADE) for Evidence-Based Clinical Recommendations: Validation Study. The Open Dentistry Journal 2012 6:31-40. Dousti et al. Evidence-Based Clinical Significance in Health Care: Toward an Inferential Analysis of Clinical Relevance. Dental Hypotheses, 2011 2:165-77.C. Improved means of dissemination of the best available evidence to the stakeholders, and development, standardization & validation of instruments for the assessment of efficiency of dissemination (i.e., increased health literacy, etc.), such as for example, Barkhordarian et al. Dissemination of Evidence-Based Standards of Care. Bioinformation, 2011 7:315-9. Barkhordarian et al. Disseminating the Best Available Evidence: New Challenges in Public Reporting of Health Care. Bioinformation, 2012 8:293-5. Chiappelli et al. Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT. Bioinformation, 2012 8: 691-4.

https://pcori.org/sites/default/files/public-comments/methodology-report/MITA-Comments-to-PCORI-on-the-Draft-Methodology-Report-FINAL.pdf

Author

Judith Baggs

Date

Friday, September 14, 2012

Zip Code

97239

Stakeholder Category

Researcher

Author

Sary Beidas

Date

Friday, September 14, 2012

Zip Code

30813

Stakeholder Category

Clinician

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

There are 2 variables in which patient-centered research methodology should be expanded beyond a randomized controlled trial. The first variable is patient perspectives and the second is the escalating use of information technology.An example of methodologies that are not given sufficient weight in the report are: qualitative studies and mixed methods research. I recommend that the committee gives weight to these methodological gaps for acceptable PCORI submissions.

Author

J Chan

Date

Friday, September 14, 2012

Zip Code

94127

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes, the recommendations are consistent with PCORI's mission and vision.

2. What methodological gaps should be addressed in future reports?

As PCORI gains experience with application of methodological standards to PFA submissions in upcoming cycles, the process will facilitate identification of gaps and contribute to the evolution of methodological standards.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Yes, this report provides general information on PCORI policy regarding grant reviews and submissions.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Email and Webcast, assistance from affiliated organizations.Other Notes>At the current state of healthcare policy, infrastructure and technology development, the timeliness of our collective research efforts to demonstrate patient-centered outcomes effectiveness is perhaps of significance, the ability for our research efforts to provide evidence and potentially influence healthcare policy that will foster improved patient-centered outcomes is a window of opportunity. Thank you for inviting us to this comment process!

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Section 4.1.3>Pg 26> Use Patient-Reported Outcomes When Patients or People at Risk of a Condition Are the Best Source of Information. Notes> Section prerequisites > adequate health literacy and no language barrier

Author

European network for Health Technology Assessment

Date

Friday, September 14, 2012

Stakeholder Category

Researcher

5. Please enter any additional comments or concerns here.

This feedback is basically intended to send a signal that activities similar to those behind the report are taking place within the network of government nominated agencies and institutions in 26 member counties in the European Union supported by the European Commission (www.eunethta.eu). We currently have 9 guidelines for relative effectiveness assessment of pharmaceuticals in "public consultation" (public comment). With obvious similarities in important remits between PCORI and us the European network for Health Technology Assessment, EUnetHTA, wish to express an interest in developing contacts with PCORI with the intention of exchanging information and raising standards in patient centrered outcomes research at a global level.

Date

Thursday, September 13, 2012

Please provide your comments on Chapter 4 of the Draft Methodology Report.

September 13, 2012Dr. Joe SelbyExecutive DirectorPatient-Centered Outcomes Research Institute1828 L St., NW, Suite 900Washington, DC 20036Sherine Gabriel, MD, MScProfessor of Medicine and of Epidemiology, William J. and Charles H. Mayo ProfessorMayo ClinicChairPCORI Methodology CommitteeRe: Public Comment on Standard 4.1.5 in the PCORI Draft Methodology ReportDear Dr. Selby and Dr. Gabriel:We appreciate the openness demonstrated by the Patient Centered Outcomes Research Institute (PCORI) in providing this solicited opportunity to comment on the Draft Methodology Report (hereafter referred to as the Report). We recognize the strong efforts made by the PCORI Methodology Committee to develop proposed standards for patient-centered outcomes research (PCOR) under the break-neck timelines legislated by the Patient Protection and Affordable Care Act (ACA).We are commenting only on Standard 4.1.3: Use Patient-Reported Outcomes When Patients or People at Risk of a Condition Are the Best Source of Information.To assist the PCORI Methodogology Committee in formulating this standard, PCORI contracts were granted to two independent teams of researchers. The first team included present and past members of the International Society of Quality of Life Research (ISOQOL) board of directors led by the current ISOQOL president, Bryce Reeves at the University of North Carolina-Chapel Hill and Zeeshan Butt at Northwestern University in Evanston, Illinois. This research team, hereafter referred to as the ISOQOL group, provided background and a proposal for minimum methodologic standards in the design and selection of patient-reported outcomes measures (PROs) for use in PCOR through expert input, followed by a thorough literature review to prepare draft minimal methodologic standards in the design and selection of patient-reported outcomes measures (PROMs) for use in patient centered outcomes research. In addition, an online survey of the entire ISOQOL membership was conducted, and a standard was designated as recommended when over 50% of ISOQOL respondents endorsed it as “required as a minimum standard.” It is important to disclose that as a current ISOQOL board member, Kathleen Wyrwich and Dennis Revicki, former board member, volunteered to participate in all stages of this team’s research, including completion of the online survey and review of the results presented to PCORI.The second contracted team from Oxford Outcomes lead by Sarah Acaster (San Francisco, California) and Andrew Lloyd (Oxford, United Kingdom) also proposed and justified minimum methodologic PCOR. Their approach included discussion with internal resources, a brief targeted literature review and website search to identify concepts and themes as either potential minimum standards or issues for consideration. These concepts and themes were then reviewed by internal and external experts. With less than three months between the announcement of contract awarding and the presentation of the final reports, both selected teams completed their respective contracted projects within this narrow timeframe at a total cost of $153,613 ($83,908 and $69,705, respectively).The resulting minimal methodologic standards in the design and selection of PROs for use in patient-centered outcomes research were quite similar, with each team purposely seeking the appropriate balance of scientific detail without over-reaching as to create minimal standards so prescriptive that they would discourage the use of PROs in PCORI research. Sadly, many of these scientific details are not included in the Report. Therefore, we ask that PCORI reconsider the following issues and supported statements for inclusion in the next version of the Report.1. Patient Burden Although “overly burdensome patient-reported outcome measures” is used as a potential reason that a patient may withdraws from the registry that should be identified and addressed in Standard 8.2.8 (Data Registries), both final reports stressed this as an important minimum standards. Indeed both groups’ reports to PCORI also addressed that a PRO measure must not be overly burdensome for patients, and that the length of the PRO measure should be considered in the context of other PRO measures included in the assessment, the frequency of PRO data collection, and the characteristics of the study population. Although these specific issues may appear to be included in several other standards within the Report, their direct inclusion in Standard 4.1.3 will provide additional guidance to future PCORI research teams. To determine burden to patients, the targeted patient group should be consulted, perhaps through pilot testing before the main study is implemented.In addition, both reports addressed PRO literacy levels, with close consensus. The ISOQOL-organized group recommends “The literacy demand of the items in the PRO measure should usually be at a 6th grade education level or lower; however, it should be appropriately justified for the context of the proposed application” (ISOQOL), while the Oxford Outcomes group noted that “approximately a fifth of American adults read and comprehend below a fifth grade level, with the figure rising to approximately half the US population if level 2 literacy (reading and comprehending at 5th-7th grade levels)” in their discussion of issues for consideration. I hope that PCORI will reconsider the addition of the ISOQOL group’s recommendation given that it focuses future PCORI researchers to address and justify PRO literacy concerns. We request that PCORI consider the addition of the following statement in Standard 4.1.3:A PRO measure must not be overly burdensome for patients. The length of the PRO measure should be considered in the context of other PRO measures included in the assessment, the frequency of PRO data collection, and the characteristics of the study population. The literacy demand of the items in the PRO measure should usually be at a 6th grade education level or lower; however, it should be appropriately justified for the context of the proposed application.2. ReliabilityBoth the ISOQOL and Oxford groups submitted to PCORI minimal standards that address the need for internal consistency and test-retest reliability should be estimated and reported for each domain when appropriate (based on the nature of the PRO and the patient population), with Cronbach’s alpha coefficient ≥0.70 as a threshold guideline. The Oxford Outcomes group noted that detailed reporting is required so reliability can be appropriately evaluated, and assists meta analyses. Disappointingly, the Report states that PCORI proposals should describe “evidence of measurement properties including …reliability,” with no directives on how reliability should be assessed or any relevant thresholds. Again, for the purpose of providing additional guidance to future PCORI research teams on this important PRO issue, we request that PCORI consider the addition of the following statement in Standard 4.1.3:The reliability of a PRO measure should preferably be at or above 0.70 for group level comparisons, but may be lower if appropriately justified. Reliability for multi-item scales should include an assessment of internal consistency and test-retest reliability, and reliability for a single item measure should be assessed by test-retest reliability.3. Recall PeriodWhen patients are asked their input about a specific concern, identifying the appropriate timeframe for consideration (e.g., currently, past 24 hours, past 7 days, past 2 weeks) is essential for standardizing the measurement across all participants. This timeframe is often referred to as the recall period, and it must be appropriate for the purpose/intended use, the patient population, the disease/condition, the treatment/device, and the study design. Ideally, patient input regarding the most appropriate recall period is assessed throughout the PRO measure’s development and psychometric evaluation process, and is essential to patient understanding of the PRO and subsequent use in PCORI research.Although patient involvement in the PRO development process is included in Standard 4.1.3, the importance of the recall period is not addressed, although both the ISOQOL and Oxford Outcomes reports to PCORI stressed this important issue. We request that PCORI consider the addition of the following statement in Standard 4.1.3:PRO measures must provide justification for the recall period for the measurement application.4. TranslationBoth the ISOQOL and Oxford Outcomes reports to PCORI stressed the need for translatability and linguistic validation for PROs. Although this there is mention of the phrase “cultural and language translations” on page 130 of the Report, the importance of this PRO concern often disregarded without consideration of patient understanding of the translated PRO measure, as well as the problematic use of culture-specific terms (e.g., shoveling snow, bowling, gardening, etc.). Again, for the purpose of providing additional guidance to future PCORI research teams on this important PRO issue, we request that PCORI consider the addition of the following statement in Standard 4.1.3:A PRO measure translated to one or more languages should have documentation of the methods used to translate and evaluate the PRO measure in each language. Studies should at least include evidence from qualitative methods (e.g., cognitive testing) to evaluate the translations.With the investment by PCORI in these two strong awardees entrusted to provide minimum standards, the two specific approaches used by each group and the high level of consensus achieved by both groups in their reports to PCORI, we ask that these four concerns receive a close re-examination to ensure that appropriate minimal standards are used to solicit patients’ voices PCORI proposals and subsequently funded research. Much of the literature utilized in these reports and recommended by these two groups has been incorporated into the Report’s discussion on page 130, but there is far more to be learned by all in the PCORI research community from these efforts. If needed, we can provide PCORI with detailed examples of how neglecting any one of these concerns above can lead to uninterpretable or inappropriate study results, and, indeed, patient confusion during the PRO measurement process.Also, as a very small recommended change, the Wyrwich et al. (2012) reference on page 133 now available as an electronic publication. The citation is:Wyrwich KW, Norquist JM, Lenderking WR, Acaster S, and the Industry Advisory Committee of International Society for Quality of Life Research (ISOQOL). Methods for Interpreting Change over Time in Patient-reported Outcome Measures. Quality of Life Research. 2012 Apr 17. [Epub ahead of print.]Many thanks for this opportunity to provide my comments on this important standard. We look forward to reading the next iteration of this living document.Sincerely, Kathleen W. Wyrwich, PhD Dennis A. Revicki, PhDSenior Research Leader Senior Vice PresidentUnited BioSource Corporation United BioSource Corporation

Author

Shire Development LLC

Date

Thursday, September 13, 2012

Zip Code

19087

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Please see the attached document summarizing them.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Symposium, Congresses, email distribution, etc.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Shire-Collective-Comments-to-PCORI-Draft-Methods-report-Sept_14_2012_clean.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

ES1-2: “Over time, these reports, standards, translation tables, and public engagement forums are expected to produce better research methods, which in turn will provide information of benefit to all stakeholders—researchers planning an investigation, policy-makers weighing the value of healthcare interventions, and patients and their caregivers facing health decisions.”1.) Comment: The term stakeholder(s) is used broadly throughout the document and has 2 definitions provided in Appendix F: Glossary that are exactly the same. PCOR Stakeholders and Stakeholders are both defined as “Anyone affected by health decisions. Stakeholders may include patients, clinicians, caregivers, policymakers.” (p 182 &185) This is not an inclusive list of the stakeholders that could be included when mentioned through the document. One example is the term “researchers” that is used in the excerpt above but not included in the definitions of stakeholders. This needs to be thought out more clearly. One suggestion is to be more inclusive in the general definition of stakeholders, including researchers, payers, industry, etc and then have several sub-definitions that are used more explicitly, such as PCOR stakeholders or research stakeholders – academia, industry, government funded, etc. We should think about all stakeholders that could benefit from this methods report, including the producers of the research that this is aimed at guiding, for the patients’ benefit. 2.) Comment: In order to produce better research methods in comparative effectiveness research (CER), other methods such as indirect treatment comparisons (ITC), meta-analysis, and mixed treatment comparisons (MTC) should be considered the starting point for all patient-centered research. This report begins with and focuses on improving and producing better research methods for CER, yet it skips over the initial stages where CER begins—the existing literature to inform the adaptive designs, registries and diagnostic testing designs which are called for later in chapter 8 of the report. For example, recent advances in methods such as match-adjusted indirect comparisons (MAIC) have been created as a sub-type of ITC which can better inform decision making. We enclose a summary from a recent publication on the subject which was presented at the AHRQ symposium on novel CER methods in June 2011 and published inthe journal Pharmacoepidemiology and Drug Safety as a special edition focused on CER methods in 2012. There are 5 other publications in the recent literature available1-5[Cite 4 Signorovitch et al publications] including ISPOR Good Research Practices on Interpreting Indirect Treatment Comparisons and Network Meta-Analysis for Health-Care Decision Making that state that “[O]btaining patient-level data for all RCTs in the network may be considered unrealistic. As an alternative, one could use patient-level data when available and aggregate-level data for studies in the network for which such data are not available, thereby improving parameter estimation over aggregate data–only models.”6 An example excerpt describing MAIC and some of the benefits of it are described in appendix A. Appendix A: Excerpt Describing MAIC method from Signorovitch et al from Pharmacoepidemiol Drug Saf. 2012:“Comparative effectiveness research requires the comparison of alternative therapies in a timely fashion, often in the absence of direct head-to-head randomized trials. For example, healthcare payers may wish to compare drug A versus drug B when randomized trials have only compared each with drug C (e.g., placebo). A traditional method for indirect comparison in this setting is the adjusted indirect comparison.2–6 This method uses the common comparator drug C to infer the relative effects of drugs A and B. This approach can incorporate multiple trials per treatment2 and has been extended to simultaneously compare multiple treatments linked through a network of direct comparisons7 and to adjust for aggregate baseline characteristics using meta-regression.3–5 Comprehensive and detailed reviews of indirect treatment comparisons have been published,5,6 and guidelines have been developed for researchers and decision makers.8–11 Traditional indirect comparison methods have important and well-recognized limitations. First, indirect comparisons can be biased by cross-trial differences in patient populations.2,5,6,12–14 Second, different measures of the treatment effect [e.g., risk difference (RD) versus odds ratio (OR)] can lead to different T1 conclusions, as shown in the example in Table 1. When only a few trials are available for each treatment of interest, these limitations become more pronounced because meta-regression cannot be used to adjust for cross-trial differences in multiple baseline characteristics, and the choice of effect measure (e.g., OR and RD) cannot be justified by assessments of heterogeneity across trials. If individual patient data (IPD) were available from all trials of interest, potential biases stemming from cross-trial differences could be mitigated by regression adjustment.15–19 However, IPD from all relevant trials are rarely available to researchers. It is more common to have access to IPD for trials involving one drug and only published aggregate data for trials of comparators. Researchers who conduct their own trials, or who collaborate with or are affiliated with trial sponsors, can often find themselves in this situation. Despite the widespread interest in using all available data in indirect comparisons,20 few published methods combine IPD with aggregate data for indirect comparisons, leaving IPD as a largely untapped resource for CER. Matching-adjusted indirect comparison is a recently developed method that can use IPD from trials for only one comparator to adjust for cross-trial differences in baseline characteristics.22 This method works by re-weighting patients with IPD such that their average baseline characteristics match those reported for trials without available IPD (i.e., with summary statistics only). Treatment outcomes can then be compared between balanced populations.”2. Signorovitch et al Excerpt References :Bucher HC, Guyatt GH, Griffith LE, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997; 50:683–91.3. Eckert L, Falissard B. Using meta-regression in performing indirect-comparisons: comparing escitalopram with venlafaxine XR. Curr Med Res Opin 2006; 22(11):2313-21.4. Nixon RM, Bansback N, Brennan A. Using mixed treatment comparisons and meta-regression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis. Stat Med 2007; 26:1237–54.5. Glenny AM, Altman DG, Song F, et al. Indirect comparisons of competing interventions. Health Technol Assess 2005; 9(26):1-134.6. Sutton A, Ades AE, Cooper N, et al. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics 2008; 26:753–67.7. Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002; 21:2313–24.8. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting Indirect-Treatment-Comparison and Network-Meta-Analysis Studies: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices-Part 2. Value Health. 2011 Jun;14(4):429-37.9. Jansen JP, Fleurence R, Devine B, et al.. Interpreting Indirect Treatment Comparisons and Network Meta-Analysis for Health-Care Decision Making: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1. Value Health. 2011 Jun;14(4):417-28.10. Wells GA, Sultan SA, Chen L, et al. Indirect Evidence: Indirect Treatment Comparisons in Meta-Analysis. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.11. NICE Guide to the Methods of Technology Appraisal, http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf12. Phillips A 2003 Trial and Error: cross-trial comparisons of antiretroviral regimens. AIDS 2003 17:619-623.13. Cranney A, Guyatt G, Griffith L, et al. Meta-analyses of therapies for postmenopausal osteoporosis. IX: Summary of meta-analyses of therapies for postmenopausal osteoporosis. Endocr Rev 2002; 23(4):570-8.14. Deeks JJ, Higgins, JPT, et al. Analysing and presenting results. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6; Section 8. In: The Cochrane Library, Issue 4, 2006. 15. Pignon JP, Arriagada R, Ihde DC, et al. A meta-analysis of thoracic radiotherapy for small-cell lung cancer. N Engl J Med 1992; 327:1618–24.16. Homocysteine Lowering Trialists’ Collaboration. Lowering blood homocysteine with folic acid based supplements: meta-analysis of randomised trials. BMJ 1998; 316:894–8.17. Non-small Cell Lung Cancer Collaborative Group. Chemotherapy in non-small cell lung cancer; a meta-analysis using updated data on individual patients from 52 randomised clinical trials. BMJ 1995; 311:899–909.18. Moore RA, McQuay HJ. Single-patient data metaanalysis of 3453 postoperative patients: oral tramadol versus placebo, codeine and combination analgesics. Pain 1997; 69:287–94.19. Turner RM, Omar RZ, Yang M, et al. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat Med 2000; 19:3417–32.20. Neumann PJ, Drummond MF, Jönsson B, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? International Journal of Technology Assessment in Health Care, 26:1 (2010), 71–78. 21. Ross SD. Trends in meta-analysis. Drug Inf J 2009; 43:171–76.22. Signorovitch JE, Wu EQ, Yu AP, et al. Comparative effectiveness without head-to-head trials: a method for matching-adjusted indirect comparisons applied to psoriasis treatment with adalimumab or etanercept. Pharmacoeconomics 2010;28(10):935-45

Please provide your comments on Chapter 2 of the Draft Methodology Report.

1.) Comment: The term stakeholder(s) is used broadly throughout the document and has 2 definitions provided in Appendix F: Glossary that are exactly the same. PCOR Stakeholders and Stakeholders are both defined as “Anyone affected by health decisions. Stakeholders may include patients, clinicians, caregivers, policymakers.” (p 182 &185) This is not an inclusive list of the stakeholders that could be included when mentioned through the document. One example is the term “researchers” that is used in the excerpt above but not included in the definitions of stakeholders. This needs to be thought out more clearly. One suggestion is to be more inclusive in the general definition of stakeholders, including researchers, payers, industry, etc. and then have several sub-definitions that are used more explicitly, such as PCOR stakeholders or research stakeholders – academia, industry, government funded, etc. We should think about all stakeholders that could benefit from this methods report, including the producers of the research that this is aimed at guiding, for the patients’ benefit. 3.) Comment: See Comment 1 on stakeholders. Here the term research stakeholders may replace the term for ”researchers, publishers, and industry”. Otherwise the examples are redundant and not all inclusive. Industry does much research, although there are also many other organizations/entities that do research and are not mentioned. Throughout the document, other stakeholders options could be “decision maker stakeholders, payer stakeholders, etc.”.Comments 1 & 3 can also be applied to (these are examples, not an exhaustive list)Chapt 2, pg 10: “It is expected that the translation table will eventually include versions appropriate for use by different stakeholder groups”Chapt 2 , pg 11: “This includes engaging all stakeholders who might use the standards, creating reporting and surveillance opportunities, and developing enforcement functions over time.”

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Comment 4 also applies to:Chapter 3 pg.15. “One aim is to counter over generalization from studies in narrow populations or in highly controlled research settings to the broader range of people and settings in “real life.” (17, 18) But another aim, undoubtedly, is to learn more about how well the different treatment choices can work, and for whom they work best and are safest.” Pg. 7 “The next three years of Methodology Committee work will be a continual process of reconsidering, refining, and widening the scope of the standards to include the full spectrum of PCOR questions and approaches. Similarly, the translation table within this report will be expanded over time to include more examples, methodological issues, and approaches.”5.) Comment: It is a good and fundamental place to start with the existing methods which can be improved with novel approaches as described above and that the report could be expanded now to incorporate PCM and ITC, MTC and meta-analytic methods into the PCORI methodology report.“In 2008, the Institute of Medicine stated that methodological standards for the conduct of one type of research— systematic reviews—would help decisionmakers involved in PCOR “with respect to transparency, minimizing bias and conflict of interest, and clarity of reporting.”(15) In 2011, the Institute of Medicine published such standards.(16)”6.) Comment: These guidelines don’t include novel methods such as MAIC or other advances in ITC, MTC, and meta-analytic methods which have become more readily utilized and standardly applied to more and more therapeutic area reviews for pharmacologic or device interventions.Chapter 3, Table 3.1, pg. 18 “Create conceptual framework Specify hypothesized mechanisms of effect, influences on outcomes, and clinical context of affected care processes to guide data collection and analysis plans”7.) Comment: Patient perspective should be taken into account somewhere along the way, whether it is in the conceptual stages quoted above or in earlier stages of identifying the research questions. “Because some standards apply only to certain types of studies, a portfolio of templates applicable to various study designs should be developed.”8.) Comment: As before, a portfolio of novel methods should be considered such asPCM, MAIC, etc.Chapter 3, pg 20. “Standards for Formulating Research Questions” Box9.) Comment: A statistical analysis plan should be created and finalized (ie signed off) either in parallel or subsequent to the protocol. If it is not specified here, then it should be included in Section 7.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Chapter 4 (Patient centeredness): 10.) Comment: This chapter is very process oriented, and broadly highlights the key issues and the thinking framework needed to think about “patient centeredness” to advance this field. These points should be further elaborated upon and described in greater detail in this chapter. The following is a suggested framework of thinking that could be proposed: a. Define outcomes that are meaningful to patients that are measurable and concrete, especially in disease areas where outcomes have not been defined (this is mentioned but could be further elaborated upon). Who should be involved? How it should be done, etc.b. Not only should meaningful outcomes be defined, but in order to be able to use this outcome to “personalize” care, this outcome variable should also be able to highlight and accentuate “heterogeneity” across different populations. For example: in attention deficit hyperactivity disorder (ADHD), a patient with the outcomes measure CGI-I score 1 to 2 is defined as a “responder” and those not reaching this threshold classified as “non-responders”. Obviously, this outcome is not that sensitive in picking out differences across the different patients and highlighting the potential heterogeneity across the different individuals, because there may be patients who move from a score of 7 to 4 in the CGI-I, but it this patient group is still going to be classified as “non-responder” whereas in reality they are actually responding but are not meeting the threshold of “response defined”. If there was a more sensitive outcome measure that could be used to categorize patients into the different levels of response and thus “break them into different” sub-groups based on this outcome, then treatment could be potentially tailored to these different subgroups depending on the care needs of each subgroups. c. Suggestions to get to this point include looking at their genetics, coupled with their actual reported outcomes (i.e., PRO), or other patient specific characteristics. This was not mentioned or clearly elaborated in this chapter. Additional suggestions include development of novel innovative methods to identify subgroups of patients who may be responding better to one therapy vs. another. One example in using personalized medicine methods is a paper that was recently accepted in Journal of Medical Economics by Erder et al.7

Please provide your comments on Chapter 5 of the Draft Methodology Report.

11.) Comment: A key element in this report is the key focus on patient involvement and peer review process that has been stipulated. While VOI analysis is a useful tool, it should be supplemented with practical, real-world applications. With regard to research priority factors/focus, these seem very broad and may not be meaningful enough unless further expanded upon. Suggestions to consider would be to focus on heterogeneity among patients and effective mechanisms for meaningful subgroup evaluations and analyses, consider predictive modeling research approaches that can lead to results tailored to patient differences, aim for practical design and outputs that can realistically effect change with prescribers in a patient-centered treatment approach, and aim to assess treatment outcomes of comparators (vs PBO) to identify high value products based on patient outcomes.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

12.) Comment: The Translational Framework & Decision Tree could be very useful tools as long as either all possible study scenarios are considered or the interpretation of them leaves room for unique or new situations. This is important so that studies are not automatically labeled as ”wrong”, especially in grey areas. One way to do this would be to distinguish between minimum requirements and those characteristics that are important, but not required.The validation of this framework is very important to encourage future use and increase the level of confidence around the tool.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Chapt 7 pg 59. 7.1.2 “Researchers should describe the analytic approaches that will be used to address the major research aims prior to data collection” 13.) Comment: Instead of stating the above, it should state “Researchers should DOCUMENT the analytic approaches that will be used to address the major research aims prior to data collection”

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Chapter 814.) Comment: This chapter seems to be focused on “adaptive RCT design” approaches vs. other non-RCT designs that could be adaptive and useful in PCORI approaches. There could also be consideration of other types of modeling used (MSM or prop score model approaches employed) and a trial could be adaptive with regard to sub-group analyses being pre-specified. Indirect modeling comparisons might also be useful for informing trial design, rationale of trial, and endpoint selection or study power. The section on observational registries focused on the drawbacks and limitations of these registries and did not expand upon the benefits that could be achieved by using these methods, though they are briefly mentioned. The registry / real-world use approach is very likely to be applicable for collecting patient centered data and tracking outcomes of broader external generalizability.Chapter 8 pg. 80-81 “Adaptive designs are particularly appealing for PCOR because they have the potential to maintain many of the advantages of randomized clinical trials while minimizing some of the disadvantages.” and “The flexibility and efficiency that are gained in adaptive trials have to be balanced with the risk that such trials typically require a longer design period and involve more logistical complexity.”15.) Comment: There are a fair number of challenges in adaptive trials which need to be mentioned in the document in order to ensure clarity of decision making by the researchers considering utilizing them. First, adaptive clinical trials should model initially to predict the sample properly. This is not clear in the report. If it is not modeled appropriately, the sample may need to be re-forecast up or down and enrollment adjusted after the “adaptive” part of the trial is completed. This could actually mean that they could be longer or require more sample than conducting a RCT directly. It could be more efficient to simulate or model out trial results in lieu of an adaptive trial and be more time-saving. It could also mean that adaptive trials are more expensive than other RCT trials. Worse, when checking that whatever the primary measure of effect is in the adaptive trial at the interim analysis stage, it could be found that the required sample to complete the adaptive trial is so large that funding for such a trial is not available, especially since the majority of the costs for a study are typically at study start-up (e.g. enrolling sites, patient incentives, etc). Lastly, because the interim sample and analysis are often less powered than the full trial, one could power the remainder of the trial based upon the initial interim analysis which could be incorrect simply due to study changes or low power. In medical device and diagnostic studies, this is particularly problematic since there is an operator learning that must be taken into account. The results for most human-operated devices or diagnostics initially will have more variation than later on when they become experienced users. The effect can cause great variation in the reliability and estimates for powering the remainder of the study.Chapter 8 pg. 82 section 8.1.3 “This should include any statistical models used either during the conduct of the trial or for the final analysis, prior probability distributions and their basis, utility functions associated with the trial’s goals, and assumptions regarding exchangeability (of participants, of trials, and of other levels).”16.) Comment: when there is the possibility for multiple cut-offs or probabilistic approaches via Bayesian approach, sensitivities should be explored and modeled out to understand the potential impact of the extreme assumptions at the very least. Selection of which prior probabilities to use can cause the greatest variation in Bayesian adaptive trials. Chapter 8 pg. 83 “For example, these studies typically require simulations in the design phase in order to define the error rates, and descriptions of the design both in protocols and published papers must include more elements than a non-adaptive trial. Good adaptive trial design requires pre-planning and specification of procedures at the outset.”17.) Comment: We agree with the comment above, but just to clarify and suggest: the MAIC approach is an appropriate method which could be used for this tool as well. It can “simulate” a trial outcome by matching patients within current trials. It gives the prior probability for an outcome of interest as long as that is the same outcome of interest that is sought for the adaptive approach; it can be very useful. The same applies to pg. 94- “Support development and use of software for adaptive trials that can simulate complex designs.” The MAIC approach can be a method that could be modeled out in the future using such software. Chapt 8 pg. 85 “Standards for Data Registries” 18.) Comment: Systematic literature review and synthesis of some kind is required as the first step before any registry can be planned to know what has previously been collected and what hasn’t been collected, but should be in the registry (in the interest of the patients).

Author

Anonymous

Date

Thursday, September 13, 2012

Zip Code

27510

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

But see uploaded comments on composition of Methodology Committee.

2. What methodological gaps should be addressed in future reports?

Standards for methods of qualitative data collection and analysis within the context of comparative effectiveness research..An empirical evaluation of the benefit of including patients as partners in CER research.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

..with the addition of more information regarding the integration of patients as partners in research..

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

When finished you may have enough to publish a book.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Feedback-to-PCORI-on-Draft-Methodology-Report.docx

Author

Personalized Medicine Coalition

Date

Thursday, September 13, 2012

Zip Code

20005

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PMC-PCORI-Methodology-Committee-Report-Comments.pdf

Date

Thursday, September 13, 2012

Zip Code

46285

Stakeholder Category

Industry

2. What methodological gaps should be addressed in future reports?

Please see uploaded document or recommendations and comments provided on specific chapters.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/FINAL-Lilly-Comment-Letter-on-PCORI-Draft-Methodology.doc

Please provide your comments on Chapter 4 of the Draft Methodology Report.

•Increasing the relevance, meaningfulness, and interpretability of data to all stakeholders (including patients) is an important progression in conducting and reporting scientific research as it pertains to drug development. An important component of this is including measures that are most relevant to patients and recruiting representative populations. Consideration of patient-centeredness during study design is very important in understanding patient’s needs, deciding inclusion/exclusion criteria, developing the data collection forms, and evaluating the feasibility of the study. •The proposed standard (4.1.1), states “At a minimum, patient informants should be engaged in formulating research questions; defining essential characteristics of study participants, comparators, and outcomes; monitoring study conduct and progress; and disseminating results.” Although it is key to include patients in determining the relevant questions for research, there are several issues regarding further discussion with participants following the initiation of a study. These issues include, but are not limited to, patient confidentiality, ethics, legal, and regulatory concerns, comprehension of medical practices and study design, as well as methodological complications with ‘intervening’ with patients under Good Clinical Practice. PCORI should consider how these issues should be addressed in the context of the proposed standard. •We recommend that the standards specifically state that researchers should follow all applicable laws and regulations.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

•This section provides a solid framework for research and will help focus research where it is needed. However, the use of systematic reviews as a means for identifying gaps in research priorities is a complex concept, and the guidance provided in the draft methods report is vague. While systematic reviews are a key component of PCOR, patient-level data may also be used to perform hypothesis-generating analyses using methods such as regression models, cluster analysis, CART analysis, etc. Therefore, this section should include additional methods for studying efficacy and effectiveness at the patient level in addition to systematic reviews.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

•The non-randomized study section of this chapter should incorporate observational study designs. Furthermore, PCORI should update Figure 6.2 entitled Decision Tree for Comparative Effectiveness of Therapeutics to reflect the appropriate nature of observational study designs. Observational research does not rely on recruitment strategies and instead considers inclusion/exclusion criteria.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

•It would be a benefit to have PCORI endorse standard definitions that would apply to all patient-centered outcomes research (e.g. definition of patient sample, definition of intervention group, and definition of relevant outcomes).Causal Inference:•The draft report on causal inference provided sound scientific rationale and a clear example supporting the need for proper bias adjustment techniques in non-randomized studies – as well as a few suggested key principles for analysts to follow. The section outlining causal inference methods should be expanded to address the strength, weakness, and scope of different methodologies in regards to adjusting for selection bias. •There are various causal inference methods which may not apply in every situation, make different assumptions, may be more appropriate than another, and may demand different sensitivity analyses in each situation. We recommend PCORI provide more clarity in order to provide guidance that will lead to trusted and reliable information. The information provided in the draft methods report is a starting point, but much more detail is needed. Without more specificity or guidance, there is a risk that conflicting results on the same question will occur without clarity as to why or how to resolve the differences.•The causal inference standards are limited and only focused on a small number of analytical methods. There is only a brief mention of multivariate regression, propensity scoring, and instrumental variable, meaning more information is necessary to provide clearer directions and standards for researchers. More guidance is needed in regards to retrospective data issues, pre-specification, multiplicity, and replication. •There is little guidance and little understanding of the operating characteristics of causal inference research based on observational data. Without an understanding of causal inference, it is difficult to appropriately interpret and make healthcare decisions on this data. Detailed guidance would be a beneficial step toward appropriate interpretation of such data. Heterogeneity of Treatment Effects (HTE):•The recognition of the importance of HTE in the draft methods report is instrumental for advancing patient-centered research. The proposed standards are most applicable in subgroup identification and HTE analyses. They are a logical and defensible set of standards, analyses types, and criteria for each of the HTE study types. Descriptive HTE analysis is a novel concept, which will provide more robust data to enable future meta-analyses. •The draft report emphasized the importance of an interaction test to support the evidence of heterogeneity of treatment effects, however, it did not provide practical instructions regarding implementation of the test. PCORI should clearly define the level of evidence required to support a meaningful interpretation of the interaction effect (e.g. subgroup mean, least square mean) as well as best practices for selecting variables of interest and conducting the tests. Furthermore, we recommend that PCORI should state that interactions beyond two-way interactions are to be avoided. The interpretation of interactions beyond two-way becomes more difficult to interpret and implement in practice.•Not all subgroups in HTE are equally important or meaningful for use in treatment decisions and predictive models. This should be acknowledged, and the use of causal diagrams to address this issue should be included. Causal graphs (also known as causal diagrams), such as Directed Acyclic Graphs (DAGs) or Path Diagrams, offer researchers a better insight into causality and confounding by visually representing the a priori assumptions about the underlying biological mechanisms or causal pathways that relate exposures, outcomes, and covariates.•The standards for HTEs list a few examples of variables such as gender, age, co-morbidity, and lifestyle. There should also be consideration for race, ethnicity, and/or regional factors. The FDA recognizes and has done workshops in the past that address the multi-regional HTE. Finally, disease state specific variables should also be included in the standards, and a clear rationale for collection and analysis should be provided.Preventing and Handling Missing Data:•The draft methods report makes a causal statement that clinical studies often are based on restricted populations because the researchers are attempting to avoid missing data which might not be realistic. Clinical studies often have restricted enrollment, but avoidance of missing data is not the primary cause for these design choices. The restricted population enables easier causal interpretation. The standards for missing data should emphasize statistical methods for handling such data and that evaluating the effect of missing data on the interpretation of results should be carefully planned and executed.•Guidance for trials is included in this section, and should incorporate methods related to observational study designs, including database studies. The proposed standard (7.4.4) details reasons for dropout and missing data, and we recommend PCORI include an additional section within this standard that includes possible reasons for missing data when conducting a study using secondary datasets. Issues such as enrollment in claims databases (ICD-9 codes) and data gaps when using electronic health records are critical when applying statistical methods to these data with missing gaps especially as it pertains to the various categories of missing data such as Missing At Random (MAR), Missing Completely At Random (MCAR), and Missing Not At Random (MNAR).Data Networks:•This section captures the critical elements in data networks. It describes the core aspects to consider for quality data from a data network in a high-level comprehensive manner: Data Harmonization (structure and content), Data Privacy, Data Ownership, Intellectual Property, Data Access, Network Establishment and Network Maintenance. The rationale for standards is also logical and compelling.•Standard 7.5.5. outlines the need for standardized terminology encoding in data networks. However, multiple terminology systems for electronic health care databases exist with different terminology such as ICD codes and READ codes. It is unclear if standardized terminology encoding refers to within one terminology system or across different systems. A standard set of terminology would be ideal, but it would be very difficult to achieve broad agreement and use. In addition, mapping standards are needed to convert codes from one system to another and should be considered for future work.•On report page 78, the first sentence of the second paragraph mentions the importance of “quality of data,” but no standard specifically addresses the quality of incoming data to networks. PCORI should consider including a standard that addresses the quality of incoming data.•In addition to the recommended actions on page 79, for future work, we propose PCORI consider promoting a priori industry standards (including structural, syntactic, and semantic standards) for electronic health records to support easier harmonization of data elements across the spectrum of care. For example, the Certification Commission for Health Information Technology (CCHIT) has established certification requirements and standards for electronic health records that PCORI could recommend. http://www.cchit.org

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Adaptive and Bayesian Trial Designs:•The section and supplemental materials are in line with current practice and raise the awareness of adaptive designs and Bayesian methods in general. The supplemental materials are written by some of those on the forefront of adaptive designs and include relevant references, etc. The use of Bayesian trials methods has broad potential and more information about methods for this type of trial should be included in the final report. In addition, we recommend PCORI consider including additional sections and standards for various studies including: newly emerging methods for observational studies primarily evaluating safety issues including high dimensional propensity scores, self control study design, and sequential probability testing.Data Registries:•The justification of the use of data registries is relatively complete. Defining minimal standards for data registries to ensure quality, standardization, and rigor is a step in the right direction. For low-prevalence conditions, registries many be the only opportunity to have meaningful power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches. Randomized trials and other observational studies with the ability to detect interactions with sufficient power that accounts for multiplicity of testing multiple subgroups may be rare, and registries may fill that important need.•Additional benefits of registries include: ability to provide data on populations not typical in RCTs (e.g. children, elderly, minorities, pregnant, and co-morbidity); ability to examine impact of physician practice behaviors on quality of care, prescribing preference, consequences of healthcare decisions, etc.; and usefulness for signal amplification, particularly for rare outcomes. •The proposed minimal standards for data registries tend to be more directed at good practices for designing new data registries. Furthermore, there are three uses of data registries for PCOR studies that call for somewhat different quality considerations and standards: (1)Observational PCOR studies for which a Registry is being newly designed to provide data.(2)Observational PCOR studies which propose to use data elements from an existing ongoing Registry to address a research question.(3)Observational PCOR studies which propose to modify an existing ongoing Registry to address a research questions.We recommend that PCORI consider including standards in the report that account for these types of registries and address related quality requirements for each. •When using an existing registry, the standard should require a feasibility assessment of a proposed registry based on its history of operation (taking into account potential sponsor or clinician biases) and quality to date.•There are numerous centralized computer assisted registry technologies that are based in phone, internet, laptop/computer, home interview, daily diary/experience sampling, and smart phone technologies. These registries are less tied to immediate treatment setting and patient visits, and may offer more control in minimizing follow-up attrition and missing data. In addition, these types of registries may be more suited not only to patient needs, but also collecting patient-reported outcomes. The proposed data registry standards should recognize the emerging use of non-traditional registries and survey panel registry designs versus site-based registries.•A comparison group should be included in the standards for conducting research using data registries.•There is no guidance in the standards for data registries relating to whether registry patients for proposed PCOR can also be simultaneously enrolled in clinical trials or not. Clinical trials impose specific treatment protocols, potentially confounding the registry sample and possibly changing the generalizability to real world outcomes.•When outlining the use of registries, the document should include an emphasis on the advantage of having a larger sample size in a data registry. Particularly for conditions and rare diseases with small patient populations, registries may be the only opportunity to have meaningful statistical power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches. •In standard 8.2.4 regarding the safety and security of data registries, the inclusion of a process for reporting serious adverse events associated with treatments should be included.•Standard 8.2.9 should include the proper methods to address biases and effect modification as well as confounding. Sensitivity analysis to assess the impact of key assumptions in studies should be added. In addition, an issue with many registries is whether individuals can be enrolled multiple times within a site or between sites, and standards should discuss how this can be avoided while protecting patients’ privacy.•Standards for data registries should include information to assist researchers in ensuring the registry population used within a study is comparable to the target population. Studies of Diagnostic Tests:•The report should endorse the Grace Principles (2010), which describes a hierarchy of evidence for observational research on comparative effectiveness that can be used by decision-makers, as well as key elements of good practice including defining research questions and methods a priori; collecting valid, clinically relevant data; analyzing, interpreting and reporting data, including sensitivity analyses and alternative explanations for findings; and conducting these studies in accordance with accepted good practices. Currently, there are no clear standards or requirements for what infromation should be captured or how long it should be stored and made available. http://www.pharmacoepi.org/resources/GRACE_Principles.pdf•The report should differentiate between diagnostic or screening lab tests for biomarkers versus subjective assessment tools where an objective biomarker is not available for patient-reported outcomes (e.g. measurements of pain, depression, or anxiety). •We recommend that PCORI consider discussing the significance of understanding the sensitivity and specificity of diagnostic tests and the importance of having this information readily available for medical personnel and patients alike. This was not included in the draft report. Finally, it would be beneficial to reference in the draft report that the current process of drug and diagnostic tests approval lacks a systematic approach to benefit-risk analysis, leading to inconsistency, lack of transparency, and an inability to challenge or defend decisions- in both the pre-approval and post-marketing setting. The risk-benefit analysis that currently goes into regulatory decisions appears to be ad hoc, informal, and qualitative, without defined and tested algorithm or summary metric that combines benefit and risk data that might permit straightforward quantitative comparative analysis. This is an important gap that needs to be addressed.

Date

Thursday, September 13, 2012

Zip Code

19850

Stakeholder Category

Industry

2. What methodological gaps should be addressed in future reports?

AstraZeneca supports the comments on the PCORI Draft Methodology Report submitted by PhRMA as part of this public comment opportunity, and offer additional comments on specific provisions or standards. • Please consider clarifying how the Committee selected certain methods & the criteria used for not selecting others. • Consider addressing standards (such as for patient registries) that would apply to other types of studies such as observational studies. • Citations are not included in the Report. An appendix including citations would be helpful• Several extensive reports were commissioned and referenced in the draft report. It would be helpful to provide a synopsis of the finding of these research reports highlighted in the report body.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Page 43, table 6.1. The choice of specific randomized clinical trial designs picked out is unusual, and a standard parallel group design is not mentioned. The Delayed Start trial is not commonly used. (See D’Agostino et al., http://www.nejm.org/doi/full/10.1056/NEJMsm0904209). There are many other trial designs where patient allocation or other features differ in different phases that are not mentioned, so it seems odd to pick out the Delayed Start trial design for particular attention in the table.Also in Table 6.1, the definitions for selected study designs for Assessing the Effectiveness of Interventions could use one of many accepted definitions using authoritative textbooks in the fields of pharmacoepidemiology and experimental design. The description of a controlled time trend analysis design as a difference-in-difference or quasi-experimental design is unfamilar and should not infer that all quasi-experimental designs follow this designation (see Cook TD and Campbell DT. Quasi-Experimentation Design & Analysis Issues for Field Settings. Rand McNally 1979.)

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Page 59, section 3.1.4 Please clarify what is meant by “viable”. See: http://www.google.se/url?sa=t&rct=j&q=eu%20choice%20of%20comparator%20&source=web&cd=4&ved=0CFYQFjAD&url=http%3A%2F%2Fwww.eunethta.eu%2Fupload%2FPublic%2520Cons%2FJune%252029%25202012%2FChoice_of_comparator_draft%25204%2520clean%2520final%2520version.doc&ei=KJAoUPbVLYH28gT18IGwDg&usg=AFQjCNH_P9HfEeB-6lh9ZxT9ZKKarNjgLg. Page 59, Recommendation 7.1.2. A priori Specify Plans for Data Analysis – the intended statistical methods and their underlying assumptions should also be included in this plan.Page 61 Standards for Causal Inference Methods. Consider clarifying the need to identify potential confounding factors at the design stage & ensure efforts to collect data on confounding factors during the study.Page 62, paragraph 3 gives the impression that propensity scores are a way to address both the limitations mentioned in paragraph 2 the lack of a clearly articulation of “cause” and being able to adjust only for observed confounders. Although this is corrected on page 63, please consider rewriting paragraph 3, page 62 to minimize confusion.Chapter 7, section on standards for causal inference. Consider including advice on when causal inference may be feasible. For the mentioned methodology (regression, matching, instrumental variables) to work the treatment or exposure groups must overlap to a certain degree, meaning that there must be at least some patients that could be considered to be comparable. If this is not the case the regression analysis will be based on extrapolation and rely heavily on the model assumptions and any matching method will give a very little analysis set to work with. Typical situations that results in comparable groups are when existing treatment guidelines are unclear leading to variability in treatment practice. Also, on Page 64, Causal Inference Standards. Consider rewriting standard 7.2.5 or add a new section that includes assessment of overlap in the distribution of confounders used in a multiple regression model to ascertain the degree of extrapolation involved in the analysis both for each confounder included in the model as well as over all using either propensity scores or the convex hull. This is often forgotten and leads to conclusion being based on very strong model assumptions that go untested if not explored.Page 65, HTE Use of the phrase “traditional clinical research” here may cause confusion. Systematic variability in the treatment effect can be masked (with or without intent) in a study of multiple designs. Page 66 paragraph 4 line 10. Please consider rewording the definition of an interaction test. As worded, the current definition may not be entirely correct as it mentions stratified treatment effects.Page 68 table 7.1 – Row 3 – there should always be at least some prior scientific rationale for treatment effect heterogeneity in a subgroup to minimize false findings, even if the inferential goal is descriptive or exploratory. Data dredging multiple subgroups with no scientific basis is not good statistical practice.Row 4 – There is no reason why exploratory analysis subgroups could not be pre-specified, and it would be good practice to recommend pre-specification. Row 5 – Pre-specifying subgroups in exploratory analysis as recommended above would remove the difficulty in controlling for multiple testing. It should always be reported how many tests were performed, even if error rate control is not formally applied, to aid in interpretation of the likelihood of the findings being true or false.Page 69, 7.3.2 and 7.3.3. Studies should be designed to enable such analyses to be conducted in the most appropriate way. Suggest the recommendations consider design features (eg, stratification) in addition to analysis features.Page 69. 7.3.3. While statistical power of any HTE analyses should be stated up front, it raises the possibility that this will encourage the interpretation of the combination of high power and high p-value as evidence of absence of HTE which is not necessarily the case. Please consider rephrasing. Page 69, 7.3.4: Views differ on the need to always perform an interaction test as part of the investigation of HTE. The appropriate methodology should be linked to the investigational goals of HTE (see 7.3.1). In many cases, an interaction test will have low power and will be difficult to interpret (e.g., if there are many degrees of freedom). In addition, the concept of qualitative versus quantitative interactions should be discussed in this context. Lastly, an important technique would be simulation, but the present report explicitly excludes this discussion. Consider adding a discussion of simulation in future iterations. Page 69, Recommendation 7.3.4 Please clarify whether the interaction test recommended is a global test (i.e. testing for any interaction across all subgroup clusters) or multiple tests for each subgroup cluster. Also, suggest that the confidence intervals need not be 95% (e.g. if an adjustment for multiple testing is applied then they may not be). Forest plots can be very informative in assessing HTE, so it is not clear why these are only recommended for analyses of descriptive intent or why they cannot inform the assessment of HTE. Indeed, relying on statistical testing alone is not recommended, as tests may be underpowered, or may detect differences in subgroup effects that are statistically significant but not clinically relevant.Page 72, 7.4.3: The bolded section header “all forms of single imputation are discouraged” has stronger language than the text that follows. We believe that, as a sensitivity analysis (like in 7.4.5), single imputation has a place so it is inappropriate to unilaterally discourage it.Page 75, second paragraph. Consider adding a discussion of databases containing the same patient multiple times due to switching of health care providers.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Adaptive & Bayesian Design. In future iterations, please consider clarifying how to link research questions to designs rather than providing details on specific designs. Page 81 Standard for Adaptive and Bayesian Designs. Please clarify who should perform the analyses to inform adaptation while the study is ongoing, and who would have access to the results. Knowledge of accruing data and results while a study is ongoing can introduce bias into study conduct and subsequent final analysis and results. It is good practice for the analyses to be done by an independent statistician and reviewed by an independent data monitoring committee (EMEA reflection paper on adaptive designs, 2007). See: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003616.pdf.Consider acknowledging the potential disadvantages of adaptive designs. According to Fleming, “these procedures have several undesirable properties, including lesser statistical efficiency, reduced interpretability of primary outcome results, basing design changes on unreliable interim estimates of efficacy, risks to the integrity and credibility of the trial, loss of flexibility to use emerging results from external sources to alter key design features, and overemphasis of the importance of statistical significance relative to clinical significance.” http://onlinelibrary.wiley.com/doi/10.1002/sim.2641/abstract

Author

American Academy of Family Physicians

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CAFM-AAFP-PCORI-Comments1.docx

Author

Council of Academic Family Medicine

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

See attached document

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CAFM-AAFP-PCORI-Comments.docx

Author

Dominick Esposito & Jeffrey P. Ballou

Date

Thursday, September 13, 2012

Zip Code

8540

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Please see our comments.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Response-to-Draft-Methodology-Report.docx

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see our comments in the attachhed file

Author

Richard Tannen

Date

Thursday, September 13, 2012

Stakeholder Category

Researcher

2. What methodological gaps should be addressed in future reports?

See my comments in the uploaded document

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-RESPONSE-8-13-2012.doc

Author

Joanne Lynn

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

What is in the report is consistent, just incomplete in not addressing methods needed to work with system improvement and learning organizations.

2. What methodological gaps should be addressed in future reports?

If not addressed in a revision of this report, these two major gaps deserve development in future reports: 1. Methods from system management fields: e.g., operations research, process improvement, realist evaluation.2. Standardization of data, e.g., through merging CARE/MDS/OASIS into meaningful use as a uniform assessment strategy for Medicare’s elderly population.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Sure – but it mainly addresses problems in standard “frequentist” research design, so it would be both a weighty tome for some (with a straightforward “drug trial” design) and irrelevant for others (with projects that need to use other methods).

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Regarding distribution: Publish a shortened version in Annals of Internal Medicine, like the US Preventive Services Task Force does (and the long version can be on-line there as well as on PCORI’s site). Then it is readily retrieved through any medical literature search engine.Regarding content: I am very concerned that PCORI’s methodology does not seem to have made room for learning organizations, Shewhart statistics, control charts, quality improvement, realist evaluation, or any other method that will be essential in reshaping service delivery to enable patient-centered care. Please see uploaded document.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/JoanneLynnCommentsPCORI_Draft_Methodology_Report.doc

Please provide your comments on Chapter 3 of the Draft Methodology Report.

The "standards" on page 20 apply to an important subset of the research that PCOR covers, not to all. It presumes that "patient-centered outcomes research" has entirely to do with patient decisions in a vacuum - as soon as one has to deal with complex systems and environments, other methods and standards are needed.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

These are generally good, though largely untested. One would expect them to mature some as difficult situations arise.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Again - this is one important approach and good standards for that approach. If one is working on a complex system-anchored problem like "falls in the elderly" or "teenage pregnancy," other criteria and methods will apply.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

The "research categories" in Figure 6.1 again illuminate the narrow nature of the methodology report.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

The methods issues are conventional and appropriate for conventional trials. But each would have exceptions in a dynamic learning organization. For example, one might use a simple tally or count rather than a validated scale. One might develop a better metric during the process.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

These are extremely restrictive design elements for adaptive design. It is true that pre-planning analyses and adaptations on the basis of analyses eliminates some biases. However, it is not an iron-clad rule. Again, have a look at the IOM report on learning organizations, or check on the improvement potential of a model like Intermountain. Do you really intend to denigrate those methods? Consider the implications of the sentence at the end of page 93: "As patients enrolled in RCT tend to be more homogenous, observational studies may be used to augment results from the definitive prospective studies." Taken seriously, this would mean that all of epidemiology is merely useful to augment results from (mostly non-existent) RCTs. It would mean that learning from process control is always second-rate. It elevates RCTs to "definitive," even though most are quite limited. If, in order to help patients overcome addiction, one has to use public information channels, then one cannot have an uncontaminated control group. If, in order to improve capacity for patient self-care, one needs to change the behaviors throughout a hospital, again, one can map the progress and the patients' responses, but one cannot have a matched control group.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Next steps need to include a broader perspective, or an authoritative demarcation of this limited perspective so that CMS, HRSA, ASPE, ARHQ, and private funders are on notice that PCORI will not be using those methods.

Author

AcademyHealth

Date

Thursday, September 13, 2012

Zip Code

20037

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AcademyHealth-PCORI-MC-FINAL1.pdf

Author

Alliance for Regenerative Medicine

Date

Thursday, September 13, 2012

Zip Code

20006

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see general comments uploaded attached document.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Social media and web is best.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ARM-PCORI-comments-sept-2012.docx

Author

Families USA

Date

Thursday, September 13, 2012

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report-Comments.pdf

Author

American Occupational Therapy Association (AOTA)

Date

Thursday, September 13, 2012

Zip Code

20814-3449

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While it's important to strive for methodolgical rigor, some of the populations that PCORI was designed to help will not have high-quality evidence, due to the rare occurrence of the disorder/condition. It will then be important to offer guidance for designing and appraising other research designs, including single-subject designs, if those are the only type of evidence available.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

We would like to see additional information about other designs (see # 2 comments above).

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

In addition to research communities, please disseminate the final report to healthcare associations, including the American Occupational Therapy Association (AOTA) and other allied health associations. We often have researcher groups/communities that can be helpful for efficient dissemination.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments_PCORI_Methodology.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

We are pleased that the purpose of PCORI was emphasized; “to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed…” Another point that could be added is that the best available evidence, as Sackett (2000) urged, should be considered by the patient and professionals in order to make informed decisions. Otherwise, there is a risk that researchers, clinicians, and policy-makers interpret the term “quality” as Level I evidence only, ignoring other levels of evidence. With rare disorders and many conditions seen in rehabilitation, study designs other than the randomized controlled trial can be helpful to inform patients and clinicians’ decisions, but may not always be considered in the development of practice guidelines. We agree that there is a lengthy timeframe between research results to implementation of research into practice. A discussion between researchers, journal editors, and guideline developers could offer ideas for new strategies and communication/dissemination methods, while preserving researchers’ and journals’ rights.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

This chapter offered a comprehensive explanation about the process of the committee’s work.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

This chapter was organized and clearly explained the phases of PCOR. We particularly liked the focus on engaging patients in prioritizing and refining research topics, and look forward to PCORI’s resources and guidance on this critically important process of patient-centered care.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

We agree with PCOR’s approach to not prescribe a particular method of patient engagement, given the types of studies and questions PCOR will cover.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

We wholeheartedly support obtaining patients’ input for research topic selection, but we also support clinicians’ input as well, since they often have clinical questions that researchers have not yet addressed.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

In Table 6.1, Selected Study Designs for Assessing the Effectiveness of Therapeutic Interventions, would Regression Discontinuity Designs be a valid non-randomized design? With its high internal validity, Regression Discontinuity Designs could be useful.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

The section discussing Missing Data is greatly appreciated. There are published intervention studies who have not handled missing data in a valid way, such as including the last collected data from subjects who dropped out in the analyses, citing intention-to-treat analysis. It would be helpful to inform researchers more about options for analyses when there are certain percentages of missing data.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

We appreciate the standards for adaptive and Bayesian trial designs. We agree that registries have the potential to contribute valuable information about PCOR, but we think there should be more forethought to the design and structure of registries. For example, it is critically important to identify the research questions, as much as possible, in advance, so that the data collected can answer these questions. The quality of data’s output is commensurate with the quality of data’s input.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

We look forward to future resources and education/training opportunities to build capacity among researchers to understand, deploy, and improve PCOR methods. Our association is willing to help with informing and disseminating opportunities and resources to patients and clinicians.

Author

Sanofi US

Date

Wednesday, September 12, 2012

Zip Code

8807

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-9_12_12.pdf

Author

Iris Simonis

Date

Wednesday, September 12, 2012

Zip Code

77536

Stakeholder Category

Patient

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Chapter 6 Scope P. 42 Design study is listed as either observational or experimental. I see research being observational or interventional. I feel research is unproven or experimental.Appendix D-2 P. 153#19 Quality outcomes data measured by length of stay, cost, and by providers could also help patients make decisions.Appendix D-3 P. 156#4 Give considerations to over medications of elderly with regard to impact on quality of life, placement, and cost.Appendix D Translation Table Box 1Reference is made to burden to individual, but does not address access to care or research center and financial burden.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Distribute to support organization of various diseases, copies to physician waiting room, social mediaThe terms benefit and harm are used in multiple places in the report. Harm implies negligence with a negative outcome, risk may be a better choice.Report is too long, notice how few patients comments were received. I would suggest a summary report in additional to executive summary and abbreviations list.

Author

Philips Healthcare

Date

Wednesday, September 12, 2012

Zip Code

1810

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Philips-Final-Comments-re-PCORI-draft-report-D0439242.PDF

Author

American College of Rheumatology

Date

Wednesday, September 12, 2012

Zip Code

30319

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ACR-Comment-Letter-to-PCORI-Sep-2012_final.pdf

Date

Wednesday, September 12, 2012

Author

American Academy of Dermatology Association

Date

Wednesday, September 12, 2012

Zip Code

20005

Stakeholder Category

Other

1 Comments

Please read attached comments.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report-AADA-Comment-Final.pdf

Author

Academic Pediatric Association

Date

Wednesday, September 12, 2012

Zip Code

22101

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/APA-PCORI-Comments.pdf

Author

Allergy & Asthma Network Mothers of Asthmatics

Date

Wednesday, September 12, 2012

Zip Code

22101

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Kudos for looking beyond the disease to the person. Policy must be humanized if we are to solve health care cost issues and return decision making back to those of us who live with the outcomes.

2. What methodological gaps should be addressed in future reports?

Would love to answer this questions thoughtfully in a room filled with people who've read the report. There are always gaps and missing links - and this format may not be the most helpful way to do the report justice.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

That said, NHLBI's Asthma Guidelines, if applied in policy and practice, would eliminate asthma death and suffering, put families back to work, keep kids at school and reduce hospitalizations, emergency department visits, urgent care visits and improve productivity. Poorly managed symptoms cost more than getting well and staying well. It's the best example of patient centered care and it was developed by national experts with federal funding lead by NIH/NHLBI/NAEPP. It was adopted by some but not all federal agencies or state public health programs receiving federal funding. And policymakers change over time and are influenced by a variety of factors. AANMA and many other nonprofit organizations consider it the most defining document to revolutionize asthma care in our country and should be continually updated.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

How can I participate? How can Allergy & Asthma Network Mothers of Asthmatics help?

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Patient input is critical. I am unclear what your intent is.

Author

Dev Pathak

Date

Tuesday, September 11, 2012

Zip Code

34203

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While the report is excellent from a researcher's perspective, I don't believe one area of research from the patient perspective seems to be missing. My suggestion refers to two questions adopted by the committee: 2. “What are my options and what are the potential benefits and harms of those options?” 3. “What can I do to improve the outcomes that are most important to me?” The current report does not address the methodological issues related how the patient receives the information about options and how s/he assimilates the information to make the final decision in selecting the option which would result in improving the outcomes that are important to her/him. There are many steps involved in making the final decision by a patient. And, there are many decision rules used by an individual in selecting an alternatives. Psychological and marketing literature have outlined many decision rules such as compensatory vs. noncompensatory rules. Unfortunately, these decision rules are not tested in health care environment. If only information about options is provided to patients without understanding how patients combine the information about the options to make his/her decision. the information provided may not achieve its goal/s! Understanding the process used by the patient to obtain information about the options AND how s/he assimilates the information to improve his/her decision to achieve his/her goals is as important a methodological consideration as outlining the steps described in the current report to improve the research process!

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

The current approach is fine, but presentations of this report in variety of setting may be useful to popularize the report.

Author

Martha Mihaly

Date

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report_CAHMI-Memo.pdf

Author

Anonymous

Date

Monday, September 10, 2012

Stakeholder Category

Clinician

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

1 Comments

We have lived through so many attempts to help patients; PORTs; clinical trials; clinical trial registration; FDA standards; quality and safety; AHRQ, now this. Same ol' same ol' not one new idea in the report at all.Most problematic; many on the committee have written and appropriately critiqued methods of science and yet fall victim to the same limitations.The main problem with this report is that it does not advance the science for individuals. As someone who counsels those with tough medical decisions, the information we have is so problematic. I don't see you advancing this in the report at all. You are talking population based science in a patient informed decision making domain. Wrong process; time to go back to the drawing board.

2. What methodological gaps should be addressed in future reports?

First, the report is hard to read; well written but complex and all ove the place. You should be able to make the document be 3 pages long and get to the essesence of what is new and innovative. I could not find anything advancing science for the care of individuals at all. The only method to advance science for individuals is to include all individuals with single options for care; picked by public consensus not researchers. Dabbling in RCTs is a waste of time and money. Time to face it; I am not a heratic; I simulate trials; understand economics; can build models of care and have researched many aspects of care. Our scientific process has failed us; we need a new one for patients as individuals.

Sunday, September 9, 2012

Zip Code

50200

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Being transparent in developing policies to encourage public registration of all PCORI studies and the sharing of study, protocols, statistical code, and data, there is nothing than a person of any race that honors more than getting to know that he or she is dealing with people who are transparent and much devoted in their work the way PCORI is with their staff in running their work in all ways and means. In fact Neemeka herbal clinic and research centre honors nothing than being transparent in all that it does on patients and in research study, so that patients gets the really high quality herbal medicine findings needed especially on these chronic and multiple chronic diseases that can serve patients on large scale population worldwide.Forming a standing committee within PCORI to recommend appropriate methods for data sharing and to ensure that proper scientific credit is given to those sharing protocols, code data, this shows PCORI’s profile(layout)on how the organization operates is committed with their work by ensuring that proper scientific credit is given to those sharing protocols, code and data. Speeding up implementation of standards in funding announcements, peer, review and other internal process, PCORI staff developed templates for the preparation and review of proposals that incorporate the key elements of the standards; this makes the process of working to become very easy to operate or to run in all channels of work, to be keenly monitored, since some standards apply only to certain types of studies, a portfolio of templates applicable to various study designs should be developed.Support development and use of software for adaptive trials that can stimulate complex designs, broaden experience with adaptive trials for PCORI, perhaps through funding of a cohort of adaptive trials on priority topic areas. Mentor investigators and develop a ‘how to’ guide and a forum to share experiences with adaptive trials, develop a course work and training opportunities for statisticians and other methodologists interested in developing expertise in adaptive trials. Sponsor an institute of medicine committee to develop standards for research on medical tests.

2. What methodological gaps should be addressed in future reports?

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Should ensure that a sufficient number and range of topics are considered, before topics for research findings are selected. Engagement of multiple stakeholders, especially patients, involving patients and other stakeholders in developing questions for systematic review, researchers can compare what people want and need to know with what is and is not known. As a result, systematic reviews can identify gaps in knowledge that underlie UN certainty among patients and clinicians. Sometimes systematic reviews can generate new questions. For example, a pooled analysis of several studies can reveal an important finding that was not evident in the individual studies.Value of information analysis may be used to identify questions that have the greatest potential to improve population health by considering uncertainty in the health benefits and risks associated with alternative treatment choices, the ability of research findings to alter uncertainty, and the resulting care decisions. Reviews process identifies those proposals most likely to fulfill PCORI’s objective and agenda.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Neemeka herbal clinic and research centre the personnel investigator would like to upload full findings that are in the letter of support of Kenya Medical Research Institute (Centre for Traditional Medicine and Drug Research, P.O.BOX 54840-00200 NAIROBI, Kenya) Kemri and also a letter of support Kenyatta University office of the chairman Department of Health Sciences, Centre for Complementary Medicine and Biotechnology P.O Box 43844 Nairobi Kenya. Neemeka herbal clinic and research centre plus entire staff is in atonally research of herbs that are medicine for these chronic diseases. The centre is working round the clock in ensuring that they are in research for high quality findings required for these chronic diseases, in near future the solution will be achieved. In the letter of Kenya Medical Research Institute (Kemri) page2, second paragraph states that search of available literature revealed no published report on the toxicity of s.longepedunculata in humans despite the wide-spread use of the plant in various herbal remedies. Considering the potential toxicity of s.longepedunculata, herbal practitioners should be educated on this especially when they recommend this plant as part of complex regimen in the long term management of chronic illness.Neemeka herbal clinic and research centre would like to appreciate and thank the entire staff of PCORI about the good work they are doing for patients especially those with chronic and multiple chronic diseases worldwide, working hand in hand in all means and ways in doing more research and being honest in submitting the really high quality findings needed for these chronic diseases that gathers large population worldwide. Neemeka herbal clinic and research centre also reflects thanks giving to anybody whose main objective is putting patients first by funding researchers in all means and ways as per research study is concerned, so that high quality findings is achieved for these chronic diseases.Neemeka herbal clinic and research centre is doing more research on Diseases like;-Heart diseases-Diseases of liver-Kidney problems-A.I.D.S Virus-Diabetes-Lungs problems-Cancer-

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/DOC-1.jpg

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Establishing a specific research agenda is a core duty of PCORI unless the mismatch between current research priorities and information needs of patients and clinicians is addressed.Neemeka herbal clinic and research centre fully supports a core duty of PCORI unless the mismatch between current research priorities and information needed of patients and clinicians is addressed, this is naked truth as the two major points states. Neemeka herbal clinic and research centre supports two major statements of PCORI in research especially in these chronic and multiple chronic diseases sometimes, it depends on the source virus that brought the disease in existing .Most of varieties of the diseases are the products of original diseases virus e.g. a disease like syphilis, same syphilis affects aorta to course heart disease, same syphilis brings about arteriosclerotic problems thus causing highly blood pressure ,same syphilis affects kidney causing cancer, same syphilis affects lung and liver, same syphilis cause diabetes, same syphilis affects brain causing cerebral malaria same syphilis cause epilepsy.Neemeka herbal clinic and research centre comments that anybody who is affected with syphilitic germs when not given proper treatments becomes easier to be affected with A.I.D.S.That patient who has syphilitic germs for a long period of time becomes difficult if not impossible to be treated once he or she is a victim of A.I.D.S virus.NB-In fact for any patient to be considered well, as a victim of A.I.D.S, must have syphilitic germs to speed up chemical reaction in patient’s body, thus making the patient very weak and suffer slowly taking a long period of years in order for he or she to die. Anybody who has syphilis A.I.D.S and has no any syphilitic germs becomes a carrier to affect others but he or she doesn’t die easily unless he or she is affected with syphilis.NB-Dr. Swartout states that the hope of a cure for cancer of liver is not even equal to the hope of a cure for cancer of lung, but if cancers in other parts of the body, which are usually sources of liver cancers are properly treated, most liver cancers would never develop. Page 863 in the book of New Modern Medical Counselor published 1943.Dr Swartout on page 807 second paragraph middle sentences, Dr. Swartout has given percentages 7-10% of all patients being cared in the hospitals are due to syphilis. Neemeka herbal clinic and research centre would like to support Dr. Swartout percentages even adding him more percentages up to 35% not 7-10% as he estimated.NB- Neemeka herbal clinic and research centre states that, if these inlet diseases, that are the sources of these complex, chronic and multiple chronic diseases are properly treated, then there will be no way of these chronic and multiple chronic diseases worldwide.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Friday, September 14, 2012

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

2. What methodological gaps should be addressed in future reports?

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

5. Please enter any additional comments or concerns here.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Author

Medical Device Manufacturers Association

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

5. Please enter any additional comments or concerns here.

MDMA supports the principles of evidence‐based medicine and comparative effectiveness research. We believe that patients and physicians should have access to the best information and data on which treatments work best in addition to which treatments are less effective. The availability of this information will be in the best interest of the patient and all stakeholders. As Congress and the Administration work to build upon the efforts on comparative effectiveness research, it is important to also examine other areas of the health care system including delivery system reform and prevention and wellness programs. The PCORI would be amiss if it were not to study all factors that are contributing to increased costs within the healthcare system, including the examination of the inherent root causes. For example, we are all sadly aware of the growing obesity epidemic among the nation’s adolescent population. Our children are living less‐healthier lifestyles than in generations past; they are less physically active and are consuming more unhealthy foods. If our country continues on this path, we can only begin to try to speculate what the related costs will be for treating obesity related conditions, such as diabetes, pulmonary hypertension and heart disease in general. The positive news is that this is a controllable condition, and by increasing education and awareness, we can make an impact. Tackling these issues early‐on will likely have a far greater impact on cost savings in the future than our current thinking on CER. To this end, efforts on comparative effectiveness research should not be focused exclusively on efforts such as technology assessment. Rather, focusing research efforts on wellness and prevention should be complimentary to the current thinking on CER. As the old adage goes, “an ounce of prevention is worth a pound of cure.” In this sense, a primary focus of CER and the PCORI should be to examine the root causes of increased health care spending. Comparative effectiveness research should also be used to address disparities in the health care system. Too often is the case that varying patient populations receive disproportionate health care treatment and coverage. PCORI has a tremendous opportunity to conduct studies and use the research to address these disparities to ensure that all Americans are receiving the appropriate care. Finally, the manner in which the PCORI and related agencies conducts its work and generates data must be as transparent as possible. As is apparent, the work produced by the PCORI will likely have a significant impact on numerous entities, including patients, payers and the industry. In keeping with President Obama’s pledge for transparency in government, it is absolutely critical that the work of the PCORI remain open and transparent for all stakeholders. This includes allowing stakeholders to submit public comment on the PCORI’S processes and methodologies for comparative studies as well as its results. To this end, it is important that expanded CER initiatives include a formal infrastructure to ensure public input on the work of the PCORI and related agencies. Furthermore, PCORI should not engage in any activities that restrict physician choice or patient access to therapies.

Author

Integrated Benefits Institute

Date

Friday, September 14, 2012

Zip Code

94105

Stakeholder Category

Researcher

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

See attached comments related to increasing focus on patient-centeredness in the process of the report develompent and use and report standards.