Public Comment Submissions

Author

Informed Medical Decisions Foundation

Date

Sunday, September 9, 2012

Zip Code

2108

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

They are consistent, but we have quite a few thoughts about how they could be strengthened

2. What methodological gaps should be addressed in future reports?

See attached memo. However, briefly, we would like to see: 1. Encouragement of focus groups and other strategies for getting patient input into what to measure 2. Value of routine measurement of patients' goals, concerns and preferences 3. Limits of studies using medical records unless they include input from patients 4. Better framing of issues related to internal and external validity, with particular emphasis on importance of external validity concerns 5. Addition of importance of standards for survey question design and good quality evaluation of questions in addition to psychometric evaluation 6. Emphasis on the importance of studies of long-term natural history of conditions, especially those that are defined by symptoms, like knee pain

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

We really think a much shorter, general user friendly document would be a lot more valuable to most potential users, with some of these chapters serving as appendices for technical specialists

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

I think just making it clear to anyone who gets instructions for how to apply for a grant that they should consult your methodological guidelines would get them to the people who need to see them.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-methods-comments.docx

Please provide your comments on Chapter 1 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 2 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 3 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 4 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 5 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 6 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 7 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 8 of the Draft Methodology Report.

See submitted overall comments

Please provide your comments on Chapter 9 of the Draft Methodology Report.

See submitted overall comments


Author

Astellas Pharma US

Date

Friday, September 7, 2012

Zip Code

20005

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Astellas-Comments-to-PCORI-Draft-Methodology-Report-FINAL.pdf


Author

Anonymous

Date

Thursday, September 6, 2012

Zip Code

10461

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Global Health/International Research and Global Collaboration/International DatabaseComplementary Medicine/Holistic Care

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

I would recommend the development of a summary brief that is more concise as the report is comprehensive but very extensive.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

I would recommend several venues:-E-mail blasts to all health care organizations, academic organizations, and government agencies-Social Media-Development of a website-Use of infomercials and teleconferences-Newsletters

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Comprehensive but too lenghty


Author

Anonymous

Date

Thursday, September 6, 2012

Zip Code

78664

Stakeholder Category

Caregiver

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Internal and External Validity controlsPopulation needsHIPPA challenges and solutionsTranslation- spread, diffusion and adoption

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Social Media, academic centers, hospitals, community centers, state/federal entities.


Author

Anonymous

Date

Thursday, September 6, 2012

Zip Code

34442

Stakeholder Category

Patient Advocate

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

I believe the work from AHRQ should be incorporated into the PECORI mission :Please see : http://effectivehealthcare.ahrq.gov/ehc/products/348/1001/MFRN-8_Framework-for-Considering-Study-Designs_DraftReport_20120308.pdf

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Please see question #2 suggestion above.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Final report should be distributed to major Healthcare organizations and Patient Advocate groups for comment.


Author

Pierre PLUYE

Date

Wednesday, September 5, 2012

Zip Code

H2W1S4

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Mixed methods research: For instance, page 18, replace "(quantitative and/or qualitative)" by "(quantitative and/or qualitative and/or mixed)".Regards,Pierre

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes


Author

David Thom

Date

Monday, September 3, 2012

Zip Code

94110

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

Please provide your comments on Chapter 3 of the Draft Methodology Report.

The table on page 25 "Standards for formulating Research Questions" seems unnecessary as (1) it does not clearly relate to the subject of this chapter, (2) all the points are made elsewhere in the report (pages 25 ad 59) and (3) The bullet points don't relate to the title of the table.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Page 64 Bullet point 7.25, i believe the word 'intervention' should be changed to 'exposure'


Author

National Committee to Preserve Social Security and Medicare (and a member of CUE)re

Date

Saturday, September 1, 2012

Zip Code

22044

Stakeholder Category

Patient Advocate

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/consumer-rep-comment.odt


Author

Richard Albert

Date

Thursday, August 30, 2012

Zip Code

80204-4507

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While I recognize that observational studies can generate clinically important information, the lack of randomization precludes them being used to assess the comparative effects of different treatments. The report states that lack of clinical equipoise may make randomization impossible. This is true, but if there is no clinical equipoise it will be equally if not more difficult to find a "control" group in an observational study against which to compare the results with the given intervention.The report states that inability to recruit sufficient patient numbers is an additional constraint making randomization impossible. And why might that be? Lack of clinical equipoise? (see above). Patients being unwilling to take the risks associated with a given intervention? (see above). Rare problem? (then why spend time and money addressing it when so much is needed with respect to common diseases - prioritization has to consider the scope of the problem).The "constraints" noted in the report, as well as many others proposed as providing a rationale for using observational studies to make clinical decisions regarding comparative effectiveness (e.g., outcomes are not patient-oriented, RCTs are too expensive or take too long), are really straw men. There are far too many examples of the results of observational studies being contradicted by RCTs. While a number of methods have been proposed by which patients in observational studies can be "matched", there are numerous papers indicating that none of the methods are effective.Basing clinical decisions on observational studies, no matter how well they are performed, is hazardous at best. Rather than spend money on numerous observational studies (that can obviously be done much more quickly and less expensively than RCTs) the money should be spent on studies that will be more definitive. In clinical research, you get what you pay for. What is saved by doing observational studies is very likely to be lost in recommending ineffective, and potentially dangerous, treatment (see hormonal replacement therapy to prevent vascular disease in women).Your Figure 6.3 begins with "Is baseline randomization indicated?" My response is that randomization is ALWAYS indicated for studies comparing the benefits or side-effects of an intervention. Observational studies can describe but only RCTs can compare!

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No


Date

Thursday, August 30, 2012

Zip Code

27705

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Specific guidance on the design and measurement issues of retrospective quasi-experiments and prospective quasi-experiments (e.g., inclusion/exclusion criteria, risk adjustment, identification of the treatment group, identification of the control group, outcomes identification) would be particularly helpful.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

The listing of standards (Appendix A) is tremendously helpful, and is already informing NIH and PCORI proposal development and grant reviews of colleagues at my institution.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

To increase awarness and application of standards in a broader context and for the general public, see if NY Times, WSJ or the Washington Post would interview one of the methodology committee members to discuss the report and its purpose.As a researcher who uses administrative data based on quasi-experimental designs, my greatest concern about the report was Figure 6.2 and the text in the pages around it. While the text box lists external validity as an intrinsic study characteristic, the figure places no value on external validity for informing study design. Decisions about study design are definitely informed by external validity considerations (see papers by Tunis and Gliklich in J of CER vol 1(3) for recent example). The figure should find a way of including external validity in a parallel figure or in this figure to reflect the importance that external validity will play. The report does state in several places that a somewhat biased but generalizable result may, in some circumstances, be preferable to an unbiased but non-generalizable result. See attachment for my own opinion about the relative ranking of internal and external validity of different experimental and quasi-experimental study designs. To more fully inform thinking about the randomization decision, the Yes/No text on page 53 should list under the Yes criteria that it is ethical to randomize, it is feasible to randomize, and that external validity is not as important as internal validity. Under the No criteria, please add that it is infeasible to randomize, it is unethical to randomize or external validity is a very high priority. Without more extensive discussion of external validity in these considerations, this figure could reinforce/harden the belief that randomization is the only way in which to have optimal internal validity.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/beststudydesignforCER.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Nice summary.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

As a researcher who uses administrative data based on quasi-experimental designs, my greatest concern about the report was Figure 6.2 and the text in the pages around it. While the text box lists external validity as an intrinsic study characteristic, the figure places no value on external validity for informing study design. Decisions about study design are definitely informed by external validity considerations (see papers by Tunis and Gliklich in J of CER vol 1(3) for recent example). The figure should find a way of including external validity in a parallel figure or in this figure to reflect the importance that external validity will play. The report does state in several places that a somewhat biased but generalizable result may, in some circumstances, be preferable to an unbiased but non-generalizable result. See attachment for my own opinion about the relative ranking of internal and external validity of different experimental and quasi-experimental study designs. To more fully inform thinking about the randomization decision, the Yes/No text on page 53 should list under the Yes criteria that it is ethical to randomize, it is feasible to randomize, and that external validity is not as important as internal validity. Under the No criteria, please add that it is infeasible to randomize, it is unethical to randomize or external validity is a very high priority. Without more extensive discussion of external validity in these considerations, this figure could reinforce/harden the belief that randomization is the only way in which to have optimal internal validity.


Author

Association of American Medical Colleges

Date

Monday, August 27, 2012

Zip Code

20037

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

See attached.

2. What methodological gaps should be addressed in future reports?

See attached.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

See attached.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

See attached.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AAMC.PCORI_.Methodology.pdf


Author

Lynn Etheredge

Date

Monday, August 27, 2012

Zip Code

20815

Stakeholder Category

Policymaker

5. Please enter any additional comments or concerns here.

There are many more opportunities for creating a high performance CER research system than discussed in the report.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Etheredge1.pdf


Author

Neemeka Herbal clinic and Research Centre

Date

Friday, August 24, 2012

Zip Code

50200

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

; Being transparent in developing policies to encourage public registration of all PCORI studies and the sharing of study, protocols, statistical code, and data, there is nothing than a person of any race that honors more than getting to know that he or she is dealing with people who are transparent and much devoted in their work the way PCORI is with their staff in running their work in all ways and means. In fact Neemeka herbal clinic and research centre honors nothing than being transparence in all that it does on patients and in research study, so that patients gets the really high quality herbal medicine findings needed especially on these chronic and multiple chronic diseases that can serve patients on large scale population worldwide.Forming a standing committee within PCORI to recommend appropriate methods for data sharing and to ensure that proper scientific credit is given to those sharing protocols, code data, this shows PCORI’s profile(layout)on how the organization operates is committed with their work by ensuring that proper scientific credit is given to those sharing protocols, code and data. Speeding up implementation of standards in funding announcements, peer, review and other internal process, PCORI staff developed templates for the preparation and review of proposals that incorporate the key elements of the standards; this makes the process of working to become very easy to operate or to run in all channels of work, to be keenly monitored, since some standards apply only to certain types of studies, a portfolio of templates applicable to various study designs should be developed.Support development and use of software for adaptive trials that can stimulate complex designs, broaden experience with adaptive trials for PCORI, perhaps through funding of a cohort of adaptive trials on priority topic areas. Mentor investigators and develop a ‘how to’ guide and a forum to share experiences with adaptive trials, develop a course work and training opportunities for statisticians and other methodologists interested in developing expertise in adaptive trials. Sponsor an institute of medicine committee to develop standards for research on medical tests.

2. What methodological gaps should be addressed in future reports?

Create an infrastructure to support research on patient engagement.-Develop a sample patient engagement plan to demonstrate the key elements required for patient engagement in the research process. -The sample plan should illustrate engagement of both patient informants and study participants.-Systematically collect information about patient engagement methods from PCORI sponsored studies.-Evaluate the effectiveness of patient informant engagement.-Synthesizes results across studies.-Disseminate findings to improve patient engagement in PCOR.-Support training in patient engagement methods for investigators and patient informants.-Improve the patient-reported outcomes( PRO) evidence base by supporting research on methods for assessing measurements properties ( based on qualitative and quantitative evaluations),score interpretability,meangfulness of score changes and strategies for minimizing and interpreting missing PRO data in PCOR.-Evaluate patient dissemination activities and require incorporation in future research of relevant learning from this evaluation.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

. Should ensure that a sufficient number and range of topics are considered, before topics for research findings are selected. Engagement of multiple stakeholders, especially patients, involving patients and other stakeholders in developing questions for systematic review, researchers can compare what people want and need to know with what is and is not known. As a result, systematic reviews can identify gaps in knowledge that underlie un certainty among patients and clinicians. Sometimes systematic reviews can generate new questions. For example, a pooled analysis of several studies can reveal an important finding that was not evident in the individual studies.Value of information analysis may be used to identify questions that have the greatest potential to improve population health by considering uncertainty in the health benefits and risks associated with alternative treatment choices, the ability of research findings to alter uncertainty, and the resulting care decisions. Reviews process identifies those proposals most likely to fulfill PCORI’s objective and agenda.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

• Neemeka herbal clinic and research centre the personnel investigator would like to upload full findings that are in the letter of support of Kenya Medical Research Institute (Centre for Traditional Medicine and Drug Research, P.O.BOX 54840-00200 NAIROBI, Kenya) Kemri and also a letter of support Kenyatta University office of the chairman Department of Health Sciences, Centre for Complementary Medicine and Biotechnology P.O Box 43844 Nairobi Kenya. Neemeka herbal clinic and research centre plus entire staff is in atonally research of herbs that are medicine for these chronic diseases. The centre is working round the clock in ensuring that they in research high quality findings required for these chronic diseases. In the letter of Kenya Medical Research Institute (Kemri) page2, second paragraph states that search of available literature revealed no published report on the toxicity of s.longepedunculata in humans despite the wide-spread use of the plant in various herbal remedies. Considering the potential toxicity of s.longepedunculata, herbal practitioners should be educated on this especially when they recommend this plant as part of complex regimen in the long term management of chronic illness.• Neemeka herbal clinic and research centre would like to appreciate and thank the entire staff of PCORI about the good work they are doing for patients especially those with chronic and multiple chronic diseases worldwide, working hand in hand in all means and ways in doing more research and being honest in submitting the really high quality findings needed for these chronic diseases that gathers large population worldwide. Neemeka herbal clinic and research centre also reflects thanks giving to anybody whose main objective is putting patients first by funding researchers in all means and ways as per research study is concerned, so that high quality findings is achieved for these chronic diseases.• Neemeka herbal clinic and research centre is doing more research on Diseases like;• -Heart diseases• -Diseases of liver• -Kidney problems• -A.I.D.S Virus• -Diabetes• -Lungs problems-

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/SECTION-A.docx

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Neemeka herbal clinic and research centre comments that it would be very much competitive if methodology Board of Committee Governors can look for a way of or any means of awarding those researchers and committee members who are much devoted with their work in researching the most high quality findings that are medicine needed for these chronic and multiple diseases that gathers large population worldwide. By so doing it will stiff the competition in researching for high quality findings, they can also be awarded with certificates. By raising a bar for researchers, it would be better if the bar of researchers goes by the most quality findings for chronic and multiple chronic diseases, with their awarding price.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Neemeka herbal clinic and research centre fully supports the methodology committee by developing the recommended standards, by forming a translation table, methodological standards for research and a proposal of activities for enforcing the methodological standards. These puts in peoples mind that methodology committee puts much and are very much committed in considering their in all ways as per every research findings is required in orderly manner thus making every findings gotten easily in its last stage, by fallowing proper channels , arrangements ready for patients.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Providing accurate studies for medical will ensure proper running of studies in research. Also designing in informing decision making, by indicating considerable doubts about the existence of the effect, this shows how keen, accurate and transparent in their research. The ways organization lays their work shows the smoothness and clearness how the organization is set up to operate their work.Experience in their fields leads to appreciation of the potential pitfalls in practice and in theory thus making methods to evolve and improve overtime. The way methodology committee developed the recommended standard in becoming more effective in ways of sponsors in research to articulate standards for the conducts of research they found .This shows how much willing the organization is set in assisting as much as they can in all ways and means in their research by struggling to get the really high quality findings so that they can assist in arresting these chronic and multiple chronic diseases worldwide CORI authorizing legislation which tasked the methodology committee in developing and promulgating such standards shows how keen and well organized PCORI is in the organization. These is true about PCORI by going a wider range of research studies used to inform clinical decision being governed by less formal approaches such as a peer review of research proposals and reports submitted to scientific journals.Neemeka herbal clinic and research centre plus its entire staff indeed appreciate the work the methodology committee Board of Governors are doing by ensuring that each and every work takes its course as research is cornered .

Please provide your comments on Chapter 4 of the Draft Methodology Report.

To produce information that is meaningful and useful to people when making specific health decision in research proposals and protocols is a very matter this shows how PCORI is much and more in minding and considering patients problems as their first priorities in dealing and struggling to get the right information plus the findings in their long run. Also engaging patients in formulating research questions, defining essential characteristics of study participants is a very wisdom approach as he who wears the shoes knows where the shoes pinches, especially those patients, with critical conditions. Neemeka herbal clinic and research centre, plus the entire staff are together with PCORI organization in working hand in hand in making sure and ensuring high quality findings in research is achieved, especially for these chronic and multiple chronic diseases in order to save their live worldwide. Neemeka herbal clinic and research centre is working and co-operating with other specialists doctors here in Kenya in doing more research in herbs for these chronic diseases so that the proper findings is gotten to save those who are victims worldwide.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Establishing a specific research agenda is a core duty of PCORI unless the mismatch between current research priorities and information needs of patients and clinicians is addressed.Neemeka herbal clinic and research centre fully supports a core duty of PCORI unless the mismatch between current research priorities and information needed of patients and clinicians is addressed, this is naked truth as the two major points states. Neemeka herbal clinic and research centre supports two major statements of PCORI in research especially in these chronic and multiple chronic diseases sometimes, it depends on the source virus that brought the disease in existing .Most of varieties of the diseases are the products of original diseases virus e.g. a disease like syphilis, same syphilis affects aorta to course heart disease, same syphilis brings about arteriosclerotic problems thus causing highly blood pressure ,same syphilis affects kidney causing cancer, same syphilis affects lung and liver, same syphilis cause diabetes, same syphilis affects brain causing cerebral malaria same syphilis cause epilepsy.Neemeka herbal clinic and research centre comments that anybody who is affected with syphilitic germs when not given proper treatments becomes easier to be affected with A.I.D.S.That patient who has syphilitic germs for a long period of time becomes difficult if not impossible to be treated once he or she is a victim of A.I.D.S virus.NB-In fact for any patient to be considered well, as a victim of A.I.D.S, must have syphilitic germs to speed up chemical reaction in patient’s body, thus making the patient very weak and suffer slowly taking a long period of years in order for he or she to die. Anybody who has syphilis A.I.D.S and has no any syphilitic germs becomes a carrier to affect others but he or she doesn’t die easily unless he or she is affected with syphilis.NB-Dr. Swartout states that the hope of a cure for cancer of liver is not even equal to the hope of a cure for cancer of liver is not even equal to the hope of a cure for cancer of lung, but if cancers in other parts of the body, which are usually sources of liver cancers are properly treated, most liver cancers would never develop. Page 863 in the book of New Modern Medical Counselor published 1943.Dr Swartout on page 807 second paragraph middle sentences, Dr. Swartout has given percentages 7-10% of all patients being cared in the hospitals are due to syphilis. Neemeka herbal clinic and research centre would like to support Dr. Swartout percentages even adding him more percentages up to 35% not 7-10% as he estimated.NB- Neemeka herbal clinic and research centre states that, if these inlet diseases, that are the sources of these complex, chronic and multiple chronic diseases are properly treated, then there will be no way of these chronic and multiple chronic diseases worldwide.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

These shows a clear guidance regarding the selection of appropriate research designs for specific research questions and also it avoids studies that are inappropriate for research question and could direct researchers to designs that reach the right answers sooner, thus improving the efficiency of research. Research designs clearly match the question patient s and their Health care a devisers, consider important, research results that are more readily accepted and implemented. A translation table helps to build a comprehensive research program, recognizing and balancing the inherent tradeoffs of each study design and analytical methodology developed asset of principles and a translation framework ,which refined by applying case studies contributed by committee members and members of the public. This shows how PCORI is devoted in designing their work to appear descent in research and proper arrangement so that it’s easily understood by everyone at the end stage of their high quality findings. Translation table makes information more informative and less complex. Translation Framework provides the theoretical underpinning and organizing structure for the translation table. It identify the range of appropriate research designs and analytic approaches to answer specific research questions also guides the user in making choices in study design and analysis methods based on current scientific knowledge in research categories and corresponding translation table.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

It identifies and assesses participant’s subgroups, selects appropriate interventions and comparators. It also assesses data source adequacy, specifying plans for data analysis that correspond to major aims, documents validated scales and tests. It uses sensitivity to analysis and to determine the impact of key assumptions, providing sufficient information in reports to allow assessment of the study internal and external validity. This is approval that PCORI is very much devoted in each and every work it does, by putting everything in transparent and in accurate standards in its final stages so that they are clearly understood in study internal and external validity to everybody.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Randomized trials can provide the strongest evidence about the comparative effectiveness of different treatments on intervention. Adaptive trials allow changes to be made to a study while it’s ongoing. Examples of adaptations include changing what proportion of patients are randomized to which intervention group ,altering the sample size, changing the eligibility criteria, dropping or adding comparison arms, changing endpoints and stopping early. Rather than waiting until the end of the study period to implement changes, the changes are planned for as part of the trial design and executed based on the analyses conducted during the trial. Neemeka herbal clinic and research centre supports randomized clinical trials indeed. Randomized clinical trials have potential to maintain many of the disadvantages of randomized clinical trials, while minimizing some of the disadvantages. Adaptive trials can sometimes provide results faster, and can also increase the relevance of trial results by adjusting both the composition of patient groups and treatments based on interim results and clinical questions. The flexibility and efficiency that are gained in adaptive trials have to be balanced with the risk that such trials typically require a longer design period and involve more logistical complexity. Also there are relatively few statisticians with expertise or experience in designing or carrying out such trial research.Neemeka herbal clinic and research centre supports randomized clinical trials, but should only be exercised by experienced professional experts in the field of research and treatment as this field carries weight in research study of the findings in its final stage and treatments. For example we have some samples of herbal medicines that can treat almost 50 different kinds of diseases depending on how experienced and professional you are in mixing this composition of herbs from different plants for you to get a high quality findings in your study of these complex diseases and in treatment of patients. Dosage will depend on how strong or weak the patient is, duration of the disease in the patient’s body, the kind of a disease you are dealing on, type of composition of herbs you have come up with, how strong are they in your randomized clinical trial research study? Also you need to put in mind, apart from treatment what other side effect these composition has in your randomized clinical trial, in your final stage of your study findings? If it shows side effect or if it has the side affect, what kind of composition in your randomized clinical study, you will use to end this side effect? What composition of herbs in your randomized clinical trial you need to use in your research study to solve this problem of side effect?Neemeka herbal clinic and research centre comments that Randomized Clinical Trials is the best system in research study, if it’s being operated by philosophic and pharmacological Doctors only as this field carries weight and its very technical to operate its research study and research findings.NB-Neemeka herbal clinic and research centre comments that, it’s through Randomized Clinic Trials in study of research herbal medicine of varies plants, that are medicine for these chronic and multiple chronic diseases that will be a solution for these diseases worldwide. Neemeka herbal clinic and research centre is not in dream to comment that the only solution for these chronic and multiple chronic diseases is alternative medicine.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Neemeka herbal clinic and research centre would like to comment that anything discussed publically with professionals in every step as research is concerned after submitting all the findings from different destinations, especially on these chronic and multiple chronic diseases or any other findings that can attribute to problems of patients that is how chain operates. Neemeka herbal clinic and research centre supports public discussion as two heads make light work. Let every professional air out his or her comment freely without any restriction. We learn from one another and learning has no end.Neemeka herbal clinic and research centre sees that in near future most of these chronic and multiple chronic diseases, PCORI nongovernmental organization, formed by Patient Centered Research Institute, when given a concrete foundation and support in all means and ways in running the organization, in more research of medicine that can arrest these problems of patients on large scale population, worldwide then these will be thanks giving and much appreciation for the good work they are doing for patients especially those patients who are affected with these chronic and multiple chronic disease worldwide, PCORI need to be remembered and be appreciated for good work you are doing for patients throughout of your life thanks for these indeed.When PCORI is given thorough support in submitting the really findings of high quality, that can deal with these chronic and multiple chronic diseases worldwide, then these problems of chronic and multiple chronic diseases, will come be on small scale worldwide.GENERAL COMMENT.Neemeka herbal clinic and research centre sees that it would be in total ease if PCORI Board of Governors committee puts into consideration the institute to have its indigenous herbal medicine plantation trees that are herbs for these chronic and multiple chronic diseases, to ease the findings to be gotten at the central place. Neemeka herbal clinic and research centre is ready to look for suitable place to plant these trees that are herbs for these chronic and multiple chronic diseases. Neemeka Herbal Clinic and Research Centre. Thanking you all.


Author

Wiley Jenkins

Date

Friday, August 24, 2012

Zip Code

62794

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Methodologies to conduct and report patient-centered systematic reviews.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Overall – This Report is quite long and a lot to wade through. I suspect many researchers will be forced to skim sections looking for important parts. I would suggest creating an executive summary-type version with lots of bullet points and hot-links to more detailed info. Thus, an individual can bypass the parts of chapters of less immediate interest for writing a proposal, but click for more info on specific scenarios and examples.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Chapter 1 – 1st paragraph under PCOR – “No matter how we make these decisions, we all agree on one thing: we want to rely upon the best possible information.” I wonder if it would be worthwhile to mention explicitly, and include as a research component, one of the most pressing issues facing a patient: patient cost. This is a driving decision factor for many individuals and recognizing this and placing it as a factor in research would serve patient decision making.Chapter 1 – page 6 1st paragraph – “Sometimes investigators are less inclined to publish…” I suggest that PCORI establish an open registry of all PCOR studies (similar for clinical trials). I also suggest that PCORI establish a listing of all patient-centered study results which can inform decision making (e.g. as USPSTF has a listing of preventive measures). There needs to be a respected place where PCOR data can be accessed.Chapter 1 – page 8 3rd paragraph – “…1) an Institute of Medicine study to develop standards for conducting systematic reviews…” I think it would be good for PCORI to both establish systematic review criteria (a la PRISMA) and specifically fund investigators who wish to conduct such reviews. Furthermore, is there any reason why PRISMA is not actually listed anywhere in the Report when other standards such as CONSORT, STARD and STROBE are?

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Chapter 3 – Standards for formulating research questions – I think it would be of tremendous benefit to researchers if PCORI described specifically how patient involvement in research development (and the entire process) might be best described in grant proposals. Grants I’ve reviewed had a difficult time showing patient involvement, either because there was little or it was poorly described.Chapter 3 – page 22 3rd paragraph – “reporting biases, particularly publication bias…” Has PCORI considered establishing a journal specific for PCOR? Something along the lines of the CDC and Public Health Reports. This would then present to the public and policy makers a one-stop credible source for PCOR.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Chapter 6 - page 45 #3 – This discusses how different decision-makers have different evidentiary needs. Since PCORI was born of politics and seeks to inform policy (in part) I think the PCORI should consider developing a resource of executive summary-like PCOR studies, results, implications and caveats. Presenting a credible and lay version of complex study results and policy implications will be useful for advocates, policy makers, show PCOR results and progress, and highlight continuing gaps.


Author

Ilana Mittman

Date

Thursday, August 23, 2012

Zip Code

21201

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Health outcomes are impacted greatly by where people live, and what they are exposed to. Social determinants of health such as housing, employment, and education must be looked at and incorporated into the research methodology, so that these variables can be used against expected outcomes.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

This comment pertains to the definition of “stakeholders.” These are currently narrowly referred to in the draft methodology report as clinicians, the healthcare system, and patients and their caregivers. The report also urges researchers to “ensure that PCOR results accurately and effectively inform health decisions important to patients.” I maintain that in order to most broadly and effectively disseminate and translate PCOR results we include whole communities as stakeholders; faith-based, educational institutions, community leaders, etc. Patients’ decisions are often informed not only by immediate caretakers, but by their entire cultural milieu which shapes their lived experience. Engaging communities from the start of research, through its design, implementation, evaluation and dissemination (CBPR) is a proved best practice to engage the public. Finally, I suggest that PCORI includes the term of the tern “community-centered” to its mission statement.


Author

University of Rochester

Date

Monday, August 13, 2012

Zip Code

14620

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Community engagement

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

In a general sense. The central challenge is that methods will vary enormously depending on study question. While establishing guidelines for the field is likely to prove beneficial over the long run, it could make for a more onerous review by study section members - at least until they have committed criteria to memory.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

My first question is whether applicants will be expected to explicitly reference the methods report including citing each section number in their application? Given that the report may not be finalized until November, will applicant during the November cycle be expected to adhere to these criteria?


Author

Leif Solberg

Date

Sunday, August 12, 2012

Zip Code

55419

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

No

1 Comments

As stated in the PPACA, "The purpose of the Institute is to assist patients, cliniicians, purchasers, and policy makers in making informed health decision by advancing the quality and relevance of evidence,” yet one searches in vain for evidence that the standards are meant to accomodate anything but a patient perspective. I realize this report was meant to focus on patients, but this seems to be the one that will identify standards for research, so how will they be expanded to reflect a broader group of stakeholders?

2. What methodological gaps should be addressed in future reports?

Valuing and obtaining the views of clinicians and care systems about which knowledge gaps are important to them and building research in a way that incorporates their perspective

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

Not if it continues to present these guidelines as standards (see attachment)

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

see attachment

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments-on-PCORI-Draft-Methodology-Report.doc


Author

MTPPI

Date

Tuesday, July 31, 2012

Zip Code

20814

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

1) In the ‘causal inference’ section of the report, MSM models are not mentioned and should be in addition to propensity scores instrumental variable models. 2) Conducting a per protocol analysis (versus an intent to treat) is not mentioned although it is applicable to observational data.3) Interrupted time series is also not mentioned -- nor is any kind of pre/post analysis – although these are also considered to be ‘causal’ models.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No


Author

Tim Carey

Date

Saturday, July 28, 2012

Zip Code

27599

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

The document is an impressive start and overall helpful to investigators.

2. What methodological gaps should be addressed in future reports?

1. Cluster randomized trials. 2. Research prioritization issues3. How to conduct stakeholder engagement in a manner that enhances research validity and applicability as well as enhancing the research process.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Should be a consideration but not an absolute issue. That is, I would not recommend that reviewers automatically triage a proposal if an applicant does not follow a methods committee guidance, and has a better idea. Similarly, there are currently many research issues that are not present in the methods report, and we similarly should not give an applicant a 'pass' if they don't take a methods issue into account that has not been addressed yet by the methods committee.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Consider publication as a journal suppement to make sure it gets in pubmed. However, keeping track of versions of the methods manual will be key, since it is quite appropriately meant to be a living, changing document.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Chapter 1: Quite well done, nicely summarizes some of the issues that led to the need for PCORI.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Chapter 2: The Committee does take a conventional view of ‘standard’ as a “process action or procedure for PCOR that is deemed essential to producing scientifically valid….results.” The term ‘minimum requirement’ is also used. At the same time, the RFA’s from PCORI seem to indicate that the standards, while they should generally be followed by applicants and implemented in PCORI sponsored research, are not absolute. That is, if an investigator has a better approach and can justify it to the satisfaction of peers, that would be acceptable. For example, 10 years ago propensity scores and IV were little used in PCOR, we need to be open to innovation. I agree with the RFA statements, but they are not completely consistent with the methodology report. Clarification will need to take place over time as to whether these standards are a ‘must’ or ‘strongly consider’.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Chapter 3: Nice ‘phases of PCOR’ on page 17, good reformulation of standard study design jargon. Page 19: The IOM standards are generally, but not completely, congruent with the current methods manual used by AHRQ as part of the Evidence-based Practice Center (EPC) program. I am an EPC co-director, and any differences between the IOM and the EPC manual are minimal. Issues of stakeholder engagement in systematic review, topic refinement, and research needs prioritization were not prominent in the IOM report, although they were mentioned. AHRQ has been increasing emphasis in these areas. Standards for systematic review should be the same between AHRQ and PCORI to avoid confusion. Page 22: 3.1.4. A topic for further discussion is the selection of comparator interventions in PCOR/CER. Often, we hear ‘usual care’ as a comparator. While such as approach is pragmatic in that it reflects care as provided in community practice, we know from the small area variation literature that ‘usual care’ varies a great deal from area to area of the US, and even within states. Recommend that the methodology committee address this issue in the future, not through setting a standard, but rather by recommending how we should document active comparators so that the applicability of research results can be assessed by providers and the public. Page 23: Agree with additional development and discussion of ‘data sharing’ in future methods work by PCORI and the committee. This is a complex area, with some risk of publication of results that are premature, as well the need for meticulous documentation of databases prior to posting, which takes both time and resources. Secondary analyses of data can be misleading if the new investigator doesn’t understand where the data came from and its structure, strengths and weaknesses. PCORI may wish to develop additional standards as well as technical support to assist investigators with these tasks. There are real costs involved.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Chapter 4: Very helpful chapter. The research gaps section is useful as well, in that it recognizes that, while we view patient engagement as a research and social good, we are not sure how good it is or quite how to do it. While patient engagement seems to enhance the research process, is the research itself more valid, more efficient, or more applicable to the population at risk? Reviews of the CBPR literature have focused a great deal on process issues, and not enough on the outcome of the research. Was the knowledge gained useful? Research recommendations on page 30 are well thought out.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Chapter 5: No specific comments.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Chapter 6: Very complicated chapter, I recognize that the ‘translation table’ concept was part of the ACA and an initial very challenging mandate for PCORI. Agree with viewing the ‘translation table’ as a process and not a strict algorithm whereby one gets from A to B to Z for any research question according to some sort of branched logic. I think there needs to be recognition that terminology varies across disciplines involved in PCOR. For example, epidemiologists and health economists will use different terms for the same study design. Hartling and colleagues in the AHRQ EPC program tried to come up with standard study design descriptors for systematic reviews, but found that the kappa statistic agreement between even experienced investigators was only fair to poor. The approach outlined by the methodology committee seems to be fairly prescriptive, but might be best described as considerations for investigators as they move from research question to study design. A paper in the EPC program (I was an author) used the term ‘study design consideration’ to describe this process, to avoid being overly prescriptive. We felt that study validity was critical of course, but that issues such as ability to recruit patients, resource constraints, ethical issues, and urgency of need for an answer to the question were also important. I recognize that the text presented here is just getting this process started, but I would be concerned if investigators viewed the translation table as a mandate, rather than a very helpful guide.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Chapter 7: Also very helpful. Quite focused on propensity scores and IV, many other methods are useful though. We are still learning a lot about the appropriate use of these methods for non-experimental analyses. The section of missing data reads a bit like a chapter from a biostatistics text. The need for multiple imputation, to my understanding, depends in part on the frequency of the missing-ness. Overall, though, a thoughtful discussion. The discussion of the need for data security etc as a part of research protocols is indeed welcome!

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Chapter 8: I am not expert in Bayesian analysis, I know there is some controversy regarding early stopping rules in adaptive trials. Future work should specifically address multiple looks at the data and early stopping rules. There is certainly not consensus at present in the research community. Agree with the text early in the report that cluster trials should be a priority for the methods committee. My view is more important than Bayesian trials. POCRI will be sponsoring a number of health systems and health disparities interventions, and treatment assignment is generally not at the level of the individual in those studies. I think the community could use some guidance, especially in situations where the number of units of interventions is modest ( hundreds of patients but only 12 nursing homes etc). CTSA consortium might be a useful resource, since pragmatic trials are a priority for that group. Page 83: Data registry text is quite useful, this is a rapidly evolving area.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

No specific comments, nice summary of next steps.


Author

Anonymous

Date

Friday, July 27, 2012

1 Comments

I find if very funny that you want to improve the decision making process and to do this you produce a massive 200+ page document. Synthetic capabilities are a bit lacking.I thought that the purpose of PCORI was to generate results/documents that can be easily (presumably also quickly)understood by all and read by all. May be starting with this first document would have been a good example. I will read it (try to...) and provide more comments.JUST VERY FUNNY!!!

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Shorten

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Shorten


Author

Michael Power

Date

Thursday, July 26, 2012

Zip Code

NE1 2ES

Stakeholder Category

Patient

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

I have two suggestions. Please accept my apologies if I have missed their coverage in your rather large document.1 Patient preferences for alternative interventionsThe standards should include a statement that randomized controlled trials of interventions should include data on the patient's pre-randomiztion preferences for the various alternatives in the trial. Preferences can have strong placebo effects (positive if preferences are met, and negative if preferences are not met).The data should be analysed to determine the size of the preference effects, both absolute, and relative to the specific effect of the intervention(s).2. Patient's goals for treatmentTwo patients with the same condition may have different goals for treatment - for example, one person with chronic sciatica would prioritise being pain-free; anther patient might prioritize being able to resume work and recreational activities. Study designs should capture data on this that would allow "treatment success" rates to be measured according to the patient's goals for treatment-------------I make these comments from the perspectives of a physician, methodologist, and patient.


Author

Annals of Internal Medicine

Date

Thursday, July 26, 2012

Zip Code

19106

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Please see attached comments.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

See attached document.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-comments-6.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

see attached document


Date

Thursday, July 26, 2012

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

I have two suggestions. Please accept my apologies if I have missed their coverage in your rather large document.1 Patient preferences for alternative interventionsThe standards should include a statement that randomized controlled trials of interventions should include data on the patient's pre-randomiztion preferences for the various alternatives in the trial. Preferences can have strong placebo effects (positive if preferences are met, and negative if preferences are not met).The data should be analysed to determine the size of the preference effects, both absolute, and relative to the specific effect of the intervention(s).2. Patient's goals for treatmentTwo patients with the same condition may have different goals for treatment - for example, one person with chronic sciatica would prioritise being pain-free; anther patient might prioritize being able to resume work and recreational activities. Study designs should capture data on this that would allow "treatment success" rates to be measured according to the patient's goals for treatment-------------I make these comments from the perspectives of a physician, methodologist, and patient.


Author

Anonymous

Date

Wednesday, July 25, 2012

Zip Code

19102

Stakeholder Category

Clinician

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Special training for nurse researchers who may not have been exposed to this in their PhD programs. Few nurses are trained in patient-centered health outcomes research.


Author

phrst.org

Date

Monday, July 23, 2012

Zip Code

63105

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

Personal Health Records (PHRs) concept should be part of your “Patient-centered Outcomes Research”. As your report says, “patient-reported outcomes is essential to patient-centeredness”.Furthermore, an “anonymous PHRs” or “confidential PHRs” will give a patient more privacy and help more people to adopt a PHRs program sooner. An anonymous PHRs system is able to collect more unbiased patient report and help government obtain certain information that will be hard to collect otherwise.As the inventor of “anonymous PHRs” concept, we can provide more research result that is based on our pioneer anonymous PHRs online service.


Author

Prof. F. Chiappelli

Date

Monday, July 23, 2012

Zip Code

90095

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes - but incomplete Three critical and timely issues of PCOR methodology that are understated or altogether under-discussed in the Report are outlined in (2) below

2. What methodological gaps should be addressed in future reports?

The reports requires a more substantial depth in the domains of:A. development, standardization & validation of instruments for the assessment of "quality" of the evidence ("how well the research was done) (beyond the "level" of the evidence, which simply tell "what research was done") - that is, and specifically content & criterion validity, inter-rater & internal consistency reliability, coefficient of agreement - such as, for example, Kung et al. From Systematic Reviews to Clinical Recommendations for Evidence-Based Health Care: Validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for Grading of Clinical Relevance. The Open Dentistry Journal 2010, 4:84-91. Phi et al. Expanding the Grading of Recommendations Assessment, Development, and Evaluation (Ex-GRADE) for Evidence-Based Clinical Recommendations: Validation Study. The Open Dentistry Journal 2012 6:31-40. Chiappelli et al. Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT. Bioinformation, 2012 8: 691-4.B. development, standardization & validation of instruments for the assessment of clinical relevance, such as, for example, Phi et al. Expanding the Grading of Recommendations Assessment, Development, and Evaluation (Ex-GRADE) for Evidence-Based Clinical Recommendations: Validation Study. The Open Dentistry Journal 2012 6:31-40. Dousti et al. Evidence-Based Clinical Significance in Health Care: Toward an Inferential Analysis of Clinical Relevance. Dental Hypotheses, 2011 2:165-77.C. Improved means of dissemination of the best available evidence to the stakeholders, and development, standardization & validation of instruments for the assessment of efficiency of dissemination (i.e., increased health literacy, etc.), such as for example, Barkhordarian et al. Dissemination of Evidence-Based Standards of Care. Bioinformation, 2011 7:315-9. Barkhordarian et al. Disseminating the Best Available Evidence: New Challenges in Public Reporting of Health Care. Bioinformation, 2012 8:293-5. Chiappelli et al. Reliability of Quality Assessments in Research Synthesis: Securing the Highest Quality Bioinformation for HIT. Bioinformation, 2012 8: 691-4.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

The report should be useful for grant review, but as is now it misses important targets of PCOR by understating the following:Point (A) in (2) stresses the timely and critical need to validate by means of stringent psychometric means standard tools for measuring the quality of the evidence. This is an essential point in our PCOR science, since it is based on obtaining the "best" available evidence. Thus, the point above is critical and must be addressed in the Report, lest more and more PCOR be produced that yields unreliable assessments of the best available PCOR evidence.Further, point (B) in (2) above stresses the urgent need to develop, standardize and validate tools to asses the complex construct of "clinical relevance". Short of that, our PCOR science is all but an academic exercise. If indeed we seek to do patient-centered outcomes research, then we must have valid and reliable tools for evaluating the clinical relevance of the identified best available evidence. Lastly, the very purpose and goal of PCOR is to disseminate the best available evidence - but not simply for the sake of dissemination, as it is for the sake of serving the stakeholders - particularly the patients and the caregivers - in increasing health literacy. The fundamental aim of PCOR is to raise the awareness and consciousness of the patients, so that they can become active participants in the clinical decision-making process.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

cf., Barkhordarian et al. Dissemination of Evidence-Based Standards of Care. Bioinformation, 2011 7:315-9; and Barkhordarian et al. Disseminating the Best Available Evidence: New Challenges in Public Reporting of Health Care. Bioinformation, 2012 8:293-5. (btw: I do believe that question 4 has a typo: Do you have any ideas about the PCORI might distribute... should read Do you have any ideas about how the PCORI might distribute...)


Date

Monday, July 23, 2012

Zip Code

19104

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

THis is a very comprehensive document - however it is a bit lon. 200+ pages with links to other docuemnts within the text is a distractor for those that are trying to balance clinicala nd research duties.

2. What methodological gaps should be addressed in future reports?

Suggestions on how to get patient feeback during that development phase. Those that do not have a pteint centered focus will have difficuty with this. Perhaps providing grant money for patient/disease specific organizations to develop electronic patient forums and/or resigtries of individuals willing to take part in research and/or answer surveys about research will be helpful to those wanting to onduct approved research.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

It would be nice to see a list of the patient organizations that were consulted for development of this aptient-centered document. Also the list of individuals on the panel si lacking. There is no one that lists expertise in pediatrics, critical care medicine, neonatology, pulmonary medicine, anesthesia medicine or respiratory care. These are areas that have a meavy need for appropriate research. Especially the pediatric population because is has the least amount of data availabel to make educated decisions about care.


Author

Human Services Research Institute

Date

Monday, July 23, 2012

Zip Code

2140

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

This is an extremely comprehensive and thoughtful report that has both a values base (asserting the importance of outcomes from the individual's perspective) and a sound methodology that takes into account a wide range of issues and emerging/progressive methologies. The only criticism I would make -- and I know that this work is intended to be at the meta level -- is that I would like to see more commentary on the specific issues affecting people with disabilities -- physical, cognitive, developmental, and behavioral. Each sub-groups poses real issues in terms of participation in research designs as well as in responding to outcome inquiries. People with behavioral/mental health issues are particularly challenging yet their health outcomes are astoundingly bad --- their life expectancy is approximately 20 years less than the general population. Issues of physical access (addressed in the AHCA) are important to people with a range of disabilities. With respect to people with cognitive disabilities, issues surrounding communication are critical.

2. What methodological gaps should be addressed in future reports?

While there are no obvious gaps, I would to see more attention to the methological challenges faced by researchers conducting participatory research with people with a range of disabilities as well as issues involved in data collection among such individuals.Future studies should also take into consideration the places where people live and receive services -- there are increasing numbers of individuals in LTC settings such as group homes, ICF/MRs, nursing homes, etc. who receive their acute care through these faciities/homes.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Please make sure that these results are communicated to organizations associated with the LTC populations including people with disabilities. This would include primary advocacy groups as well as state policy makers. Given the dual eligibles demonstrations that in some instances combine LTC and acute managed care, there is also a need to circulate this information to managed care companies and other non-profits involved in providing integrated care.


Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Industry

Please provide your comments on Chapter 1 of the Draft Methodology Report.

1. The report repeatedly talks about PCORI stakeholders, without ever including medical product manufacturers in the definition of “stakeholder”. Given the importance of PCOR to the life science industry, the industry’s role in producing evidence, and the inclusion of industry on the PCORI governing board, we recommend that the report be revised to explicitly include manufacturers in the definition of PCORI stakeholders.2. Box, page 5: The story nicely highlights confounding by indication (i.e., give new drug to those who failed other treatments). This is relevant to use of observational studies mentioned later and might be worth repeating in that context.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

1. On p. 11 the report states “The Methodology committee and PCORI Board and staff are developing a coordinated approach to promote the uptake of PCORI methods standards. This includes engaging all stakeholders who might use the standards, creating reporting and surveillance opportunities, and developing enforcement functions over time.” The Report should make clear that these activities apply only to PCORI and to PCORI related work done by those outside of PCORI who are contracted by PCORI to do work/research for PCORI; we do not believe it is PCORI’s intent to “enforce” its standards in all research.2. We suggest that PCORI consider developing a scheme/template to evaluate and rank the quality and results of a study when it is completed. This would provide a clearer guidance to the consumer/user of the research results, something analogous to the “Consumer Report”.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

1. Page 22: “Specifically, the Methodology Committee recommends that PCORI develop policies to encourage public registration of all PCORI studies and the sharing of study protocols, statistical code, and data.”We would strongly encourage the use of existing infrastructure, if possible. Could clinicaltrials.gov be adapted? Creating a parallel universe could introduce confusion. 2. Also page 22, the report says: “An important next step for PCORI is to develop policies that can remove or overcome this obstacle, not only in its own research, but throughout the broader clinical research community.”Don’t such policies already exist (e.g., requirements to register protocols and disclose results)? Compliance seems to be the issue. NIH requires posting of data generated by publicly-funded studies, but compliance is not good. FDAAA requires posting of certain trial results, but compliance is not 100% (as has been documented in publications.) In any case, for PCORI, perhaps the focus should be on requiring that protocols be posted, and requiring future disclosure of results, as a condition of funding.3. The report should discuss how patients’ views and preferences sought as part of the process of determining the standards in this draft. It would be helpful to understand how patient input was solicited and considered for the present draft report and will be integrated in the respective phases of PCOR (e.g., Table 3.1).

Please provide your comments on Chapter 4 of the Draft Methodology Report.

1. Page 30: With respect to one of the recommended actions: “Improve the patient-reported outcomes (PRO) evidence base by supporting research on methods for assessing measurement properties (based on qualitative and quantitative evaluations), score interpretability, meaningfulness of score changes, and strategies for minimizing and interpreting missing PRO data in PCOR.”This is an area where PCORI might want to engage directly with FDA, on some level. The draft guidance on PROs from FDA is aimed at what’s required from a regulatory perspective. If the future research can inform the regulatory science, and we can work toward agreement on specific instruments in specific settings, that will bring clarity and efficiency to the development and use of PRO tools.2. The draft report mentions (p. 24) “Many other individuals are involved in health decisions, including clinicians, healthcare system administrators, payers, regulators, and policy makers. In this report we focus on patients rather than on other health decision stakeholders, who will be the focus of future standards.” It is unclear if described future standards will remain patient focused or will take the perspective of the respective stakeholder and how future standards will interface with existing standards (e.g., FDA)?

Please provide your comments on Chapter 5 of the Draft Methodology Report.

1. Page 37: Text refers to background papers on Value of Information analysis that were commissioned by the Committee. Are those papers available publicly? If so, they should be made available, somehow, perhaps on the PCORI website. If they are already available, the report should say something specific about where to find them. The report also mentions training on VOI—the final report should specify whether such training would be made available to the public.2. On p. 32 the Report lists “effect on national expenditures” as a research prioritization factor that the legislation specifically states should be considered; on the other hand the report states that elsewhere the legislation restricts use of cost. The report states that cost can be factor “in the effectiveness of care if it influences choices made by patients and clinicians”. Our interpretation is that the report is not suggesting cost-effectiveness of treatment as a standard (e.g., incremental cost-effectiveness ratios) but rather is suggesting adjustment for cost as a confounding variable in the assessment of treatment effects. The authors should add clarifying language to this effect.3. The report does not give standards for systematic reviews and states that the reason for this is that the Institute of Medicine (IOM) has published such standards. However, these IOM standards apply only to medical and surgical interventions and not to delivery systems. The report should recognize this and commit to further research on standards for systematic reviews of interventions other than medical or surgical interventions.4. To optimize resource use and avoid duplicative studies or reviews, it would be helpful to understand how PCORI will interact with other entities who fund research or have interests in establishing research standards (e.g., AHRQ, FDA, NIH, CMS, professional societies, payer organizations, manufacturers). How will PCORI avoid duplication with ongoing public (e.g., NIH, professional society) or private (e.g., manufacturer) research that might not be captured in a systematic review or elsewhere? PCOR could potentially be integrated with other research, thereby, supporting simultaneous insights to PCOR and complementary research questions.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

1. Page 43: The definition of randomized trial could be better organized for this document. Right now it plays into the stereotype of RCTs being done in narrow populations under “optimal” conditions, then only as a seeming afterthought, mentions pragmatic trials. The necessary material is all there, but we would suggest starting with the general principle that randomization balances the groups with respect to known and unknown characteristics at baseline. Then make the distinction between traditional “efficacy” trials, and pragmatic trials. You might want to reference the PRECIS paper, in which a number of dimensions of study design are described, each of which could be more toward the pragmatic or more toward the efficacy / traditional / narrow, etc., trial. Additionally, there are some places where language could be modified. For example, text on page 45 states, “A randomized trial may be required to establish the benefits or harms of the new procedure compared to the standard procedure under these ideal conditions. Regulators are likely to be very interested in the outcome of this type of design. An observational study may also be needed to determine the safety and effectiveness of the new procedure compared to the standard approaches when the procedure becomes more widely available. Patients and professional societies may be interested in the outcome of this study—patients to determine if the new procedure is a good choice for “people like me” and professional societies to determine if and how best to adopt the new procedure.”This seems like the perfect place to mention either pragmatic trials (evaluating the two surgical approaches in a broader range of patients and settings), and cluster-randomized trials (randomizing centers or surgeons to one or the other of the surgical approaches). Then there may still be a need for an observational study to address further aspects, e.g., which patients are getting referred for which procedure, and why.2. Page 47: This is a bit of a semantic point, but we suggest avoiding using the term “phases” (especially since you have 4 of them), so as to avoid confusion with the use of the word in clinical development programs (phase 1studies through phase 4 studies). Stages? 3. Figure 6.2: In deciding on a randomized trial or not, we are not sure what the intent of the list is. Are these “considerations” that would point toward or away from a randomized study? “Precision” is a consideration in any kind of study (if it’s meant in the statistical sense of reduced variance). If the idea is to present a list of conditions under which you would want to randomize, then this list needs some rewording. Figure 6.3 gets at this much more clearly. Why not just refer to Figure 6.3 here? Or draw specific elements out of 6.3? 4. Figure 6.4: It is not clear what is meant by “little exposure time trend” in this context. The figure is out of context and we don’t see where it is referenced in the text. The paragraph that immediately precedes the figure is talking about randomized trials, in which case the list doesn’t fit. (In fact, the paragraph says that the investigator has decided that it’s not possible to control baseline confounding, and then the text shifts to a discussion of non-randomized designs. As an aside, it is not clear why an exposure time trend would matter. Presumably, calendar time would be a potential confounder that could be controlled through statistical adjustment. A more important issue may be whether there is within-person confounding. Is the severity of disease changing over time? If so, then the changing exposure status could be in response to changing disease severity, and disease severity might not be well-captured in some contexts. 5. Figure 6.6: It’s a small point on the figure, but a larger point in general, but the advice given here is that a secondary health care database could be useful in situations when previously-collected data are sufficient for the proposed study. The topic of existing databases is also mentioned in the response to the RFI in the appendices. It’s probably obvious, but may need some emphasis, that many benefit questions will NOT be addressable through claims or even EMR data. The systems are set up to capture the experience of patients with medical problems. Symptoms and functional status are generally not captured unless there’s an associated bill. If you want to study pain relief (or depression, for example), you need indirect measures like changes in prescriptions. If you want to study success of schizophrenia treatments, you have to look at hospitalization (since nobody records symptoms). Again, these points probably need some emphasis in the document, perhaps in the context of encouraging research on how such measures MIGHT be incorporated into the existing data collection infrastructure.6. The schematic on p. 52 implies that all observational studies are based either on primary data or on secondary data and does consider studies which are based on both primary and secondary data. It will be important to recognize this in the report, especially since this diagram might be taken by many to describe all possibilities.7. Given the importance of patient reported outcomes (PRO’s) for PCOR we recommend that this initial report include standards for assessment and analysis of PRO’s.8. Although the report specifically mentions health delivery system as one of the types of interventions of interest (p. 3, “assess the benefits and harms of preventive, diagnostic, therapeutic, palliative, or health delivery system interventions”), in general there is a lack of examples in the report of health services research; the report could be enhanced by more examples of such interventions, especially given the potential benefit of such research.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

1. Page 64, standards: 7.2.4 says “In general, variables for use in confounding adjustment (either in the design or analysis) should be ascertained and measured prior to the first exposure to the therapy (or therapies) under study.” There are situations in which you can adjust for time-varying confounders, e.g., by recalculating the propensity score at later time points. The report is also not getting into other methods, e.g., G-estimation, that are more flexible with regard to time-varying variables. Maybe the committee can modify the standard slightly to accommodate this, or at least say something in the following rationale section to make the qualifications. 2. Page 64, standard 7.2.5: It’s not JUST a matter of assessing balance on propensity scores, but also assessing the extent of overlap. If only 20% of the exposed group can be matched with an unexposed person with a similar propensity score, the groups can be balanced, but the lack of overlap is an important bit of information to describe. 3. Page 72, standard 7.4.4: we suggest the committee add a specific point about recording reasons for discontinuation. In many trials, a common reason for discontinuation is “subject choice.” A variant on this is “withdrew permission.” These are not helpful comments, as it’s not clear if the issue is really an underlying lack of efficacy, an adverse event, the subject has been “cured,” or some unrelated issue (e.g., subject moved to another city). This committee really should tell people to be more explicit about reasons than “subject choice.” 4. Page 77: Standards for Data Networks: You separate the point on standard terminology (7.5.5) from the point on the need for a common data model (7.5.7). The connection between these needs to be made very explicit. It’s not just the structure of a common data model, but the fact that the data elements are standardized that makes the CDM useful. 5. Also on data networks: A point you may want to make explicit, is the need for complete data, in the sense of capturing all interactions between a patient and the healthcare system. It’s a problem if the patient receives some, but not all, care within a specific network, and we’re unable to make the link between care inside and outside the network. Do networks work better if they are somewhat geographically isolated? 6. Also on data networks: It would be great to have an example or two of existing data networks (e.g., Regestreif may meet the definition of such a network). How and why are they successful? How do they get “buy-in” from participating clinicians and institutions? Why not propose actions aimed at encouraging the formation of data networks? What about research aimed at how to motivate / incent physicians (and patients) to participate?

Please provide your comments on Chapter 8 of the Draft Methodology Report.

1. Adaptive Designs: In interactions with FDA, concerns sometimes arise about the operational aspects of implementation. The worry is that somehow bias will be introduced in the conduct of the trial as a result of the adaptation. One concern is that participating investigators will try to second guess what the results are based on the adaptation, and will somehow modify who gets entered into the study, or how patients get treated. Perhaps a research agenda could include some investigation about implementation issues. (You mention the points about the infrastructure needing to be present to make sure data entry is always up to date, data cleaning is done very promptly, etc., so that the analyses necessary to inform the adaptation can be completed in a timely manner (i.e., before enrollment is completed under the original design.))2. Page 82: Could some detail be provided on the examples of studies using adaptive designs? It would be nice to illustrate the points. Or are the details included in a separate document? 3. Page 85: You make the point that registries do not typically have control groups. This statement needs supporting evidence. Many registries are disease-based, not product based, so comparisons between (or among, if more than two) therapeutic approaches can, in fact, be made in the context of the registry. 4. Page 90: Minor wording point. You say, “Many studies of diagnostic tests evaluate how a test affects the likelihood of a disease” The test does NOT affect the likelihood of disease. The test result helps predict or assess the likelihood of disease, and the study of accuracy relates to this ability to predict. Action based on the test may affect outcome, but that’s not what your statement said.5. On diagnostic tests: an example or two of randomized studies of diagnostic tests and their effects on outcome, would be useful. There are several good examples, including mammography (especially under age 50), PSA screening, and there are likely to be several others.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

1. Page 95: You say you hope to “Distinguish between standards that are minimum requirements and those that may be aspirational or best practice but are not required.” This will be an interesting challenge (that was faced by the IOM committee on standards for systematic reviews.) Is it a standard or not? If not, is it a “recommendation”? What’s the incentive for a researcher to adhere to a standard that’s specifically mentioned as being “aspirational” but not necessary? (It will be non-trivial coming to agreement on which standards fall into which category.)


Author

Merck & Co., Inc.

Date

Friday, September 14, 2012

Zip Code

19454

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

PCORI should provide more guidance on hypotheses and hypothesis testing, pragmatic clinical trials, selection of appropriate comparators, unmeasured or onknown confounders, and adjustment for multiple comparisons.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

All funding announcements and responses to inquiries about grants should be accompanied by a link to the methodology guide.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Merck-Comments-on-the-PCORI-Methods-Guide-9-14-12.pdf


Author

American Academy of Neurology

Date

Friday, September 14, 2012

Zip Code

48109

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

None

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

The report needs to be more concise.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

No problems

Please provide your comments on Chapter 2 of the Draft Methodology Report.

No problems

Please provide your comments on Chapter 3 of the Draft Methodology Report.

No problems

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Sections 4-8 need to be more concise and perhaps in a separate document than sections 1-3.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

See comment to 4

Please provide your comments on Chapter 6 of the Draft Methodology Report.

See comment to 4

Please provide your comments on Chapter 7 of the Draft Methodology Report.

See comment to 4

Please provide your comments on Chapter 8 of the Draft Methodology Report.

See comment to 4

Please provide your comments on Chapter 9 of the Draft Methodology Report.

See comment to 4


Author

Yongmei Li

Date

Friday, September 14, 2012

Zip Code

94112

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

The recommended actions are closely in accordance with PCORI's mission and vision. However, this report is largely methodological, and appears to have paid insufficient attention to patient-centeredness and preferences (this might be because it is a new territory). In addition, there is a lack of incorporation of payers' perspective in the report. The multiple-payer system plays a big role in determining patient preferences, outcomes and utility.

2. What methodological gaps should be addressed in future reports?

On the topic of Standards for Causal Inference Methods, this report explains very well the challenges of using data from observational studies and methods to mitigate potential biases, such as propensity score and instrument methods. It would be greatly beneficial if future reports recommend other statistical tools (such as Marginal Structural Model" for "channeling effects". For example, during follow-up, the treatment strategies might vary and the baseline characteristics might become less informative. Future reports could also attach more importance to information beyond baseline (possibly modify or add additional items to "Causal Inference Standards" on page 64). What is the role of genetics in patient-centered outcomes research?How to incorporate payers' perspective and impact in evaluating patient preferences, outcomes and utility?

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

The report highlighted priorities of research and key methodologies in carrying out different types of studies. These are state-of-the-art and comprehensive guidelines for planning for studies.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

PCORI might disseminate the final report to stakeholders who are interested in patient-centered outcomes research, mainly the health care system, researchers, and less likely to payers. It's not practical to distribute the final reports to patients who might not understand the technical terms in this report.


Date

Friday, September 14, 2012

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Definitely it is consistent with the PCORI mission!It follows the guidelines of Patient Protection and focuses on affordable care.

2. What methodological gaps should be addressed in future reports?

At this point there seem to be not many gaps, the methodology committee has done an exceptional job!

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

My only suggestion there is to have the simpler language, many people don't know how to interpret technical terms.

5. Please enter any additional comments or concerns here.

I LOVE the mission and contributions of PCORI so far and feel proud to be a very small part of this mission. Reading this report made me realize what a monumental task we have ahead of us. I must congratulate the methodology committee for an excellent and exhaustive report!

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Very reader friendly introduction!Reference to Appendix A is perfect right in the beginning.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Defining terms is the most difficult task and addressing this issue is done very well here!Very useful information in appendix E-5!

Please provide your comments on Chapter 3 of the Draft Methodology Report.

This chapter really catches the attention towards pursuit of knowledge, the box with 'A Thought Experiment' is truly thought provoking- an exteremely important step in research!

Please provide your comments on Chapter 4 of the Draft Methodology Report.

I love the standards in this section-they are absolutely geared towards patient's health!

Please provide your comments on Chapter 5 of the Draft Methodology Report.

This chapter is the backbone of entire report. It emphasizes very pertinent points of review and analysis.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

This is extremely important guideline for the researchers.Most people get diffused without this kind of approach, this chapter is the only publication addressing the data source selection in such clear and concise way. The diagrams and figures and priceless!

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Assessment of standards is the key to a great proposal!The systematic approach to crosscutting and inference standards is extremely well done here. The table with 'standards for heterogeneity of treatment effect' is extremely useful.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

It is very clear and flowing with the reader. Very well written section with comprehensive and measurable standards.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

I love the emphasis on Patient-centeredness and research priorities in this chapter.


Author

American Medical Association

Date

Friday, September 14, 2012

Zip Code

20001

Stakeholder Category

Clinician

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comment-Letter-on-PCORI-Methodology_09-14-12_2.pdf


Author

Medtronic, Inc.

Date

Friday, September 14, 2012

Zip Code

55432

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Chapter 6The last bullet point on page 50 reads “A study design without a comparator, based on information from a device registry, might be acceptable for assessing device failure rates, but not to assess device effectiveness.” The unqualified rejection of a study design without a comparator for assessing device effectiveness is inappropriate. While the presence of a comparator would certainly make it easier to assess effectiveness, the absence of a comparator does not preclude such assessments entirely (as the current language would seem to suggest). For example, the study design might call for de novo data collection for a single device, while the planned analysis might include a comparative analysis between previously collected data on another device, and the newly collected data on the device currently being studied. The problem stems from confusing data collection strategies and data analysis strategies. Refer back to item #4 on page 45: “Place individual research studies in the context of a research program.” It might be entirely appropriate for a set of studies to collect data on single devices in parallel, while an overall research program analysis plan provided for subsequent analysis across all of those studies. Taken in isolation, it might be difficult to assess effectiveness of a given device in each study. However, taken together, the group of studies might offer an efficient means of determining effectiveness across several devices. To take a less complicated example, suppose that there was interest in determining how a new device performed in relation to a well-studied benchmark therapy. In such a case it might be acceptable to use fixed values of the benchmark therapy (means, lower confidence intervals, or other metrics of interest), and compare new data from a competing therapy. This would be a study design without a comparator, but one in which effectiveness was assessed in the analysis phase.This issue continues in the associated example on the top of page 51. It might be appropriate to add something along the lines of the following at the end of the example. “Such comparators could, however, be framed in terms of different populations within the study, be fixed quantities (like objective performance criteria or performance goals), or be obtained from previous data, incorporated into analysis of data from the current study.” Again, the document is perhaps confounding issues of design and analysis.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Chapter 7 Item 3.1.3 in the table on “General and Crosscutting Methods for All PCOR” on page 59 calls for some studies to be designed “with adequate precision and power to reach conclusions specific to” certain subgroups. There's a difference between being able to make inference about a subgroup in the context of a larger model, and requiring that there is sufficient data within a given subgroup (effectively ignoring data outside the subgroup for the analysis). An extra sentence along the lines of “Modeling techniques incorporating all available data may allow inference to be made on certain subgroups with fewer observations than would be required if the members of the subgroup were considered in isolation” might be useful here.On page 61, the first paragraph on “Standards for Causal Inference Methods” discusses the notion of an average treatment effect in a randomized trial. Average here seems to mean “averaged across replications of similar trials”, not averaged across individual subjects within a trial. Such distinction does not appear to be accurately conveyed here. It might be sufficient to add a sentence as follows: “Average in this context means averaged across multiple possible realizations of data in the trial”, or something to that extent. On page 64, consider changing the last sentence in the first new paragraph. The new sentence would read “In some cases, statistical methods such as propensity scores can combine several characteristics into one variable that is used for matching or stratification.” The “or stratification” phrase is useful to indicate a broad area of application for propensity scores apart from direct matching. On page 66, it may be useful to consider changing a sentence near the end of the page. The new sentence would read “Use of subgroups considered marginally also divides the sample, reducing statistical power.” The italicized words are new, and are an important qualification for the statement about dividing the sample. On page 74, there is a discussion about various missing data methods. It would be appropriate here to indicate that analysis methods should be encouraged which deal in an effective manner with unequally spaced observation times. Examples of this include survival analysis for time to event endpoints, or MMRM with a spatial power covariance matrix to deal with irregular spacing in a parsimonious fashion.On page 77, item 7.5.5 covers “Standardized Terminology Encoding of Data Content”. It might be useful here to make explicit reference to specific standards, such as CDISC or MedDRA.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Chapter 8 On page 81, second sentence, consider changing “Adaptive designs refer to randomized trials…” to “Adaptive designs generally refer to randomized trials…” It is possible to construct a randomized design for a single arm (hence non-randomized) trial. On page 82, item 8.1.1 addresses the role of the statistical analysis plan (SAP) for adaptive and Bayesian trial design. Instead of being finalized prior to study enrollment, the SAP needs to be finalized before access to information that might influence the analysis. For blinded trials, it could be sufficient to finalize certain aspects (such as secondary analyses) after the start of enrollment, but before the study blind is broken. On page 89, the second paragraph of “Rationale for the Standards” touches on the issue of enrollment for registries. The plans for enrollment should include mention of the type of sampling to be used (e.g., random, convenience, or universal sampling). This impacts both design and analysis considerations.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Appendix AOn page 107, item 3.1.2 should consider including a statement that the research proposals and protocols should address how the results might be calibrated or extrapolated to various sub-populations of interest. This would be one mechanism to address some of the issues involved in various subgroup analyses.On page 107, item 3.1.4, consider changing the last sentence, and adding in one addition. The modified sentence: “Generally, non-use (or no specific treatment) comparator groups should be avoided in analyses unless no specific treatment is a likely option in standard care.” Italicized words are new. A proposed additional sentence would be as follows: “However, it may be appropriate to collect data from one group at a time, for use in subsequent or separate comparative analyses.”


Author

Clyde Schechter

Date

Friday, September 14, 2012

Zip Code

90630

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Circulating a 200+ page report will probably not work well. A much shorter version is needed. But in addition, I would propose creation of an on-line decision-support tool that will enable researchers to respond to questions about their research ideas and get back advice on methodology.The methodological standards proposed here set a fairly high bar. For example, the standards for heterogeneity of treatment effect (HTE), while representing best statistical practice, will entail very large sample sizes in non-exploratory studies. Yet, at least in its funding announcements so far, there is a limit of $500,000 per year direct costs. It is very difficult to follow large patient populations within that budget. Many clinical researchers use a rule-of-thumb that it costs about $1,000 per patient per year to do longitudinal studies, assuming the measurements themselves are not unusually expensive or unusually cheap. By that rule of thumb, study samples > 500 will be difficult or impossible to support. But for most realistic HTE questions, the heterogeneity will be too small to be detectable in a total sample that small, even if there are only 2 treatment groups of equal size. If PCORI wants these standards to promote HTE research (as I believe it should) it will need to provide much larger amounts of funding for that type of study.I admire and endorse the attempt to set high standards. But it can't be done on the cheap. If PCORI is serious about getting the best quality research, it will have to narrow its reach to provide high levels of support to a smaller number of carefully selected projects (or obtain additional resources from Congress-good luck with that!).

Please provide your comments on Chapter 1 of the Draft Methodology Report.

No comments.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

No comments.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

The standards are good in principle. Bear in mind that it takes a lot of space in a proposal to comply with these standards in a clear and convincing way. Page limits for grant poposals will need to be quite generous for these standards to gain adherence. (This in turn may make it more difficult to recruit reviewers, or require recruiting a larger pool of reviewers.)

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Same as my comment for Chapter 3.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

No comments.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

This translation framework might serve as a starting point for on-line software to help researchers with designing their projects and writing proposals.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

I disagree with the strong endorsement of multiple imputation (MI) and the disparagement of approaches like last observation carried forward (LOCF). There are many areas of research where the hypothesis that data are missing at random (MAR) is not tenable. While MI (or FIML) can, in principle, be applied to these situations, it requires positing a model of missing data that, inherently, has no empirical basis. Furthermore, MI estimation with such models is difficult at best and is not supported in commercially available statistical software--it requires custom software creation and there are few people even in the statistics community who are capable of doing it. I think the standard should be modified so that if the investigator makes a reasonable case that data missingness is not at random, then MI (as usually implemented in commercial software) may be inappropriate and may create the illusion rather than the reality of more valid results. In such a situation, particularly if attrition is driven by the unobserved missing values (e.g. in a study with symptoms as outcomes, those who are not experiencing improvement are more likely to drop out), an approach such as LOCF might actually be more informative and less misleading than an MI-based analysis. True, as a single-imputation method it will underestimate variances, but its results may be far less biased than would be obtained applying MAR-MI techniques.In my experience, situations where the MAR assumption is simply implausible are common, perhaps even the rule in clinical research with outcomes that patients themselves can perceive.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

No comments.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

No comments.


Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

CAP did not identify any methodological gaps.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

CAP believes the report can be used to inform policy regarding grant reviews and submissions.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CAP-PCORI-Statement-FNL.pdf


Author

Jeannette Vicknair

Date

Friday, September 14, 2012

Zip Code

70065

Stakeholder Category

Patient

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Does PCORI plan to distribute this to meetings with oncologists, such as ASCO, San Antonio breast, etc. Also including the nurses, who actively treat and interact with the patients, whose lives you are addressing, could provide positive input to this discussion.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Including patients in topic generation is unconventional. while this is t rue, it very much needed in the current treatment enviroment of cancer. Patients are living longer. And outliving the expected times frames that most researchers, practioners expect. This includes the insurance companies whose actuaries, I assume, include the decreasing and increasing numbers of policyholders in their projections. Therefore the patients, such as I , that have outlived the usual timeframe for cancer, can have a greaeter imput into the actual patient needs.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Expect that several versions of a translation table might be needed for differentdecision-makers and different research categories. So true.


Author

MITA

Date

Friday, September 14, 2012

Zip Code

20815

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/MITA-Comments-to-PCORI-on-the-Draft-Methodology-Report-FINAL.pdf


Author

Judith Baggs

Date

Friday, September 14, 2012

Zip Code

97239

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

No

1 Comments

I think this is a very exciting opportunity, but I think the Methodology Committee failed to take a crucial first step - to define "patient" in patient-centered. I come from the discipline of nursing. We define patient as an individual, a family, a community. This report is extremely individual patient centered. As an analogy in the election year, it has a Republican perspective - individual above all, rather than a more Democratic perspective on the importance of the community and helping each other. You do reach out to include caregivers or legally responsible individuals, but there is no room in here for the family as patient or for public health issues.

5. Please enter any additional comments or concerns here.

I was troubled by the make-up of the committee. Physicians are heavily represented, social workers not at all, and nurses, the largest healchcare profession have a single representative. Given that nurses and social workers have been traditionally patient focused, the problems I identified above (lack of a definition of patient that is more inclusive than an individual) are not at all surprising.


Author

Lance Armstrong Foundation

Date

Friday, September 14, 2012

Zip Code

78702

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While we strongly support report recommendations, we also hope that rigorous standards be applied in a way that invites both quantitative statistical and qualitative descriptive research.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Patient advocacy groups can serve as a key resource for disseminating recommendations. However, it will be important to translate standards into layperson terms and a user-friendly format in order to educate and engage a broader range of patient and caregiver or informant stakeholders.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-draft-methodology-LIVESTRONG-comments.docx

Please provide your comments on Chapter 4 of the Draft Methodology Report.

The standards proposed for achieving patient-centeredness and engagement in research are of particular interest to Lance Armstrong Foundation, given our research objectives and activities. The framework outlined reflects those on which we have come to rely in our work, and the anecdotes are especially poignant in highlighting the common lack of focus on what matters to patients themselves. We are keen to assist the committee in further exploring this aspect of PCORI’s methodology, should the opportunity arise. Specifically, the Foundation could offer insight in the development or evaluation of investigator and informant training, dissemination and integration of findings, and a sample engagement plan with elements that emphasize patient-centered principles that are shared by our broad network of patient and research stakeholders.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Regarding standards for research prioritization and review, the foundation agrees with the role of patients in topic generation and peer/stakeholder review and has extensive experience in this area that can also be offered to PCORI or its grantees for further case studies or technical assistance.


Author

American Academy of Orthopaedic Surgeons

Date

Friday, September 14, 2012

Zip Code

20002

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

See attachement

2. What methodological gaps should be addressed in future reports?

See attachement

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

See attachement

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

See attachement

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AAOS_PCORI-comment-letter-Draft-Methodology-Report-091412.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 2 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 3 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 4 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 5 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 6 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 7 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 8 of the Draft Methodology Report.

See attachement

Please provide your comments on Chapter 9 of the Draft Methodology Report.

See attachement


Author

Partnership to Improve Patient Care

Date

Friday, September 14, 2012

Zip Code

20006

Stakeholder Category

Patient Advocate

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PIPC-Methods-Comments-FINAL.pdf


Date

Friday, September 14, 2012

Zip Code

44109

Stakeholder Category

Clinician

5. Please enter any additional comments or concerns here.

I just want to say that the overall process is so long and detailed that only already funded researchers (and/or other members of the guild) can possibly hope to participate. This may exclude many people, for example, full time clinicians who are interested in improving the evidence foundation for practice, from even participating in the process. Maybe you could develop a mechanism of small grants based on less massive applications processes that would provide such people with some protected time to do the foundational work to develop a grant proposal and prepare your required lengthy submissions. Thanks for your consideration.


Author

Elaine Larson

Date

Friday, September 14, 2012

Zip Code

10032

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

This is a terrific report but in Chapter 5, Methods for Prioritizing Patient-Centered Outcomes Research and for PeerReview, the recipients of care are essentially absent from the priority setting process--their input is quite passive. This is surprising because PCORI is precisely charged to identify research of major health relevance. The Report still implies that providers are the 'experts' who decide. This is subtle, but present in the Report.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

We at Columbia will certainly plan symposia to 'distribute the final report' and discuss implications. Other institutions need to be encouraged to do so. To make that happen, it would be helpful for PCORI to prepare a slide set that could be used for such symposia.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

None

Please provide your comments on Chapter 5 of the Draft Methodology Report.

This is a terrific report but in Chapter 5, Methods for Prioritizing Patient-Centered Outcomes Research and for PeerReview, the recipients of care are essentially absent from the priority setting process--their input is quite passive. This is surprising because PCORI is precisely charged to identify research of major health relevance. The Report still implies that providers are the 'experts' who decide. This is subtle, but present in the Report.


Author

Sary Beidas

Date

Friday, September 14, 2012

Zip Code

30813

Stakeholder Category

Clinician

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

There are 2 variables in which patient-centered research methodology should be expanded beyond a randomized controlled trial. The first variable is patient perspectives and the second is the escalating use of information technology.An example of methodologies that are not given sufficient weight in the report are: qualitative studies and mixed methods research. I recommend that the committee gives weight to these methodological gaps for acceptable PCORI submissions.


Author

J Chan

Date

Friday, September 14, 2012

Zip Code

94127

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes, the recommendations are consistent with PCORI's mission and vision.

2. What methodological gaps should be addressed in future reports?

As PCORI gains experience with application of methodological standards to PFA submissions in upcoming cycles, the process will facilitate identification of gaps and contribute to the evolution of methodological standards.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Yes, this report provides general information on PCORI policy regarding grant reviews and submissions.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Email and Webcast, assistance from affiliated organizations.Other Notes>At the current state of healthcare policy, infrastructure and technology development, the timeliness of our collective research efforts to demonstrate patient-centered outcomes effectiveness is perhaps of significance, the ability for our research efforts to provide evidence and potentially influence healthcare policy that will foster improved patient-centered outcomes is a window of opportunity. Thank you for inviting us to this comment process!

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Section 4.1.3>Pg 26> Use Patient-Reported Outcomes When Patients or People at Risk of a Condition Are the Best Source of Information. Notes> Section prerequisites > adequate health literacy and no language barrier


Author

European network for Health Technology Assessment

Date

Friday, September 14, 2012

Stakeholder Category

Researcher

5. Please enter any additional comments or concerns here.

This feedback is basically intended to send a signal that activities similar to those behind the report are taking place within the network of government nominated agencies and institutions in 26 member counties in the European Union supported by the European Commission (www.eunethta.eu). We currently have 9 guidelines for relative effectiveness assessment of pharmaceuticals in "public consultation" (public comment). With obvious similarities in important remits between PCORI and us the European network for Health Technology Assessment, EUnetHTA, wish to express an interest in developing contacts with PCORI with the intention of exchanging information and raising standards in patient centrered outcomes research at a global level.


Date

Thursday, September 13, 2012

Please provide your comments on Chapter 4 of the Draft Methodology Report.

September 13, 2012Dr. Joe SelbyExecutive DirectorPatient-Centered Outcomes Research Institute1828 L St., NW, Suite 900Washington, DC 20036Sherine Gabriel, MD, MScProfessor of Medicine and of Epidemiology, William J. and Charles H. Mayo ProfessorMayo ClinicChairPCORI Methodology CommitteeRe: Public Comment on Standard 4.1.5 in the PCORI Draft Methodology ReportDear Dr. Selby and Dr. Gabriel:We appreciate the openness demonstrated by the Patient Centered Outcomes Research Institute (PCORI) in providing this solicited opportunity to comment on the Draft Methodology Report (hereafter referred to as the Report). We recognize the strong efforts made by the PCORI Methodology Committee to develop proposed standards for patient-centered outcomes research (PCOR) under the break-neck timelines legislated by the Patient Protection and Affordable Care Act (ACA).We are commenting only on Standard 4.1.3: Use Patient-Reported Outcomes When Patients or People at Risk of a Condition Are the Best Source of Information.To assist the PCORI Methodogology Committee in formulating this standard, PCORI contracts were granted to two independent teams of researchers. The first team included present and past members of the International Society of Quality of Life Research (ISOQOL) board of directors led by the current ISOQOL president, Bryce Reeves at the University of North Carolina-Chapel Hill and Zeeshan Butt at Northwestern University in Evanston, Illinois. This research team, hereafter referred to as the ISOQOL group, provided background and a proposal for minimum methodologic standards in the design and selection of patient-reported outcomes measures (PROs) for use in PCOR through expert input, followed by a thorough literature review to prepare draft minimal methodologic standards in the design and selection of patient-reported outcomes measures (PROMs) for use in patient centered outcomes research. In addition, an online survey of the entire ISOQOL membership was conducted, and a standard was designated as recommended when over 50% of ISOQOL respondents endorsed it as “required as a minimum standard.” It is important to disclose that as a current ISOQOL board member, Kathleen Wyrwich and Dennis Revicki, former board member, volunteered to participate in all stages of this team’s research, including completion of the online survey and review of the results presented to PCORI.The second contracted team from Oxford Outcomes lead by Sarah Acaster (San Francisco, California) and Andrew Lloyd (Oxford, United Kingdom) also proposed and justified minimum methodologic PCOR. Their approach included discussion with internal resources, a brief targeted literature review and website search to identify concepts and themes as either potential minimum standards or issues for consideration. These concepts and themes were then reviewed by internal and external experts. With less than three months between the announcement of contract awarding and the presentation of the final reports, both selected teams completed their respective contracted projects within this narrow timeframe at a total cost of $153,613 ($83,908 and $69,705, respectively).The resulting minimal methodologic standards in the design and selection of PROs for use in patient-centered outcomes research were quite similar, with each team purposely seeking the appropriate balance of scientific detail without over-reaching as to create minimal standards so prescriptive that they would discourage the use of PROs in PCORI research. Sadly, many of these scientific details are not included in the Report. Therefore, we ask that PCORI reconsider the following issues and supported statements for inclusion in the next version of the Report.1. Patient Burden Although “overly burdensome patient-reported outcome measures” is used as a potential reason that a patient may withdraws from the registry that should be identified and addressed in Standard 8.2.8 (Data Registries), both final reports stressed this as an important minimum standards. Indeed both groups’ reports to PCORI also addressed that a PRO measure must not be overly burdensome for patients, and that the length of the PRO measure should be considered in the context of other PRO measures included in the assessment, the frequency of PRO data collection, and the characteristics of the study population. Although these specific issues may appear to be included in several other standards within the Report, their direct inclusion in Standard 4.1.3 will provide additional guidance to future PCORI research teams. To determine burden to patients, the targeted patient group should be consulted, perhaps through pilot testing before the main study is implemented.In addition, both reports addressed PRO literacy levels, with close consensus. The ISOQOL-organized group recommends “The literacy demand of the items in the PRO measure should usually be at a 6th grade education level or lower; however, it should be appropriately justified for the context of the proposed application” (ISOQOL), while the Oxford Outcomes group noted that “approximately a fifth of American adults read and comprehend below a fifth grade level, with the figure rising to approximately half the US population if level 2 literacy (reading and comprehending at 5th-7th grade levels)” in their discussion of issues for consideration. I hope that PCORI will reconsider the addition of the ISOQOL group’s recommendation given that it focuses future PCORI researchers to address and justify PRO literacy concerns. We request that PCORI consider the addition of the following statement in Standard 4.1.3:A PRO measure must not be overly burdensome for patients. The length of the PRO measure should be considered in the context of other PRO measures included in the assessment, the frequency of PRO data collection, and the characteristics of the study population. The literacy demand of the items in the PRO measure should usually be at a 6th grade education level or lower; however, it should be appropriately justified for the context of the proposed application.2. ReliabilityBoth the ISOQOL and Oxford groups submitted to PCORI minimal standards that address the need for internal consistency and test-retest reliability should be estimated and reported for each domain when appropriate (based on the nature of the PRO and the patient population), with Cronbach’s alpha coefficient ≥0.70 as a threshold guideline. The Oxford Outcomes group noted that detailed reporting is required so reliability can be appropriately evaluated, and assists meta analyses. Disappointingly, the Report states that PCORI proposals should describe “evidence of measurement properties including …reliability,” with no directives on how reliability should be assessed or any relevant thresholds. Again, for the purpose of providing additional guidance to future PCORI research teams on this important PRO issue, we request that PCORI consider the addition of the following statement in Standard 4.1.3:The reliability of a PRO measure should preferably be at or above 0.70 for group level comparisons, but may be lower if appropriately justified. Reliability for multi-item scales should include an assessment of internal consistency and test-retest reliability, and reliability for a single item measure should be assessed by test-retest reliability.3. Recall PeriodWhen patients are asked their input about a specific concern, identifying the appropriate timeframe for consideration (e.g., currently, past 24 hours, past 7 days, past 2 weeks) is essential for standardizing the measurement across all participants. This timeframe is often referred to as the recall period, and it must be appropriate for the purpose/intended use, the patient population, the disease/condition, the treatment/device, and the study design. Ideally, patient input regarding the most appropriate recall period is assessed throughout the PRO measure’s development and psychometric evaluation process, and is essential to patient understanding of the PRO and subsequent use in PCORI research.Although patient involvement in the PRO development process is included in Standard 4.1.3, the importance of the recall period is not addressed, although both the ISOQOL and Oxford Outcomes reports to PCORI stressed this important issue. We request that PCORI consider the addition of the following statement in Standard 4.1.3:PRO measures must provide justification for the recall period for the measurement application.4. TranslationBoth the ISOQOL and Oxford Outcomes reports to PCORI stressed the need for translatability and linguistic validation for PROs. Although this there is mention of the phrase “cultural and language translations” on page 130 of the Report, the importance of this PRO concern often disregarded without consideration of patient understanding of the translated PRO measure, as well as the problematic use of culture-specific terms (e.g., shoveling snow, bowling, gardening, etc.). Again, for the purpose of providing additional guidance to future PCORI research teams on this important PRO issue, we request that PCORI consider the addition of the following statement in Standard 4.1.3:A PRO measure translated to one or more languages should have documentation of the methods used to translate and evaluate the PRO measure in each language. Studies should at least include evidence from qualitative methods (e.g., cognitive testing) to evaluate the translations.With the investment by PCORI in these two strong awardees entrusted to provide minimum standards, the two specific approaches used by each group and the high level of consensus achieved by both groups in their reports to PCORI, we ask that these four concerns receive a close re-examination to ensure that appropriate minimal standards are used to solicit patients’ voices PCORI proposals and subsequently funded research. Much of the literature utilized in these reports and recommended by these two groups has been incorporated into the Report’s discussion on page 130, but there is far more to be learned by all in the PCORI research community from these efforts. If needed, we can provide PCORI with detailed examples of how neglecting any one of these concerns above can lead to uninterpretable or inappropriate study results, and, indeed, patient confusion during the PRO measurement process.Also, as a very small recommended change, the Wyrwich et al. (2012) reference on page 133 now available as an electronic publication. The citation is:Wyrwich KW, Norquist JM, Lenderking WR, Acaster S, and the Industry Advisory Committee of International Society for Quality of Life Research (ISOQOL). Methods for Interpreting Change over Time in Patient-reported Outcome Measures. Quality of Life Research. 2012 Apr 17. [Epub ahead of print.]Many thanks for this opportunity to provide my comments on this important standard. We look forward to reading the next iteration of this living document.Sincerely, Kathleen W. Wyrwich, PhD Dennis A. Revicki, PhDSenior Research Leader Senior Vice PresidentUnited BioSource Corporation United BioSource Corporation


Author

Shire Development LLC

Date

Thursday, September 13, 2012

Zip Code

19087

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Please see the attached document summarizing them.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Symposium, Congresses, email distribution, etc.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Shire-Collective-Comments-to-PCORI-Draft-Methods-report-Sept_14_2012_clean.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

ES1-2: “Over time, these reports, standards, translation tables, and public engagement forums are expected to produce better research methods, which in turn will provide information of benefit to all stakeholders—researchers planning an investigation, policy-makers weighing the value of healthcare interventions, and patients and their caregivers facing health decisions.”1.) Comment: The term stakeholder(s) is used broadly throughout the document and has 2 definitions provided in Appendix F: Glossary that are exactly the same. PCOR Stakeholders and Stakeholders are both defined as “Anyone affected by health decisions. Stakeholders may include patients, clinicians, caregivers, policymakers.” (p 182 &185) This is not an inclusive list of the stakeholders that could be included when mentioned through the document. One example is the term “researchers” that is used in the excerpt above but not included in the definitions of stakeholders. This needs to be thought out more clearly. One suggestion is to be more inclusive in the general definition of stakeholders, including researchers, payers, industry, etc and then have several sub-definitions that are used more explicitly, such as PCOR stakeholders or research stakeholders – academia, industry, government funded, etc. We should think about all stakeholders that could benefit from this methods report, including the producers of the research that this is aimed at guiding, for the patients’ benefit. 2.) Comment: In order to produce better research methods in comparative effectiveness research (CER), other methods such as indirect treatment comparisons (ITC), meta-analysis, and mixed treatment comparisons (MTC) should be considered the starting point for all patient-centered research. This report begins with and focuses on improving and producing better research methods for CER, yet it skips over the initial stages where CER begins—the existing literature to inform the adaptive designs, registries and diagnostic testing designs which are called for later in chapter 8 of the report. For example, recent advances in methods such as match-adjusted indirect comparisons (MAIC) have been created as a sub-type of ITC which can better inform decision making. We enclose a summary from a recent publication on the subject which was presented at the AHRQ symposium on novel CER methods in June 2011 and published inthe journal Pharmacoepidemiology and Drug Safety as a special edition focused on CER methods in 2012. There are 5 other publications in the recent literature available1-5[Cite 4 Signorovitch et al publications] including ISPOR Good Research Practices on Interpreting Indirect Treatment Comparisons and Network Meta-Analysis for Health-Care Decision Making that state that “[O]btaining patient-level data for all RCTs in the network may be considered unrealistic. As an alternative, one could use patient-level data when available and aggregate-level data for studies in the network for which such data are not available, thereby improving parameter estimation over aggregate data–only models.”6 An example excerpt describing MAIC and some of the benefits of it are described in appendix A. Appendix A: Excerpt Describing MAIC method from Signorovitch et al from Pharmacoepidemiol Drug Saf. 2012:“Comparative effectiveness research requires the comparison of alternative therapies in a timely fashion, often in the absence of direct head-to-head randomized trials. For example, healthcare payers may wish to compare drug A versus drug B when randomized trials have only compared each with drug C (e.g., placebo). A traditional method for indirect comparison in this setting is the adjusted indirect comparison.2–6 This method uses the common comparator drug C to infer the relative effects of drugs A and B. This approach can incorporate multiple trials per treatment2 and has been extended to simultaneously compare multiple treatments linked through a network of direct comparisons7 and to adjust for aggregate baseline characteristics using meta-regression.3–5 Comprehensive and detailed reviews of indirect treatment comparisons have been published,5,6 and guidelines have been developed for researchers and decision makers.8–11 Traditional indirect comparison methods have important and well-recognized limitations. First, indirect comparisons can be biased by cross-trial differences in patient populations.2,5,6,12–14 Second, different measures of the treatment effect [e.g., risk difference (RD) versus odds ratio (OR)] can lead to different T1 conclusions, as shown in the example in Table 1. When only a few trials are available for each treatment of interest, these limitations become more pronounced because meta-regression cannot be used to adjust for cross-trial differences in multiple baseline characteristics, and the choice of effect measure (e.g., OR and RD) cannot be justified by assessments of heterogeneity across trials. If individual patient data (IPD) were available from all trials of interest, potential biases stemming from cross-trial differences could be mitigated by regression adjustment.15–19 However, IPD from all relevant trials are rarely available to researchers. It is more common to have access to IPD for trials involving one drug and only published aggregate data for trials of comparators. Researchers who conduct their own trials, or who collaborate with or are affiliated with trial sponsors, can often find themselves in this situation. Despite the widespread interest in using all available data in indirect comparisons,20 few published methods combine IPD with aggregate data for indirect comparisons, leaving IPD as a largely untapped resource for CER. Matching-adjusted indirect comparison is a recently developed method that can use IPD from trials for only one comparator to adjust for cross-trial differences in baseline characteristics.22 This method works by re-weighting patients with IPD such that their average baseline characteristics match those reported for trials without available IPD (i.e., with summary statistics only). Treatment outcomes can then be compared between balanced populations.”2. Signorovitch et al Excerpt References :Bucher HC, Guyatt GH, Griffith LE, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997; 50:683–91.3. Eckert L, Falissard B. Using meta-regression in performing indirect-comparisons: comparing escitalopram with venlafaxine XR. Curr Med Res Opin 2006; 22(11):2313-21.4. Nixon RM, Bansback N, Brennan A. Using mixed treatment comparisons and meta-regression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis. Stat Med 2007; 26:1237–54.5. Glenny AM, Altman DG, Song F, et al. Indirect comparisons of competing interventions. Health Technol Assess 2005; 9(26):1-134.6. Sutton A, Ades AE, Cooper N, et al. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics 2008; 26:753–67.7. Lumley T. Network meta-analysis for indirect treatment comparisons. Stat Med 2002; 21:2313–24.8. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting Indirect-Treatment-Comparison and Network-Meta-Analysis Studies: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices-Part 2. Value Health. 2011 Jun;14(4):429-37.9. Jansen JP, Fleurence R, Devine B, et al.. Interpreting Indirect Treatment Comparisons and Network Meta-Analysis for Health-Care Decision Making: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1. Value Health. 2011 Jun;14(4):417-28.10. Wells GA, Sultan SA, Chen L, et al. Indirect Evidence: Indirect Treatment Comparisons in Meta-Analysis. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2009.11. NICE Guide to the Methods of Technology Appraisal, http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf12. Phillips A 2003 Trial and Error: cross-trial comparisons of antiretroviral regimens. AIDS 2003 17:619-623.13. Cranney A, Guyatt G, Griffith L, et al. Meta-analyses of therapies for postmenopausal osteoporosis. IX: Summary of meta-analyses of therapies for postmenopausal osteoporosis. Endocr Rev 2002; 23(4):570-8.14. Deeks JJ, Higgins, JPT, et al. Analysing and presenting results. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6; Section 8. In: The Cochrane Library, Issue 4, 2006. 15. Pignon JP, Arriagada R, Ihde DC, et al. A meta-analysis of thoracic radiotherapy for small-cell lung cancer. N Engl J Med 1992; 327:1618–24.16. Homocysteine Lowering Trialists’ Collaboration. Lowering blood homocysteine with folic acid based supplements: meta-analysis of randomised trials. BMJ 1998; 316:894–8.17. Non-small Cell Lung Cancer Collaborative Group. Chemotherapy in non-small cell lung cancer; a meta-analysis using updated data on individual patients from 52 randomised clinical trials. BMJ 1995; 311:899–909.18. Moore RA, McQuay HJ. Single-patient data metaanalysis of 3453 postoperative patients: oral tramadol versus placebo, codeine and combination analgesics. Pain 1997; 69:287–94.19. Turner RM, Omar RZ, Yang M, et al. A multilevel model framework for meta-analysis of clinical trials with binary outcomes. Stat Med 2000; 19:3417–32.20. Neumann PJ, Drummond MF, Jönsson B, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? International Journal of Technology Assessment in Health Care, 26:1 (2010), 71–78. 21. Ross SD. Trends in meta-analysis. Drug Inf J 2009; 43:171–76.22. Signorovitch JE, Wu EQ, Yu AP, et al. Comparative effectiveness without head-to-head trials: a method for matching-adjusted indirect comparisons applied to psoriasis treatment with adalimumab or etanercept. Pharmacoeconomics 2010;28(10):935-45

Please provide your comments on Chapter 2 of the Draft Methodology Report.

1.) Comment: The term stakeholder(s) is used broadly throughout the document and has 2 definitions provided in Appendix F: Glossary that are exactly the same. PCOR Stakeholders and Stakeholders are both defined as “Anyone affected by health decisions. Stakeholders may include patients, clinicians, caregivers, policymakers.” (p 182 &185) This is not an inclusive list of the stakeholders that could be included when mentioned through the document. One example is the term “researchers” that is used in the excerpt above but not included in the definitions of stakeholders. This needs to be thought out more clearly. One suggestion is to be more inclusive in the general definition of stakeholders, including researchers, payers, industry, etc. and then have several sub-definitions that are used more explicitly, such as PCOR stakeholders or research stakeholders – academia, industry, government funded, etc. We should think about all stakeholders that could benefit from this methods report, including the producers of the research that this is aimed at guiding, for the patients’ benefit. 3.) Comment: See Comment 1 on stakeholders. Here the term research stakeholders may replace the term for ”researchers, publishers, and industry”. Otherwise the examples are redundant and not all inclusive. Industry does much research, although there are also many other organizations/entities that do research and are not mentioned. Throughout the document, other stakeholders options could be “decision maker stakeholders, payer stakeholders, etc.”.Comments 1 & 3 can also be applied to (these are examples, not an exhaustive list)Chapt 2, pg 10: “It is expected that the translation table will eventually include versions appropriate for use by different stakeholder groups”Chapt 2 , pg 11: “This includes engaging all stakeholders who might use the standards, creating reporting and surveillance opportunities, and developing enforcement functions over time.”

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Comment 4 also applies to:Chapter 3 pg.15. “One aim is to counter over generalization from studies in narrow populations or in highly controlled research settings to the broader range of people and settings in “real life.” (17, 18) But another aim, undoubtedly, is to learn more about how well the different treatment choices can work, and for whom they work best and are safest.” Pg. 7 “The next three years of Methodology Committee work will be a continual process of reconsidering, refining, and widening the scope of the standards to include the full spectrum of PCOR questions and approaches. Similarly, the translation table within this report will be expanded over time to include more examples, methodological issues, and approaches.”5.) Comment: It is a good and fundamental place to start with the existing methods which can be improved with novel approaches as described above and that the report could be expanded now to incorporate PCM and ITC, MTC and meta-analytic methods into the PCORI methodology report.“In 2008, the Institute of Medicine stated that methodological standards for the conduct of one type of research— systematic reviews—would help decisionmakers involved in PCOR “with respect to transparency, minimizing bias and conflict of interest, and clarity of reporting.”(15) In 2011, the Institute of Medicine published such standards.(16)”6.) Comment: These guidelines don’t include novel methods such as MAIC or other advances in ITC, MTC, and meta-analytic methods which have become more readily utilized and standardly applied to more and more therapeutic area reviews for pharmacologic or device interventions.Chapter 3, Table 3.1, pg. 18 “Create conceptual framework Specify hypothesized mechanisms of effect, influences on outcomes, and clinical context of affected care processes to guide data collection and analysis plans”7.) Comment: Patient perspective should be taken into account somewhere along the way, whether it is in the conceptual stages quoted above or in earlier stages of identifying the research questions. “Because some standards apply only to certain types of studies, a portfolio of templates applicable to various study designs should be developed.”8.) Comment: As before, a portfolio of novel methods should be considered such asPCM, MAIC, etc.Chapter 3, pg 20. “Standards for Formulating Research Questions” Box9.) Comment: A statistical analysis plan should be created and finalized (ie signed off) either in parallel or subsequent to the protocol. If it is not specified here, then it should be included in Section 7.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Chapter 4 (Patient centeredness): 10.) Comment: This chapter is very process oriented, and broadly highlights the key issues and the thinking framework needed to think about “patient centeredness” to advance this field. These points should be further elaborated upon and described in greater detail in this chapter. The following is a suggested framework of thinking that could be proposed: a. Define outcomes that are meaningful to patients that are measurable and concrete, especially in disease areas where outcomes have not been defined (this is mentioned but could be further elaborated upon). Who should be involved? How it should be done, etc.b. Not only should meaningful outcomes be defined, but in order to be able to use this outcome to “personalize” care, this outcome variable should also be able to highlight and accentuate “heterogeneity” across different populations. For example: in attention deficit hyperactivity disorder (ADHD), a patient with the outcomes measure CGI-I score 1 to 2 is defined as a “responder” and those not reaching this threshold classified as “non-responders”. Obviously, this outcome is not that sensitive in picking out differences across the different patients and highlighting the potential heterogeneity across the different individuals, because there may be patients who move from a score of 7 to 4 in the CGI-I, but it this patient group is still going to be classified as “non-responder” whereas in reality they are actually responding but are not meeting the threshold of “response defined”. If there was a more sensitive outcome measure that could be used to categorize patients into the different levels of response and thus “break them into different” sub-groups based on this outcome, then treatment could be potentially tailored to these different subgroups depending on the care needs of each subgroups. c. Suggestions to get to this point include looking at their genetics, coupled with their actual reported outcomes (i.e., PRO), or other patient specific characteristics. This was not mentioned or clearly elaborated in this chapter. Additional suggestions include development of novel innovative methods to identify subgroups of patients who may be responding better to one therapy vs. another. One example in using personalized medicine methods is a paper that was recently accepted in Journal of Medical Economics by Erder et al.7

Please provide your comments on Chapter 5 of the Draft Methodology Report.

11.) Comment: A key element in this report is the key focus on patient involvement and peer review process that has been stipulated. While VOI analysis is a useful tool, it should be supplemented with practical, real-world applications. With regard to research priority factors/focus, these seem very broad and may not be meaningful enough unless further expanded upon. Suggestions to consider would be to focus on heterogeneity among patients and effective mechanisms for meaningful subgroup evaluations and analyses, consider predictive modeling research approaches that can lead to results tailored to patient differences, aim for practical design and outputs that can realistically effect change with prescribers in a patient-centered treatment approach, and aim to assess treatment outcomes of comparators (vs PBO) to identify high value products based on patient outcomes.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

12.) Comment: The Translational Framework & Decision Tree could be very useful tools as long as either all possible study scenarios are considered or the interpretation of them leaves room for unique or new situations. This is important so that studies are not automatically labeled as ”wrong”, especially in grey areas. One way to do this would be to distinguish between minimum requirements and those characteristics that are important, but not required.The validation of this framework is very important to encourage future use and increase the level of confidence around the tool.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Chapt 7 pg 59. 7.1.2 “Researchers should describe the analytic approaches that will be used to address the major research aims prior to data collection” 13.) Comment: Instead of stating the above, it should state “Researchers should DOCUMENT the analytic approaches that will be used to address the major research aims prior to data collection”

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Chapter 814.) Comment: This chapter seems to be focused on “adaptive RCT design” approaches vs. other non-RCT designs that could be adaptive and useful in PCORI approaches. There could also be consideration of other types of modeling used (MSM or prop score model approaches employed) and a trial could be adaptive with regard to sub-group analyses being pre-specified. Indirect modeling comparisons might also be useful for informing trial design, rationale of trial, and endpoint selection or study power. The section on observational registries focused on the drawbacks and limitations of these registries and did not expand upon the benefits that could be achieved by using these methods, though they are briefly mentioned. The registry / real-world use approach is very likely to be applicable for collecting patient centered data and tracking outcomes of broader external generalizability.Chapter 8 pg. 80-81 “Adaptive designs are particularly appealing for PCOR because they have the potential to maintain many of the advantages of randomized clinical trials while minimizing some of the disadvantages.” and “The flexibility and efficiency that are gained in adaptive trials have to be balanced with the risk that such trials typically require a longer design period and involve more logistical complexity.”15.) Comment: There are a fair number of challenges in adaptive trials which need to be mentioned in the document in order to ensure clarity of decision making by the researchers considering utilizing them. First, adaptive clinical trials should model initially to predict the sample properly. This is not clear in the report. If it is not modeled appropriately, the sample may need to be re-forecast up or down and enrollment adjusted after the “adaptive” part of the trial is completed. This could actually mean that they could be longer or require more sample than conducting a RCT directly. It could be more efficient to simulate or model out trial results in lieu of an adaptive trial and be more time-saving. It could also mean that adaptive trials are more expensive than other RCT trials. Worse, when checking that whatever the primary measure of effect is in the adaptive trial at the interim analysis stage, it could be found that the required sample to complete the adaptive trial is so large that funding for such a trial is not available, especially since the majority of the costs for a study are typically at study start-up (e.g. enrolling sites, patient incentives, etc). Lastly, because the interim sample and analysis are often less powered than the full trial, one could power the remainder of the trial based upon the initial interim analysis which could be incorrect simply due to study changes or low power. In medical device and diagnostic studies, this is particularly problematic since there is an operator learning that must be taken into account. The results for most human-operated devices or diagnostics initially will have more variation than later on when they become experienced users. The effect can cause great variation in the reliability and estimates for powering the remainder of the study.Chapter 8 pg. 82 section 8.1.3 “This should include any statistical models used either during the conduct of the trial or for the final analysis, prior probability distributions and their basis, utility functions associated with the trial’s goals, and assumptions regarding exchangeability (of participants, of trials, and of other levels).”16.) Comment: when there is the possibility for multiple cut-offs or probabilistic approaches via Bayesian approach, sensitivities should be explored and modeled out to understand the potential impact of the extreme assumptions at the very least. Selection of which prior probabilities to use can cause the greatest variation in Bayesian adaptive trials. Chapter 8 pg. 83 “For example, these studies typically require simulations in the design phase in order to define the error rates, and descriptions of the design both in protocols and published papers must include more elements than a non-adaptive trial. Good adaptive trial design requires pre-planning and specification of procedures at the outset.”17.) Comment: We agree with the comment above, but just to clarify and suggest: the MAIC approach is an appropriate method which could be used for this tool as well. It can “simulate” a trial outcome by matching patients within current trials. It gives the prior probability for an outcome of interest as long as that is the same outcome of interest that is sought for the adaptive approach; it can be very useful. The same applies to pg. 94- “Support development and use of software for adaptive trials that can simulate complex designs.” The MAIC approach can be a method that could be modeled out in the future using such software. Chapt 8 pg. 85 “Standards for Data Registries” 18.) Comment: Systematic literature review and synthesis of some kind is required as the first step before any registry can be planned to know what has previously been collected and what hasn’t been collected, but should be in the registry (in the interest of the patients).


Author

Anonymous

Date

Thursday, September 13, 2012

Zip Code

27510

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

But see uploaded comments on composition of Methodology Committee.

2. What methodological gaps should be addressed in future reports?

Standards for methods of qualitative data collection and analysis within the context of comparative effectiveness research..An empirical evaluation of the benefit of including patients as partners in CER research.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

..with the addition of more information regarding the integration of patients as partners in research..

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

When finished you may have enough to publish a book.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Feedback-to-PCORI-on-Draft-Methodology-Report.docx


Author

Personalized Medicine Coalition

Date

Thursday, September 13, 2012

Zip Code

20005

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PMC-PCORI-Methodology-Committee-Report-Comments.pdf


Date

Thursday, September 13, 2012

Zip Code

46285

Stakeholder Category

Industry

2. What methodological gaps should be addressed in future reports?

Please see uploaded document or recommendations and comments provided on specific chapters.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/FINAL-Lilly-Comment-Letter-on-PCORI-Draft-Methodology.doc

Please provide your comments on Chapter 4 of the Draft Methodology Report.

•Increasing the relevance, meaningfulness, and interpretability of data to all stakeholders (including patients) is an important progression in conducting and reporting scientific research as it pertains to drug development. An important component of this is including measures that are most relevant to patients and recruiting representative populations. Consideration of patient-centeredness during study design is very important in understanding patient’s needs, deciding inclusion/exclusion criteria, developing the data collection forms, and evaluating the feasibility of the study. •The proposed standard (4.1.1), states “At a minimum, patient informants should be engaged in formulating research questions; defining essential characteristics of study participants, comparators, and outcomes; monitoring study conduct and progress; and disseminating results.” Although it is key to include patients in determining the relevant questions for research, there are several issues regarding further discussion with participants following the initiation of a study. These issues include, but are not limited to, patient confidentiality, ethics, legal, and regulatory concerns, comprehension of medical practices and study design, as well as methodological complications with ‘intervening’ with patients under Good Clinical Practice. PCORI should consider how these issues should be addressed in the context of the proposed standard. •We recommend that the standards specifically state that researchers should follow all applicable laws and regulations.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

•This section provides a solid framework for research and will help focus research where it is needed. However, the use of systematic reviews as a means for identifying gaps in research priorities is a complex concept, and the guidance provided in the draft methods report is vague. While systematic reviews are a key component of PCOR, patient-level data may also be used to perform hypothesis-generating analyses using methods such as regression models, cluster analysis, CART analysis, etc. Therefore, this section should include additional methods for studying efficacy and effectiveness at the patient level in addition to systematic reviews.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

•The non-randomized study section of this chapter should incorporate observational study designs. Furthermore, PCORI should update Figure 6.2 entitled Decision Tree for Comparative Effectiveness of Therapeutics to reflect the appropriate nature of observational study designs. Observational research does not rely on recruitment strategies and instead considers inclusion/exclusion criteria.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

•It would be a benefit to have PCORI endorse standard definitions that would apply to all patient-centered outcomes research (e.g. definition of patient sample, definition of intervention group, and definition of relevant outcomes).Causal Inference:•The draft report on causal inference provided sound scientific rationale and a clear example supporting the need for proper bias adjustment techniques in non-randomized studies – as well as a few suggested key principles for analysts to follow. The section outlining causal inference methods should be expanded to address the strength, weakness, and scope of different methodologies in regards to adjusting for selection bias. •There are various causal inference methods which may not apply in every situation, make different assumptions, may be more appropriate than another, and may demand different sensitivity analyses in each situation. We recommend PCORI provide more clarity in order to provide guidance that will lead to trusted and reliable information. The information provided in the draft methods report is a starting point, but much more detail is needed. Without more specificity or guidance, there is a risk that conflicting results on the same question will occur without clarity as to why or how to resolve the differences.•The causal inference standards are limited and only focused on a small number of analytical methods. There is only a brief mention of multivariate regression, propensity scoring, and instrumental variable, meaning more information is necessary to provide clearer directions and standards for researchers. More guidance is needed in regards to retrospective data issues, pre-specification, multiplicity, and replication. •There is little guidance and little understanding of the operating characteristics of causal inference research based on observational data. Without an understanding of causal inference, it is difficult to appropriately interpret and make healthcare decisions on this data. Detailed guidance would be a beneficial step toward appropriate interpretation of such data. Heterogeneity of Treatment Effects (HTE):•The recognition of the importance of HTE in the draft methods report is instrumental for advancing patient-centered research. The proposed standards are most applicable in subgroup identification and HTE analyses. They are a logical and defensible set of standards, analyses types, and criteria for each of the HTE study types. Descriptive HTE analysis is a novel concept, which will provide more robust data to enable future meta-analyses. •The draft report emphasized the importance of an interaction test to support the evidence of heterogeneity of treatment effects, however, it did not provide practical instructions regarding implementation of the test. PCORI should clearly define the level of evidence required to support a meaningful interpretation of the interaction effect (e.g. subgroup mean, least square mean) as well as best practices for selecting variables of interest and conducting the tests. Furthermore, we recommend that PCORI should state that interactions beyond two-way interactions are to be avoided. The interpretation of interactions beyond two-way becomes more difficult to interpret and implement in practice.•Not all subgroups in HTE are equally important or meaningful for use in treatment decisions and predictive models. This should be acknowledged, and the use of causal diagrams to address this issue should be included. Causal graphs (also known as causal diagrams), such as Directed Acyclic Graphs (DAGs) or Path Diagrams, offer researchers a better insight into causality and confounding by visually representing the a priori assumptions about the underlying biological mechanisms or causal pathways that relate exposures, outcomes, and covariates.•The standards for HTEs list a few examples of variables such as gender, age, co-morbidity, and lifestyle. There should also be consideration for race, ethnicity, and/or regional factors. The FDA recognizes and has done workshops in the past that address the multi-regional HTE. Finally, disease state specific variables should also be included in the standards, and a clear rationale for collection and analysis should be provided.Preventing and Handling Missing Data:•The draft methods report makes a causal statement that clinical studies often are based on restricted populations because the researchers are attempting to avoid missing data which might not be realistic. Clinical studies often have restricted enrollment, but avoidance of missing data is not the primary cause for these design choices. The restricted population enables easier causal interpretation. The standards for missing data should emphasize statistical methods for handling such data and that evaluating the effect of missing data on the interpretation of results should be carefully planned and executed.•Guidance for trials is included in this section, and should incorporate methods related to observational study designs, including database studies. The proposed standard (7.4.4) details reasons for dropout and missing data, and we recommend PCORI include an additional section within this standard that includes possible reasons for missing data when conducting a study using secondary datasets. Issues such as enrollment in claims databases (ICD-9 codes) and data gaps when using electronic health records are critical when applying statistical methods to these data with missing gaps especially as it pertains to the various categories of missing data such as Missing At Random (MAR), Missing Completely At Random (MCAR), and Missing Not At Random (MNAR).Data Networks:•This section captures the critical elements in data networks. It describes the core aspects to consider for quality data from a data network in a high-level comprehensive manner: Data Harmonization (structure and content), Data Privacy, Data Ownership, Intellectual Property, Data Access, Network Establishment and Network Maintenance. The rationale for standards is also logical and compelling.•Standard 7.5.5. outlines the need for standardized terminology encoding in data networks. However, multiple terminology systems for electronic health care databases exist with different terminology such as ICD codes and READ codes. It is unclear if standardized terminology encoding refers to within one terminology system or across different systems. A standard set of terminology would be ideal, but it would be very difficult to achieve broad agreement and use. In addition, mapping standards are needed to convert codes from one system to another and should be considered for future work.•On report page 78, the first sentence of the second paragraph mentions the importance of “quality of data,” but no standard specifically addresses the quality of incoming data to networks. PCORI should consider including a standard that addresses the quality of incoming data.•In addition to the recommended actions on page 79, for future work, we propose PCORI consider promoting a priori industry standards (including structural, syntactic, and semantic standards) for electronic health records to support easier harmonization of data elements across the spectrum of care. For example, the Certification Commission for Health Information Technology (CCHIT) has established certification requirements and standards for electronic health records that PCORI could recommend. http://www.cchit.org

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Adaptive and Bayesian Trial Designs:•The section and supplemental materials are in line with current practice and raise the awareness of adaptive designs and Bayesian methods in general. The supplemental materials are written by some of those on the forefront of adaptive designs and include relevant references, etc. The use of Bayesian trials methods has broad potential and more information about methods for this type of trial should be included in the final report. In addition, we recommend PCORI consider including additional sections and standards for various studies including: newly emerging methods for observational studies primarily evaluating safety issues including high dimensional propensity scores, self control study design, and sequential probability testing.Data Registries:•The justification of the use of data registries is relatively complete. Defining minimal standards for data registries to ensure quality, standardization, and rigor is a step in the right direction. For low-prevalence conditions, registries many be the only opportunity to have meaningful power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches. Randomized trials and other observational studies with the ability to detect interactions with sufficient power that accounts for multiplicity of testing multiple subgroups may be rare, and registries may fill that important need.•Additional benefits of registries include: ability to provide data on populations not typical in RCTs (e.g. children, elderly, minorities, pregnant, and co-morbidity); ability to examine impact of physician practice behaviors on quality of care, prescribing preference, consequences of healthcare decisions, etc.; and usefulness for signal amplification, particularly for rare outcomes. •The proposed minimal standards for data registries tend to be more directed at good practices for designing new data registries. Furthermore, there are three uses of data registries for PCOR studies that call for somewhat different quality considerations and standards: (1)Observational PCOR studies for which a Registry is being newly designed to provide data.(2)Observational PCOR studies which propose to use data elements from an existing ongoing Registry to address a research question.(3)Observational PCOR studies which propose to modify an existing ongoing Registry to address a research questions.We recommend that PCORI consider including standards in the report that account for these types of registries and address related quality requirements for each. •When using an existing registry, the standard should require a feasibility assessment of a proposed registry based on its history of operation (taking into account potential sponsor or clinician biases) and quality to date.•There are numerous centralized computer assisted registry technologies that are based in phone, internet, laptop/computer, home interview, daily diary/experience sampling, and smart phone technologies. These registries are less tied to immediate treatment setting and patient visits, and may offer more control in minimizing follow-up attrition and missing data. In addition, these types of registries may be more suited not only to patient needs, but also collecting patient-reported outcomes. The proposed data registry standards should recognize the emerging use of non-traditional registries and survey panel registry designs versus site-based registries.•A comparison group should be included in the standards for conducting research using data registries.•There is no guidance in the standards for data registries relating to whether registry patients for proposed PCOR can also be simultaneously enrolled in clinical trials or not. Clinical trials impose specific treatment protocols, potentially confounding the registry sample and possibly changing the generalizability to real world outcomes.•When outlining the use of registries, the document should include an emphasis on the advantage of having a larger sample size in a data registry. Particularly for conditions and rare diseases with small patient populations, registries may be the only opportunity to have meaningful statistical power to analyze treatment interactions within subgroups that may be a key to patient-centered approaches. •In standard 8.2.4 regarding the safety and security of data registries, the inclusion of a process for reporting serious adverse events associated with treatments should be included.•Standard 8.2.9 should include the proper methods to address biases and effect modification as well as confounding. Sensitivity analysis to assess the impact of key assumptions in studies should be added. In addition, an issue with many registries is whether individuals can be enrolled multiple times within a site or between sites, and standards should discuss how this can be avoided while protecting patients’ privacy.•Standards for data registries should include information to assist researchers in ensuring the registry population used within a study is comparable to the target population. Studies of Diagnostic Tests:•The report should endorse the Grace Principles (2010), which describes a hierarchy of evidence for observational research on comparative effectiveness that can be used by decision-makers, as well as key elements of good practice including defining research questions and methods a priori; collecting valid, clinically relevant data; analyzing, interpreting and reporting data, including sensitivity analyses and alternative explanations for findings; and conducting these studies in accordance with accepted good practices. Currently, there are no clear standards or requirements for what infromation should be captured or how long it should be stored and made available. http://www.pharmacoepi.org/resources/GRACE_Principles.pdf•The report should differentiate between diagnostic or screening lab tests for biomarkers versus subjective assessment tools where an objective biomarker is not available for patient-reported outcomes (e.g. measurements of pain, depression, or anxiety). •We recommend that PCORI consider discussing the significance of understanding the sensitivity and specificity of diagnostic tests and the importance of having this information readily available for medical personnel and patients alike. This was not included in the draft report. Finally, it would be beneficial to reference in the draft report that the current process of drug and diagnostic tests approval lacks a systematic approach to benefit-risk analysis, leading to inconsistency, lack of transparency, and an inability to challenge or defend decisions- in both the pre-approval and post-marketing setting. The risk-benefit analysis that currently goes into regulatory decisions appears to be ad hoc, informal, and qualitative, without defined and tested algorithm or summary metric that combines benefit and risk data that might permit straightforward quantitative comparative analysis. This is an important gap that needs to be addressed.


Date

Thursday, September 13, 2012

Zip Code

19850

Stakeholder Category

Industry

2. What methodological gaps should be addressed in future reports?

AstraZeneca supports the comments on the PCORI Draft Methodology Report submitted by PhRMA as part of this public comment opportunity, and offer additional comments on specific provisions or standards. • Please consider clarifying how the Committee selected certain methods & the criteria used for not selecting others. • Consider addressing standards (such as for patient registries) that would apply to other types of studies such as observational studies. • Citations are not included in the Report. An appendix including citations would be helpful• Several extensive reports were commissioned and referenced in the draft report. It would be helpful to provide a synopsis of the finding of these research reports highlighted in the report body.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Page 43, table 6.1. The choice of specific randomized clinical trial designs picked out is unusual, and a standard parallel group design is not mentioned. The Delayed Start trial is not commonly used. (See D’Agostino et al., http://www.nejm.org/doi/full/10.1056/NEJMsm0904209). There are many other trial designs where patient allocation or other features differ in different phases that are not mentioned, so it seems odd to pick out the Delayed Start trial design for particular attention in the table.Also in Table 6.1, the definitions for selected study designs for Assessing the Effectiveness of Interventions could use one of many accepted definitions using authoritative textbooks in the fields of pharmacoepidemiology and experimental design. The description of a controlled time trend analysis design as a difference-in-difference or quasi-experimental design is unfamilar and should not infer that all quasi-experimental designs follow this designation (see Cook TD and Campbell DT. Quasi-Experimentation Design & Analysis Issues for Field Settings. Rand McNally 1979.)

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Page 59, section 3.1.4 Please clarify what is meant by “viable”. See: http://www.google.se/url?sa=t&rct=j&q=eu%20choice%20of%20comparator%20&source=web&cd=4&ved=0CFYQFjAD&url=http%3A%2F%2Fwww.eunethta.eu%2Fupload%2FPublic%2520Cons%2FJune%252029%25202012%2FChoice_of_comparator_draft%25204%2520clean%2520final%2520version.doc&ei=KJAoUPbVLYH28gT18IGwDg&usg=AFQjCNH_P9HfEeB-6lh9ZxT9ZKKarNjgLg. Page 59, Recommendation 7.1.2. A priori Specify Plans for Data Analysis – the intended statistical methods and their underlying assumptions should also be included in this plan.Page 61 Standards for Causal Inference Methods. Consider clarifying the need to identify potential confounding factors at the design stage & ensure efforts to collect data on confounding factors during the study.Page 62, paragraph 3 gives the impression that propensity scores are a way to address both the limitations mentioned in paragraph 2 the lack of a clearly articulation of “cause” and being able to adjust only for observed confounders. Although this is corrected on page 63, please consider rewriting paragraph 3, page 62 to minimize confusion.Chapter 7, section on standards for causal inference. Consider including advice on when causal inference may be feasible. For the mentioned methodology (regression, matching, instrumental variables) to work the treatment or exposure groups must overlap to a certain degree, meaning that there must be at least some patients that could be considered to be comparable. If this is not the case the regression analysis will be based on extrapolation and rely heavily on the model assumptions and any matching method will give a very little analysis set to work with. Typical situations that results in comparable groups are when existing treatment guidelines are unclear leading to variability in treatment practice. Also, on Page 64, Causal Inference Standards. Consider rewriting standard 7.2.5 or add a new section that includes assessment of overlap in the distribution of confounders used in a multiple regression model to ascertain the degree of extrapolation involved in the analysis both for each confounder included in the model as well as over all using either propensity scores or the convex hull. This is often forgotten and leads to conclusion being based on very strong model assumptions that go untested if not explored.Page 65, HTE Use of the phrase “traditional clinical research” here may cause confusion. Systematic variability in the treatment effect can be masked (with or without intent) in a study of multiple designs. Page 66 paragraph 4 line 10. Please consider rewording the definition of an interaction test. As worded, the current definition may not be entirely correct as it mentions stratified treatment effects.Page 68 table 7.1 – Row 3 – there should always be at least some prior scientific rationale for treatment effect heterogeneity in a subgroup to minimize false findings, even if the inferential goal is descriptive or exploratory. Data dredging multiple subgroups with no scientific basis is not good statistical practice.Row 4 – There is no reason why exploratory analysis subgroups could not be pre-specified, and it would be good practice to recommend pre-specification. Row 5 – Pre-specifying subgroups in exploratory analysis as recommended above would remove the difficulty in controlling for multiple testing. It should always be reported how many tests were performed, even if error rate control is not formally applied, to aid in interpretation of the likelihood of the findings being true or false.Page 69, 7.3.2 and 7.3.3. Studies should be designed to enable such analyses to be conducted in the most appropriate way. Suggest the recommendations consider design features (eg, stratification) in addition to analysis features.Page 69. 7.3.3. While statistical power of any HTE analyses should be stated up front, it raises the possibility that this will encourage the interpretation of the combination of high power and high p-value as evidence of absence of HTE which is not necessarily the case. Please consider rephrasing. Page 69, 7.3.4: Views differ on the need to always perform an interaction test as part of the investigation of HTE. The appropriate methodology should be linked to the investigational goals of HTE (see 7.3.1). In many cases, an interaction test will have low power and will be difficult to interpret (e.g., if there are many degrees of freedom). In addition, the concept of qualitative versus quantitative interactions should be discussed in this context. Lastly, an important technique would be simulation, but the present report explicitly excludes this discussion. Consider adding a discussion of simulation in future iterations. Page 69, Recommendation 7.3.4 Please clarify whether the interaction test recommended is a global test (i.e. testing for any interaction across all subgroup clusters) or multiple tests for each subgroup cluster. Also, suggest that the confidence intervals need not be 95% (e.g. if an adjustment for multiple testing is applied then they may not be). Forest plots can be very informative in assessing HTE, so it is not clear why these are only recommended for analyses of descriptive intent or why they cannot inform the assessment of HTE. Indeed, relying on statistical testing alone is not recommended, as tests may be underpowered, or may detect differences in subgroup effects that are statistically significant but not clinically relevant.Page 72, 7.4.3: The bolded section header “all forms of single imputation are discouraged” has stronger language than the text that follows. We believe that, as a sensitivity analysis (like in 7.4.5), single imputation has a place so it is inappropriate to unilaterally discourage it.Page 75, second paragraph. Consider adding a discussion of databases containing the same patient multiple times due to switching of health care providers.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Adaptive & Bayesian Design. In future iterations, please consider clarifying how to link research questions to designs rather than providing details on specific designs. Page 81 Standard for Adaptive and Bayesian Designs. Please clarify who should perform the analyses to inform adaptation while the study is ongoing, and who would have access to the results. Knowledge of accruing data and results while a study is ongoing can introduce bias into study conduct and subsequent final analysis and results. It is good practice for the analyses to be done by an independent statistician and reviewed by an independent data monitoring committee (EMEA reflection paper on adaptive designs, 2007). See: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003616.pdf.Consider acknowledging the potential disadvantages of adaptive designs. According to Fleming, “these procedures have several undesirable properties, including lesser statistical efficiency, reduced interpretability of primary outcome results, basing design changes on unreliable interim estimates of efficacy, risks to the integrity and credibility of the trial, loss of flexibility to use emerging results from external sources to alter key design features, and overemphasis of the importance of statistical significance relative to clinical significance.” http://onlinelibrary.wiley.com/doi/10.1002/sim.2641/abstract


Author

American Academy of Family Physicians

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CAFM-AAFP-PCORI-Comments1.docx


Author

Council of Academic Family Medicine

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

See attached document

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CAFM-AAFP-PCORI-Comments.docx


Author

Dominick Esposito & Jeffrey P. Ballou

Date

Thursday, September 13, 2012

Zip Code

8540

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Please see our comments.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Response-to-Draft-Methodology-Report.docx

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see our comments in the attachhed file

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see our comments in the attachhed file


Author

Richard Tannen

Date

Thursday, September 13, 2012

Stakeholder Category

Researcher

2. What methodological gaps should be addressed in future reports?

See my comments in the uploaded document

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-RESPONSE-8-13-2012.doc


Author

Joanne Lynn

Date

Thursday, September 13, 2012

Zip Code

20036

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

What is in the report is consistent, just incomplete in not addressing methods needed to work with system improvement and learning organizations.

2. What methodological gaps should be addressed in future reports?

If not addressed in a revision of this report, these two major gaps deserve development in future reports: 1. Methods from system management fields: e.g., operations research, process improvement, realist evaluation.2. Standardization of data, e.g., through merging CARE/MDS/OASIS into meaningful use as a uniform assessment strategy for Medicare’s elderly population.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Sure – but it mainly addresses problems in standard “frequentist” research design, so it would be both a weighty tome for some (with a straightforward “drug trial” design) and irrelevant for others (with projects that need to use other methods).

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Regarding distribution: Publish a shortened version in Annals of Internal Medicine, like the US Preventive Services Task Force does (and the long version can be on-line there as well as on PCORI’s site). Then it is readily retrieved through any medical literature search engine.Regarding content: I am very concerned that PCORI’s methodology does not seem to have made room for learning organizations, Shewhart statistics, control charts, quality improvement, realist evaluation, or any other method that will be essential in reshaping service delivery to enable patient-centered care. Please see uploaded document.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/JoanneLynnCommentsPCORI_Draft_Methodology_Report.doc

Please provide your comments on Chapter 3 of the Draft Methodology Report.

The "standards" on page 20 apply to an important subset of the research that PCOR covers, not to all. It presumes that "patient-centered outcomes research" has entirely to do with patient decisions in a vacuum - as soon as one has to deal with complex systems and environments, other methods and standards are needed.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

These are generally good, though largely untested. One would expect them to mature some as difficult situations arise.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Again - this is one important approach and good standards for that approach. If one is working on a complex system-anchored problem like "falls in the elderly" or "teenage pregnancy," other criteria and methods will apply.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

The "research categories" in Figure 6.1 again illuminate the narrow nature of the methodology report.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

The methods issues are conventional and appropriate for conventional trials. But each would have exceptions in a dynamic learning organization. For example, one might use a simple tally or count rather than a validated scale. One might develop a better metric during the process.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

These are extremely restrictive design elements for adaptive design. It is true that pre-planning analyses and adaptations on the basis of analyses eliminates some biases. However, it is not an iron-clad rule. Again, have a look at the IOM report on learning organizations, or check on the improvement potential of a model like Intermountain. Do you really intend to denigrate those methods? Consider the implications of the sentence at the end of page 93: "As patients enrolled in RCT tend to be more homogenous, observational studies may be used to augment results from the definitive prospective studies." Taken seriously, this would mean that all of epidemiology is merely useful to augment results from (mostly non-existent) RCTs. It would mean that learning from process control is always second-rate. It elevates RCTs to "definitive," even though most are quite limited. If, in order to help patients overcome addiction, one has to use public information channels, then one cannot have an uncontaminated control group. If, in order to improve capacity for patient self-care, one needs to change the behaviors throughout a hospital, again, one can map the progress and the patients' responses, but one cannot have a matched control group.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Next steps need to include a broader perspective, or an authoritative demarcation of this limited perspective so that CMS, HRSA, ASPE, ARHQ, and private funders are on notice that PCORI will not be using those methods.


Author

AcademyHealth

Date

Thursday, September 13, 2012

Zip Code

20037

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AcademyHealth-PCORI-MC-FINAL1.pdf


Author

Alliance for Regenerative Medicine

Date

Thursday, September 13, 2012

Zip Code

20006

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see general comments uploaded attached document.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Social media and web is best.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ARM-PCORI-comments-sept-2012.docx


Author

Families USA

Date

Thursday, September 13, 2012

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report-Comments.pdf


Author

American Occupational Therapy Association (AOTA)

Date

Thursday, September 13, 2012

Zip Code

20814-3449

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While it's important to strive for methodolgical rigor, some of the populations that PCORI was designed to help will not have high-quality evidence, due to the rare occurrence of the disorder/condition. It will then be important to offer guidance for designing and appraising other research designs, including single-subject designs, if those are the only type of evidence available.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

We would like to see additional information about other designs (see # 2 comments above).

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

In addition to research communities, please disseminate the final report to healthcare associations, including the American Occupational Therapy Association (AOTA) and other allied health associations. We often have researcher groups/communities that can be helpful for efficient dissemination.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments_PCORI_Methodology.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

We are pleased that the purpose of PCORI was emphasized; “to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed…” Another point that could be added is that the best available evidence, as Sackett (2000) urged, should be considered by the patient and professionals in order to make informed decisions. Otherwise, there is a risk that researchers, clinicians, and policy-makers interpret the term “quality” as Level I evidence only, ignoring other levels of evidence. With rare disorders and many conditions seen in rehabilitation, study designs other than the randomized controlled trial can be helpful to inform patients and clinicians’ decisions, but may not always be considered in the development of practice guidelines. We agree that there is a lengthy timeframe between research results to implementation of research into practice. A discussion between researchers, journal editors, and guideline developers could offer ideas for new strategies and communication/dissemination methods, while preserving researchers’ and journals’ rights.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

This chapter offered a comprehensive explanation about the process of the committee’s work.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

This chapter was organized and clearly explained the phases of PCOR. We particularly liked the focus on engaging patients in prioritizing and refining research topics, and look forward to PCORI’s resources and guidance on this critically important process of patient-centered care.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

We agree with PCOR’s approach to not prescribe a particular method of patient engagement, given the types of studies and questions PCOR will cover.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

We wholeheartedly support obtaining patients’ input for research topic selection, but we also support clinicians’ input as well, since they often have clinical questions that researchers have not yet addressed.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

In Table 6.1, Selected Study Designs for Assessing the Effectiveness of Therapeutic Interventions, would Regression Discontinuity Designs be a valid non-randomized design? With its high internal validity, Regression Discontinuity Designs could be useful.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

The section discussing Missing Data is greatly appreciated. There are published intervention studies who have not handled missing data in a valid way, such as including the last collected data from subjects who dropped out in the analyses, citing intention-to-treat analysis. It would be helpful to inform researchers more about options for analyses when there are certain percentages of missing data.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

We appreciate the standards for adaptive and Bayesian trial designs. We agree that registries have the potential to contribute valuable information about PCOR, but we think there should be more forethought to the design and structure of registries. For example, it is critically important to identify the research questions, as much as possible, in advance, so that the data collected can answer these questions. The quality of data’s output is commensurate with the quality of data’s input.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

We look forward to future resources and education/training opportunities to build capacity among researchers to understand, deploy, and improve PCOR methods. Our association is willing to help with informing and disseminating opportunities and resources to patients and clinicians.


Author

Sanofi US

Date

Wednesday, September 12, 2012

Zip Code

8807

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-9_12_12.pdf


Author

Iris Simonis

Date

Wednesday, September 12, 2012

Zip Code

77536

Stakeholder Category

Patient

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Chapter 6 Scope P. 42 Design study is listed as either observational or experimental. I see research being observational or interventional. I feel research is unproven or experimental.Appendix D-2 P. 153#19 Quality outcomes data measured by length of stay, cost, and by providers could also help patients make decisions.Appendix D-3 P. 156#4 Give considerations to over medications of elderly with regard to impact on quality of life, placement, and cost.Appendix D Translation Table Box 1Reference is made to burden to individual, but does not address access to care or research center and financial burden.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Distribute to support organization of various diseases, copies to physician waiting room, social mediaThe terms benefit and harm are used in multiple places in the report. Harm implies negligence with a negative outcome, risk may be a better choice.Report is too long, notice how few patients comments were received. I would suggest a summary report in additional to executive summary and abbreviations list.


Author

Philips Healthcare

Date

Wednesday, September 12, 2012

Zip Code

1810

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Philips-Final-Comments-re-PCORI-draft-report-D0439242.PDF


Author

American College of Rheumatology

Date

Wednesday, September 12, 2012

Zip Code

30319

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ACR-Comment-Letter-to-PCORI-Sep-2012_final.pdf


Date

Wednesday, September 12, 2012


Author

American Academy of Dermatology Association

Date

Wednesday, September 12, 2012

Zip Code

20005

Stakeholder Category

Other

1 Comments

Please read attached comments.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report-AADA-Comment-Final.pdf


Author

Academic Pediatric Association

Date

Wednesday, September 12, 2012

Zip Code

22101

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/APA-PCORI-Comments.pdf


Author

Allergy & Asthma Network Mothers of Asthmatics

Date

Wednesday, September 12, 2012

Zip Code

22101

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Kudos for looking beyond the disease to the person. Policy must be humanized if we are to solve health care cost issues and return decision making back to those of us who live with the outcomes.

2. What methodological gaps should be addressed in future reports?

Would love to answer this questions thoughtfully in a room filled with people who've read the report. There are always gaps and missing links - and this format may not be the most helpful way to do the report justice.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

That said, NHLBI's Asthma Guidelines, if applied in policy and practice, would eliminate asthma death and suffering, put families back to work, keep kids at school and reduce hospitalizations, emergency department visits, urgent care visits and improve productivity. Poorly managed symptoms cost more than getting well and staying well. It's the best example of patient centered care and it was developed by national experts with federal funding lead by NIH/NHLBI/NAEPP. It was adopted by some but not all federal agencies or state public health programs receiving federal funding. And policymakers change over time and are influenced by a variety of factors. AANMA and many other nonprofit organizations consider it the most defining document to revolutionize asthma care in our country and should be continually updated.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

How can I participate? How can Allergy & Asthma Network Mothers of Asthmatics help?

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Patient input is critical. I am unclear what your intent is.


Author

Dev Pathak

Date

Tuesday, September 11, 2012

Zip Code

34203

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

While the report is excellent from a researcher's perspective, I don't believe one area of research from the patient perspective seems to be missing. My suggestion refers to two questions adopted by the committee: 2. “What are my options and what are the potential benefits and harms of those options?” 3. “What can I do to improve the outcomes that are most important to me?” The current report does not address the methodological issues related how the patient receives the information about options and how s/he assimilates the information to make the final decision in selecting the option which would result in improving the outcomes that are important to her/him. There are many steps involved in making the final decision by a patient. And, there are many decision rules used by an individual in selecting an alternatives. Psychological and marketing literature have outlined many decision rules such as compensatory vs. noncompensatory rules. Unfortunately, these decision rules are not tested in health care environment. If only information about options is provided to patients without understanding how patients combine the information about the options to make his/her decision. the information provided may not achieve its goal/s! Understanding the process used by the patient to obtain information about the options AND how s/he assimilates the information to improve his/her decision to achieve his/her goals is as important a methodological consideration as outlining the steps described in the current report to improve the research process!

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

The current approach is fine, but presentations of this report in variety of setting may be useful to popularize the report.


Author

Martha Mihaly

Date

Tuesday, September 11, 2012

Zip Code

20002

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Although I wish more attention had been paid to the roles of cultural competency and ethics when conducting research. NIMH has a huge initiative/commitment to ethics education, and it is an area whose importance is often overlooked in research design and peer review processes. It goes without saying that cultural competence is hugely important, but this report barely mentions the need for it or the criteria for having it.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

Overall, great job!Like others note, it is considerably lengthy; perhaps a summary report would be appropriate for widespread dissemination.


Author

Siemens Healthcare USA

Date

Tuesday, September 11, 2012

Zip Code

19355

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Siemens-Rucker-PCORI-Methodology-Comments-Text.doc


Author

Child & Adolescent Health Measurement Initiative

Date

Monday, September 10, 2012

Zip Code

97239

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Report_CAHMI-Memo.pdf


Author

Anonymous

Date

Monday, September 10, 2012

Stakeholder Category

Clinician

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

No

1 Comments

We have lived through so many attempts to help patients; PORTs; clinical trials; clinical trial registration; FDA standards; quality and safety; AHRQ, now this. Same ol' same ol' not one new idea in the report at all.Most problematic; many on the committee have written and appropriately critiqued methods of science and yet fall victim to the same limitations.The main problem with this report is that it does not advance the science for individuals. As someone who counsels those with tough medical decisions, the information we have is so problematic. I don't see you advancing this in the report at all. You are talking population based science in a patient informed decision making domain. Wrong process; time to go back to the drawing board.

2. What methodological gaps should be addressed in future reports?

First, the report is hard to read; well written but complex and all ove the place. You should be able to make the document be 3 pages long and get to the essesence of what is new and innovative. I could not find anything advancing science for the care of individuals at all. The only method to advance science for individuals is to include all individuals with single options for care; picked by public consensus not researchers. Dabbling in RCTs is a waste of time and money. Time to face it; I am not a heratic; I simulate trials; understand economics; can build models of care and have researched many aspects of care. Our scientific process has failed us; we need a new one for patients as individuals.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

More of the same; why are you doing this??? Are you all so close to the church of science as it stands that we can not stand up and say; we don't have a science of individuals. We need new models; we need to change the process (for example, why do you say the RCT is so good while so many on the committee have pointed out so many short comings. The RCT for populations may be the worse form of science for individuals; the big trial is the worst because if we need a big trial, the differences are so small that it will not matter for any individual making a choice. You have failed in your charge in my view.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Make it 3 pages; fess up that we are lost and ask the public to help you. They could care less about knowing that plavix added to ASA only affects marginally 100 people out of 8-10,000 randomized. Move on.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Nice; but same ol stuff

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Why when they don't like the standards did they promulgate; you have some great thinkers on the committee and seem to have caught a "group think" virus.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Fletcher book on epidemiology is better than this report; why are you going over old ground?

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Population based science aimed at the care of individuals; seems something is wrong here. Too much averaging kills, not liberates patients from care mistakes.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Who is a peer; patients??? Does'nt seem so in your report.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

All wrong in my view; not a single new idea or word in the entire document.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

No more worship at the shrine of bad clinical research designs like the RCT. People who enter trials are not like the one I care for. The entire document should have been about models for promoting all patients in trials.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Start over. I am sorry to write what I have written. I know many on the committee and think they are wonderful people. But I keep rejecting their papers; keep using their papers to show the foibles of science for my patients and yet they are directing this process in the methods that are so suspect. I am dismayed as I hoped for more. Many on the committe are staunch critical evaluators and have pointed out the problems of the present methods of science. Yet, here we are again with your document that looks like the old document; that looks like the old document that......


Author

Michael Gould

Date

Monday, September 10, 2012

Zip Code

91011

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes, but one important thing is unclear--who is the target audience? The document seems to want to be all things to all people. Much of the (very extensive) introductory material seems to be intended for policy makers or other stakeholders, while some sections seem understandable only to researchers. I can't help but thinking that it would better serve each of these important constituencies to have separate documents that are specific to their roles and responsibilities in the research enterprise. As a concrete example, the list of proposed standards in Appendix A is a potpouri of guidance for researchers (e.g. "assess propensity score balance", reviewers (e.g. "protect independence in peer review..."), and funding boards (e.g. "ensure adequate representation of minorities..."). Much of the guidance for researchers is boilerplate (e.g. "develop a formal study protocol", "assess data source accuracy", "a priori specifiy plans for data analysis..."). Parenthetically, one of the standards seems premature ("engage patient informants... in all phases of PCOR"). While I share the PCORI's strong belief in stakeholder involvement, I believe that we are still figuring out how and when stakeholders can have the biggest impact while respecting their limited time. I think that the full text on page 108 is more appropriate when it states the minimum role of stakeholders.

2. What methodological gaps should be addressed in future reports?

A key question with no clear guidance as far as I can tell is: when is it preferred to use one method for causal inference over another (e.g. IV vs. propensity score), as opposed to using both? I favor the latter, with thoughtful comparison and resolution of any differences in results, but I imagine that some reviewers might have strong preferences one way or the other. Neither approach is perfect: propensity scores do not address the possibility of residual confounding, and strong instruments are hard to find.Another (practical) question is how to get meaningful engagement from stakeholders. This is not as easy as it sounds. Patients and other non-researchers are also busy and have limited time and ability to engage in ongoing projects. Compensation is helpful but not necessarily sufficient.The issue of research prioritization is extremely important. With so many worthy questions and so many gaps in evidence, how should PCORI prioritize? The future needs process is a very good start, but I am not sure that systematic review is always necessary or desirable, especially as a starting point. The presence of one or more systematic reviews implies that some evidence already exists, or that the question was more likely to be amenable to systematic review. It is possible, perhaps likely, that some worthy questions would be missed if there is no existing SR. It is almost a logical quandry: how do you identify existing gaps by identifying which gaps have already been filled?

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Yes, but the "translational tool" should be used as a (very rough) guide, rather than prescriptively. I am a little concerned about two sources of potential ambiguity and confusion. First of all, the tool appears to be intended to provide a bridge or link between research questions and methods, rather than a way to translate research findings into practice. Perhaps it would be more appropriate to call it a "bridging tool", or more specifically, a "guide for applying methodology" instead.Second, it would be worth clarifying how the tool should or should not be used. I am among those who worry that an overly prescriptive approach might exclude a worthy study from consideration of funding. At the heart of it all, I am inclined to believe that any research question might be approached any number of ways and that applying diverse methodologies could only increase our confidence about what we think we know. Maybe another way to think about the tool is to be as "permissive" as possible, rather than "restrictive". For example, "the PCORI encourages researchers who are interested in heterogeneity of treatment effects to consider the following methodologies..."

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Overall, this is a very important document that was developed by a very distinguished group. I appreciate the opportunity to provide feedback and hope that some of it was helpful.One technical issue--the embedded links to the appendices and reports (e.g. Value of Information and Research Prioritization on page 33) are not operational.


Author

Neemeka Herbal Clinic and Research Centre

Date

Sunday, September 9, 2012

Zip Code

50200

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Being transparent in developing policies to encourage public registration of all PCORI studies and the sharing of study, protocols, statistical code, and data, there is nothing than a person of any race that honors more than getting to know that he or she is dealing with people who are transparent and much devoted in their work the way PCORI is with their staff in running their work in all ways and means. In fact Neemeka herbal clinic and research centre honors nothing than being transparent in all that it does on patients and in research study, so that patients gets the really high quality herbal medicine findings needed especially on these chronic and multiple chronic diseases that can serve patients on large scale population worldwide.Forming a standing committee within PCORI to recommend appropriate methods for data sharing and to ensure that proper scientific credit is given to those sharing protocols, code data, this shows PCORI’s profile(layout)on how the organization operates is committed with their work by ensuring that proper scientific credit is given to those sharing protocols, code and data. Speeding up implementation of standards in funding announcements, peer, review and other internal process, PCORI staff developed templates for the preparation and review of proposals that incorporate the key elements of the standards; this makes the process of working to become very easy to operate or to run in all channels of work, to be keenly monitored, since some standards apply only to certain types of studies, a portfolio of templates applicable to various study designs should be developed.Support development and use of software for adaptive trials that can stimulate complex designs, broaden experience with adaptive trials for PCORI, perhaps through funding of a cohort of adaptive trials on priority topic areas. Mentor investigators and develop a ‘how to’ guide and a forum to share experiences with adaptive trials, develop a course work and training opportunities for statisticians and other methodologists interested in developing expertise in adaptive trials. Sponsor an institute of medicine committee to develop standards for research on medical tests.

2. What methodological gaps should be addressed in future reports?

Create an infrastructure to support research on patient engagement.-Develop a sample patient engagement plan to demonstrate the key elements required for patient engagement in the research process. -The sample plan should illustrate engagement of both patient informants and study participants.-Systematically collect information about patient engagement methods from PCORI sponsored studies.-Evaluate the effectiveness of patient informant engagement.-Synthesizes results across studies.-Disseminate findings to improve patient engagement in PCOR.-Support training in patient engagement methods for investigators and patient informants.-Improve the patient-reported outcomes( PRO) evidence base by supporting research on methods for assessing measurements properties ( based on qualitative and quantitative evaluations),score interpretability,meangfulness of score changes and strategies for minimizing and interpreting missing PRO data in PCOR.-Evaluate patient dissemination activities and require incorporation in future research of relevant learning from this evaluation.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Should ensure that a sufficient number and range of topics are considered, before topics for research findings are selected. Engagement of multiple stakeholders, especially patients, involving patients and other stakeholders in developing questions for systematic review, researchers can compare what people want and need to know with what is and is not known. As a result, systematic reviews can identify gaps in knowledge that underlie UN certainty among patients and clinicians. Sometimes systematic reviews can generate new questions. For example, a pooled analysis of several studies can reveal an important finding that was not evident in the individual studies.Value of information analysis may be used to identify questions that have the greatest potential to improve population health by considering uncertainty in the health benefits and risks associated with alternative treatment choices, the ability of research findings to alter uncertainty, and the resulting care decisions. Reviews process identifies those proposals most likely to fulfill PCORI’s objective and agenda.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Neemeka herbal clinic and research centre the personnel investigator would like to upload full findings that are in the letter of support of Kenya Medical Research Institute (Centre for Traditional Medicine and Drug Research, P.O.BOX 54840-00200 NAIROBI, Kenya) Kemri and also a letter of support Kenyatta University office of the chairman Department of Health Sciences, Centre for Complementary Medicine and Biotechnology P.O Box 43844 Nairobi Kenya. Neemeka herbal clinic and research centre plus entire staff is in atonally research of herbs that are medicine for these chronic diseases. The centre is working round the clock in ensuring that they are in research for high quality findings required for these chronic diseases, in near future the solution will be achieved. In the letter of Kenya Medical Research Institute (Kemri) page2, second paragraph states that search of available literature revealed no published report on the toxicity of s.longepedunculata in humans despite the wide-spread use of the plant in various herbal remedies. Considering the potential toxicity of s.longepedunculata, herbal practitioners should be educated on this especially when they recommend this plant as part of complex regimen in the long term management of chronic illness.Neemeka herbal clinic and research centre would like to appreciate and thank the entire staff of PCORI about the good work they are doing for patients especially those with chronic and multiple chronic diseases worldwide, working hand in hand in all means and ways in doing more research and being honest in submitting the really high quality findings needed for these chronic diseases that gathers large population worldwide. Neemeka herbal clinic and research centre also reflects thanks giving to anybody whose main objective is putting patients first by funding researchers in all means and ways as per research study is concerned, so that high quality findings is achieved for these chronic diseases.Neemeka herbal clinic and research centre is doing more research on Diseases like;-Heart diseases-Diseases of liver-Kidney problems-A.I.D.S Virus-Diabetes-Lungs problems-Cancer-

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/DOC-1.jpg

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Neemeka herbal clinic and research centre comments that it would be very much competitive if methodology Board of Committee Governors can look for a way of or any means of awarding those researchers and committee members who are much devoted with their work in researching the most high quality findings that are medicine needed for these chronic and multiple diseases that gathers large population worldwide. By so doing it will stiff the competition in researching for high quality findings, they can also be awarded with certificates. By raising a bar for researchers, it would be better if the bar of researchers goes by the most quality findings for chronic and multiple chronic diseases, with their awarding price.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Neemeka herbal clinic and research centre fully supports the methodology committee by developing the recommended standards, by forming a translation table, methodological standards for research and a proposal of activities for enforcing the methodological standards. These puts in peoples mind that methodology committee puts much and are very much committed in considering their in all ways as per every research findings is required in orderly manner thus making every findings gotten easily in its last stage, by fallowing proper channels , arrangements ready for patients.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Providing accurate studies for medical will ensure proper running of studies in research. Also designing in informing decision making, by indicating considerable doubts about the existence of the effect, this shows how keen, accurate and transparent in their research. The ways organization lays their work shows the smoothness and clearness how the organization is set up to operate their work.Experience in their fields leads to appreciation of the potential pitfalls in practice and in theory thus making methods to evolve and improve overtime. The way methodology committee developed the recommended standard in becoming more effective in ways of sponsors in research to articulate standards for the conducts of research they found .This shows how much willing the organization is set in assisting as much as they can in all ways and means in their research by struggling to get the really high quality findings so that they can assist in arresting these chronic and multiple chronic diseases worldwide CORI authorizing legislation which tasked the methodology committee in developing and promulgating such standards shows how keen and well organized PCORI is in the organization. These is true about PCORI by going a wider range of research studies used to inform clinical decision being governed by less formal approaches such as a peer review of research proposals and reports submitted to scientific journals.Neemeka herbal clinic and research centre plus its entire staff indeed appreciate the work the methodology committee Board of Governors are doing by ensuring that each and every work takes its course as research is cornered .

Please provide your comments on Chapter 4 of the Draft Methodology Report.

To produce information that is meaningful and useful to people when making specific health decision in research proposals and protocols is a very matter this shows how PCORI is much and more in minding and considering patients problems as their first priorities in dealing and struggling to get the right information plus the findings in their long run. Also engaging patients in formulating research questions, defining essential characteristics of study participants is a very wisdom approach as he who wears the shoes knows where the shoes pinches, especially those patients, with critical conditions. Neemeka herbal clinic and research centre, plus the entire staff are together with PCORI organization in working hand in hand in making sure and ensuring high quality findings in research is achieved, especially for these chronic and multiple chronic diseases in order to save their live worldwide. Neemeka herbal clinic and research centre is working and co-operating with other specialists doctors here in Kenya in doing more research in herbs for these chronic diseases so that the proper findings is gotten to save those who are victims worldwide.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Establishing a specific research agenda is a core duty of PCORI unless the mismatch between current research priorities and information needs of patients and clinicians is addressed.Neemeka herbal clinic and research centre fully supports a core duty of PCORI unless the mismatch between current research priorities and information needed of patients and clinicians is addressed, this is naked truth as the two major points states. Neemeka herbal clinic and research centre supports two major statements of PCORI in research especially in these chronic and multiple chronic diseases sometimes, it depends on the source virus that brought the disease in existing .Most of varieties of the diseases are the products of original diseases virus e.g. a disease like syphilis, same syphilis affects aorta to course heart disease, same syphilis brings about arteriosclerotic problems thus causing highly blood pressure ,same syphilis affects kidney causing cancer, same syphilis affects lung and liver, same syphilis cause diabetes, same syphilis affects brain causing cerebral malaria same syphilis cause epilepsy.Neemeka herbal clinic and research centre comments that anybody who is affected with syphilitic germs when not given proper treatments becomes easier to be affected with A.I.D.S.That patient who has syphilitic germs for a long period of time becomes difficult if not impossible to be treated once he or she is a victim of A.I.D.S virus.NB-In fact for any patient to be considered well, as a victim of A.I.D.S, must have syphilitic germs to speed up chemical reaction in patient’s body, thus making the patient very weak and suffer slowly taking a long period of years in order for he or she to die. Anybody who has syphilis A.I.D.S and has no any syphilitic germs becomes a carrier to affect others but he or she doesn’t die easily unless he or she is affected with syphilis.NB-Dr. Swartout states that the hope of a cure for cancer of liver is not even equal to the hope of a cure for cancer of lung, but if cancers in other parts of the body, which are usually sources of liver cancers are properly treated, most liver cancers would never develop. Page 863 in the book of New Modern Medical Counselor published 1943.Dr Swartout on page 807 second paragraph middle sentences, Dr. Swartout has given percentages 7-10% of all patients being cared in the hospitals are due to syphilis. Neemeka herbal clinic and research centre would like to support Dr. Swartout percentages even adding him more percentages up to 35% not 7-10% as he estimated.NB- Neemeka herbal clinic and research centre states that, if these inlet diseases, that are the sources of these complex, chronic and multiple chronic diseases are properly treated, then there will be no way of these chronic and multiple chronic diseases worldwide.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

These shows a clear guidance regarding the selection of appropriate research designs for specific research questions and also it avoids studies that are inappropriate for research question and could direct researchers to designs that reach the right answers sooner, thus improving the efficiency of research. Research designs clearly match the question patient s and their Health care a devisers, consider important, research results that are more readily accepted and implemented. A translation table helps to build a comprehensive research program, recognizing and balancing the inherent tradeoffs of each study design and analytical methodology developed asset of principles and a translation framework ,which refined by applying case studies contributed by committee members and members of the public. This shows how PCORI is devoted in designing their work to appear descent in research and proper arrangement so that it’s easily understood by everyone at the end stage of their high quality findings. Translation table makes information more informative and less complex. Translation Framework provides the theoretical underpinning and organizing structure for the translation table. It identify the range of appropriate research designs and analytic approaches to answer specific research questions also guides the user in making choices in study design and analysis methods based on current scientific knowledge in research categories and corresponding translation table.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

It identifies and assesses participant’s subgroups, selects appropriate interventions and comparators. It also assesses data source adequacy, specifying plans for data analysis that correspond to major aims, documents validated scales and tests. It uses sensitivity to analysis and to determine the impact of key assumptions, providing sufficient information in reports to allow assessment of the study internal and external validity. This is approval that PCORI is very much devoted in each and every work it does, by putting everything in transparent and in accurate standards in its final stages so that they are clearly understood in study internal and external validity to everybody.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Randomized trials can provide the strongest evidence about the comparative effectiveness of different treatments on intervention. Adaptive trials allow changes to be made to a study while it’s ongoing. Examples of adaptations include changing what proportion of patients are randomized to which intervention group ,altering the sample size, changing the eligibility criteria, dropping or adding comparison arms, changing endpoints and stopping early. Rather than waiting until the end of the study period to implement changes, the changes are planned for as part of the trial design and executed based on the analyses conducted during the trial. Neemeka herbal clinic and research centre supports randomized clinical trials indeed. Randomized clinical trials have potential to maintain many of the disadvantages of randomized clinical trials, while minimizing some of the disadvantages. Adaptive trials can sometimes provide results faster, and can also increase the relevance of trial results by adjusting both the composition of patient groups and treatments based on interim results and clinical questions. The flexibility and efficiency that are gained in adaptive trials have to be balanced with the risk that such trials typically require a longer design period and involve more logistical complexity. Also there are relatively few statisticians with expertise or experience in designing or carrying out such trial research.Neemeka herbal clinic and research centre supports randomized clinical trials, but should only be exercised by experienced professional experts in the field of research and treatment as this field carries weight in research study of the findings in its final stage and treatments. For example we have some samples of herbal medicines that can treat almost 50 different kinds of diseases depending on how experienced and professional you are in mixing this composition of herbs from different plants for you to get a high quality findings in your study of these complex diseases and in treatment of patients. Dosage will depend on how strong or weak the patient is, duration of the disease in the patient’s body, the kind of a disease you are dealing on, type of composition of herbs you have come up with, how strong are they in your randomized clinical trial research study? Also you need to put in mind, apart from treatment what other side effect these composition has in your randomized clinical trial, in your final stage of your study findings? If it shows side effect or if it has the side affect, what kind of composition in your randomized clinical study, you will use to end this side effect? What composition of herbs in your randomized clinical trial you need to use in your research study to solve this problem of side effect?Neemeka herbal clinic and research centre comments that Randomized Clinical Trials is the best system in research study, if it’s being operated by philosophic and pharmacological Doctors only as this field carries weight and its very technical to operate its research study and research findings.NB-Neemeka herbal clinic and research centre comments that, it’s through Randomized Clinic Trials in study of research herbal medicine of varies plants, that are medicine for these chronic and multiple chronic diseases that will be a solution for these diseases worldwide. Neemeka herbal clinic and research centre is not in dream to comment that the only solution for these chronic and multiple chronic diseases is alternative medicine.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Neemeka herbal clinic and research centre would like to comment that anything discussed publically with professionals in every step as research is concerned after submitting all the findings from different destinations, especially on these chronic and multiple chronic diseases or any other findings that can attribute to problems of patients that is how chain operates. Neemeka herbal clinic and research centre supports public discussion as two heads make light work. Let every professional air out his or her comment freely without any restriction. We learn from one another and learning has no end. Neemeka Herbal Clinic and Research Centre. Thanking you all..


Author

Anonymous

Date

Friday, September 14, 2012

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Thank you for allowing public input on this very significant initiative. The report is extensive and informative, however I believe a summary or supplement guide targeting key areas is needed.Following are recommendations regarding distribution of the final report:Press ReleaseSocial MediaMass media (print, tv, etc)

2. What methodological gaps should be addressed in future reports?

How increased research standards will impact the recruitment of primary research candidates,and what mechanisms will be put in place to resolve this significant issue. For example, the barrier of trust is a major factor when recruiting research candidates, specifically in underserved populations. Transparency is helpful, however it does not guarantee consumers and/or caregivers will "buy" into participating.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes


Author

Cancer Policy Group

Date

Friday, September 14, 2012

Zip Code

20902

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

See attachment

2. What methodological gaps should be addressed in future reports?

see attachment

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

See attachment

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments-of-PCORI-Methodology-Standards.pdf


Author

Global Health Sciences & Services, LLC

Date

Friday, September 14, 2012

Zip Code

19027

Stakeholder Category

Other

2. What methodological gaps should be addressed in future reports?

Researcher intent should be scrutinized. How does the researcher identify and relate to the population(s) he/she studies. How will PCORI ensure that populations of interest are not exploited?


Author

Global Health Sciences & Services, LLC

Date

Friday, September 14, 2012

Zip Code

19027

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

The standards and recommended actions are well thought out and articulated.

2. What methodological gaps should be addressed in future reports?

When using "race" and "ethnicity" as population identifiers researchers must clearly state how they define such populations. Merely stating the population is self-identified is inadequate, as the racial and ethnic categories traditionally employed are by definition ambiguous and based on the perceptions of others.Researchers must clearly demarcate among heterogenous as well as homogeneous populations and justify their rationale. For instance, "Black/African American" does not contextually define the target population. Too often the standardized racial and ethnic categories as provided by the OMB's Directive No. 15 are utilized as definitive and clear.Researchers should disclose previous methods used when engaging target populations and provide past outcomes utilizing those stated methods. If said researcher has not previously been successful engaging the target population, he/she must state how the current application addresses and overcomes past failures.PCORI needs to ensure that efforts to engage the community in the development, implementation and dissemination phases of the research are not mere window dressing. Letters of support from community partners must reflect their full knowledge of and participation in the research.There must be a way to keep researchers who are merely "checking the box" regarding positive or successful community impact can be audited.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

The final report should be a media (e.g., radio, tv, internet, newspaper, etc.) event. Too often research is treated as an esoteric realm not to be understood by the community or individual patients. As much as possible, PCORI should remain open and transparent while actively informing the public of its objectives and outcomes.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Oddly enough, the Introduction makes no mention of patient, surrogate decision makers or other stakeholders involvement as members of the Methodology Committee in the preparation of this report.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

4.1.1. Engaging Patient Informants, Persons Representative of the Population of Interest, in All Phases of PCOR. How will PCORI measure patient involvement?

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Box 5.1 Research Prioritization Factors. How are preventive studies included in the priorities?

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Pg 49, Step 1: par 2. "This process may ultimately result in a set of studies all on the same question but with slightly different goals." PCORI should also entertain radically different goals as the experience, cultural relevance and perspective of the research may vary greatly. The phrase "too innovative" has been employed by reviewers who are used to traditional methods and ways of thinking.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Pg 59, General and Crosscutting Methods for All PCOR. 3.1.3. Identify and Assess Participant Subgroups. Be aware that the term "subgroup" has a negative connotation for some populations. Definition of the word "subgroup" must be clearly provided.Segments of the population within a population may be categorized as "subgroups" within like groups as long as comparisons are not made across groups. (i.e., not Black when compared to White, but rather urban Black when compared to rural Black). Keeping in mind that the historically used racial and ethnic terms are obsolete.Pg 72, 7.4.4 Record All Reasons for Dropout and Missing Data, and Account for All Patients in Reports. Researchers must account for significant deviations from proposed recruitment numbers and actual recruitment numbers. Merely tracking just those recruited may mask hidden problems with recruitment.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Pg 75. Data Networks as Research-fascilitating Infrastructures. Par 2. Key elements of a data network... Care must be taken to ensure that the parameters for data collected are contextually relevant and identical. This is specifically so when data collection is international. All fields must be comparable.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

The PCORI Draft Methodology Report is excellent and will be moreso as it evolves. I encourage PCORI to entertain proposed methodology from non-traditional researchers, patients, decision-makers, providers et al. For decades health disparities have persisted in spite of past and ongoing research efforts. Some answers may lie in research conducted through methods not previously deemed traditional or relevant, but get to the heart of problems from a different vantage point.


Date

Friday, September 14, 2012

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes and the AGS applauds PCORI’s efforts to improve the quality of evidence available to guide informed decisions. However, we believe that adherence to these higher standards will demand time and resources (e.g., to engage patient stakeholders throughout the research process, develop more detailed protocols, develop dissemination plans, measure additional outcomes, more fully report results, upload tools and statistical code for sharing, etc.). In an era that is marked by 1) increasing regulation and unfunded mandates and 2) diminishing funding support for research, we suggest that the overview chapter directly acknowledge this issue. It would be important to highlight support mechanisms in place or under development to assist investigators in complying with these higher standards in a manner that is cost-efficient and minimizes burden (could refer to paragraph on Page 95). Furthermore, PCORI would be justified in asserting that adherence to higher standards may mean that fewer studies can be undertaken, but the public interest is better served by targeted, well-done studies that answer key questions than by many studies of questionable quality. In any case, we recommend this report explicitly recognize that adherence to some, but not all, of these standards will require increased human resources and thus justifiably increase the cost of planning and conducting each PCORI study.

2. What methodological gaps should be addressed in future reports?

We appreciate that persons living with multiple chronic conditions and other vulnerable populations are mentioned. However, older adults are not specifically mentioned, and we believe that while there is overlap with many of the vulnerable populations included, older adults should be explicitly stated. It is important to differentiate the conditions of a 50 year old with that of an 80 year old.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

5. Please enter any additional comments or concerns here.

Overall, we believe this is a well done report. It outlines higher standards for Patient Centered Outcomes Research, and is especially inclusive of those who are living with multiple chronic conditions. As stated above, the AGS would like to reiterate the importance of specifically including older adults in the report, as this ever-increasing population must be clearly defined and properly addressed.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

1) The 2nd paragraph begins with a sentence “understanding all the contributors to uncertainty often takes years of development and application of a method…” and continues to explain that best methods evolve and improve over time and thus, standards need to be in place to ensure that best methodological practices are applied. We agree, but would avoid the implication that we ever understand “all the contributors to uncertainty.” Also, there should be acknowledgement here that new methodologies will and should continue to develop or evolve and the Committee has considered this in putting forth the current Standards (as discussed further on pages 16/17).2) The Box “A Thought Experiment” (page 16) is well done. However, due to its placement on the page, we read it before the section, “Balancing the Aims of PCOR” and it seemed like a non sequitur. We recommend titling the Box “Balancing the Aims of PCOR: A Thought Experiment” to more clearly tie it to the appropriate section of text. We also recommend explicitly stating in the box that the described study addresses one Aim of PCOR, namely to understand how treatment choices perform in “real life,” but fails to address a second Aim, which is to understand how well a treatment can work.3) We very much like the four categories detailed in Table 3.1. The questions are very intuitive, and we find this to be a helpful way to organize the approach to the 60 standards. It is a manageable context for the list in Appendix A. 4) It might be helpful if Table 3.1 denoted who would be undertaking the various activities (i.e., investigators, reviewers, funding organizations, etc.). Alternatively, it will be helpful for the text to note that some activities are more applicable to certain stakeholders in PCOR.5) Page 22 – Section 3.1.5 Measure Outcomes that People in the Population of Interest Notice and Care About. It seems that “treatment burden or cost” should be added to the list of examples of such outcomes.6) The Standards for Formulating Research Questions feel “tagged on” in this chapter. Most of Chapter 3 provides an overview of all 60 standards and the process/considerations that went into developing them, yet near the end of Chapter 3 we are presented with the 5 standards for formulating research questions. Then the chapter concludes with a section on transparency that is once again, appropriate for an overview.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

1) We think that the interest in patient engagement is well done – but would like to see mentioned that engaging a handful of patients in ones study design and oversight does not necessarily represent the heterogeneity of patients with whatever condition(s) is being studied. For example, one may see a particularly motivated, interested patient signing up to be on an advisory board. Thus, studies that look at large samples of people with a condition and assess variability in preferences are essential, as well as other modes of engagement of patients in research that will practically be on a smaller scale in terms of numbers. 2) On page 27, there is a question that is essentially a test for a patient centered outcome. Were it to be the only thing that changed, patients would be willing to undergo a treatment with associated risk cost or inconvenience. This would be true of treatments that ameliorated symptoms or prevented morbidity and mortality. It would not be true of treatments that lowered blood pressure, improved cardiac output improved bone density, or the like, without improving the quality or increasing the length of life.” This comes directly from the users guide to the literature. The full definition from the Journal of the American Medical Association (JAMA) is “Outcomes that patients value directly.” This is in contrast to surrogate, substitute, or physiologic outcomes that clinicians may consider important. One way of thinking about a patient-important outcome is that, were it to be the only thing that changed, patients would be willing to undergo a treatment with associated risk, cost, or inconvenience. This would be true of treatments that ameliorated symptoms or prevented morbidity or mortality. It would not be true of treatments that lowered blood pressure, improved cardiac output, improved bone density, or the like, without improving the quality or increasing the length of life.” We believe this definition provides more clarity, though, it still has some problems (see below). The definition clearly focuses on things that matter to most patients. But it vastly oversimplifies decision-making, without needing to, and we think it should be refined. Examples include: Some patients may value reducing their future risk of a stroke by lowering blood pressure. We think it depends on how tightly linked a surrogate is, and the person’s own values. What we find confusing about the definition is the idea that it assumes patients would value the risks, costs and inconvenience similarly, and arrive at similar decisions about whether or not to pursue a treatment with the same information. It also does not acknowledge the fact that it is not what the outcome is that effects decision making, but it is how effective the treatment is and over what period of time. For example, if a treatment reduces risk of having a hip fracture by 0.01% absolute risk reduction over 5 years, even though a hip fracture is a patient-important outcome, we are not sure we would choose the treatment. Even though this is a somewhat established definition, we suggest considering refining it.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

This chapter outlined the charter of PCORI to set a research agenda and incorporate patient perspectives into the decision-making process as well as balance the needs for evidence with the impact that the evidence could have. Basing these important priority decisions on systematic reviews and gap analysis sounds efficient for those areas in which evidence exits. Additionally, the rationale for doing systematic reviews to improve efficiency in research conduction as well as transparency and accountability in funding priority decisions is good. However, this approach may be biased towards research that is already established or ongoing and may deflect from new or neglected areas of research (e.g. studies in older adults, studies of rehabilitation interventions, etc.). There are so many areas in which evidence does not yet exist and therefore the ability to conduct a systematic review should not be requisite. Rather, the gap analysis should be the core, with assessments of the need for knowledge in a particular area, whether or not there is a pre-existing body of research. Another issue with systematic reviews is that basing decisions on those would avoid taking risks and potentially lead to PCORI funding safe, but very small incremental advancements beyond current knowledge. In addition, the basic science and mechanistic research do not appear to be addressed in this priority statement. This is likely due to the difficulty in dealing with the issue of how science moves and how to balance research with immediate results versus advancing basic understandings of science. The focus on quick results that will have a clinically meaningful impact is certainly a valuable goal. However, science benefits from asking more proximal questions regarding causality and mechanisms. Thus, there is a need to include basic science and translational science in the priorities. The recommendation to develop novel methods of obtaining patient input in research topic generation sounds good, but there is a lack of detail regarding how patients will be appropriately selected or targeted. Will selection involve educational background, age, past medical history, family history or some other factors? How will the needs of older adults be represented? How will geriatric stakeholders, who may have difficulty with travel or mobility, be included? Who will represent prevention? Patients do not benefit from prevention because it is too late by the time that they are patients. Who will represent wellness or healthy aging? A recommendation is to use Value of Information (VOI) methods to determine research prioritization. This sounds like a good approach, but we do not have experience with this method. This chapter on Prioritizing PCOR may not be the location that the PCORI Board of Governors considered knowledge transfer. However, there is a need to address the issues surrounding how will the knowledge get out to the stakeholders and be utilized. Overall, Chapter 5 indicates a reasonable start to approaching the question of how to prioritize research that should be funded. However, there is a lack of detail regarding specifically how this will be approached, possibly due to the lack of evidence regarding how this should be approached. The authors demonstrate this on top of page 34, where they recommend "support for empirical research" to learn how to select research topics (i.e. research on what to research). PCORI could look towards HealthyPeople 2020 or other work that have been considered in great detail by public health researchers. An examination of which problems, facing older adults, are in need of answers and are feasible and which public health work has been completed could be valuable in assigning priorities to research topics pertinent to older adults.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

This chapter begins by saying that the randomized clinical trial (RCT) is the most effective process. However, there is no mention of those groups who are under-represented – namely, older adults with multiple chronic conditions. We believe that the chapter should note how healthier volunteers may not generalize to those with multiple medications/conditions, as seen with many older adults. Chapter 7 also discusses observational studies as an alternative and how these studies are limited and confounding. It mentions that propensity scores are often used but do not address what is unmeasured and confounding. We suggest reviewing the following papers of relevance: • Brookhart MA, Stürmer T, Glynn RJ, Rassen J, Schneeweiss S. Confounding control in healthcare database research: challenges and potential approaches. Med Care. 2010;48(6 suppl):S114–S120. • Bosco JLF, Silliman RA, Thwin SS, Geiger AM, Buist DS, Prout MN, Yood MU, Haque R, Wei F, Lash TL. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. J Clin Epidemiol 2009 May 19. PMID: 19457638 PCORI should also consider adding that confounding issues are not as important in comparative safety studies as they are in comparative effectiveness studies where confounding by indication or disease severity comes in to play. We suggest the additional of these citations:- Klungel OH, Edwin P, Martens EP, Psaty BM, Grobbee DE, Sullivan SD, Stricker BH, Leufkens HG, de Boer A. Methods to assess intended effects of drug treatment in observational studies. J Clin Epidemiol 2004;57(12):1223-31. And Psaty BM, Koepsell TD, Lin D, Weiss NS, Siscovick DS, Rosendaal FR, Pahor M, Furberg CD. Assessment and control for confounding by indication in observational studies. J Am Geriatr Soc 1999;47(6):749-54. And Tinetti ME, Studenski SA. Comparative effectiveness research and patients with multiple chronic conditions. N Eng J Med 2011;364:2478-81.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

We believe this chapter is well constructed and that the description of the issues and the recommendations made are very reasonable. Important outcomes that we want to emphasize are functional outcomes, which are acknowledged: “Identify and select outcomes the population of interest notices and cares about (e.g., survival, function, symptoms, health-related quality of life) and that inform an identified health decision.”


Date

Friday, September 14, 2012

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes and the AGS applauds PCORI’s efforts to improve the quality of evidence available to guide informed decisions. However, we believe that adherence to these higher standards will demand time and resources (e.g., to engage patient stakeholders throughout the research process, develop more detailed protocols, develop dissemination plans, measure additional outcomes, more fully report results, upload tools and statistical code for sharing, etc.). In an era that is marked by 1) increasing regulation and unfunded mandates and 2) diminishing funding support for research, we suggest that the overview chapter directly acknowledge this issue. It would be important to highlight support mechanisms in place or under development to assist investigators in complying with these higher standards in a manner that is cost-efficient and minimizes burden (could refer to paragraph on Page 95). Furthermore, PCORI would be justified in asserting that adherence to higher standards may mean that fewer studies can be undertaken, but the public interest is better served by targeted, well-done studies that answer key questions than by many studies of questionable quality. In any case, we recommend this report explicitly recognize that adherence to some, but not all, of these standards will require increased human resources and thus justifiably increase the cost of planning and conducting each PCORI study.

2. What methodological gaps should be addressed in future reports?

We appreciate that persons living with multiple chronic conditions and other vulnerable populations are mentioned. However, older adults are not specifically mentioned, and we believe that while there is overlap with many of the vulnerable populations included, older adults should be explicitly stated. It is important to differentiate the conditions of a 50 year old with that of an 80 year old.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

5. Please enter any additional comments or concerns here.

Overall, we believe this is a well done report. It outlines higher standards for Patient Centered Outcomes Research, and is especially inclusive of those who are living with multiple chronic conditions. As stated above, the AGS would like to reiterate the importance of specifically including older adults in the report, as this ever-increasing population must be clearly defined and properly addressed.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

1) The 2nd paragraph begins with a sentence “understanding all the contributors to uncertainty often takes years of development and application of a method…” and continues to explain that best methods evolve and improve over time and thus, standards need to be in place to ensure that best methodological practices are applied. We agree, but would avoid the implication that we ever understand “all the contributors to uncertainty.” Also, there should be acknowledgement here that new methodologies will and should continue to develop or evolve and the Committee has considered this in putting forth the current Standards (as discussed further on pages 16/17).2) The Box “A Thought Experiment” (page 16) is well done. However, due to its placement on the page, we read it before the section, “Balancing the Aims of PCOR” and it seemed like a non sequitur. We recommend titling the Box “Balancing the Aims of PCOR: A Thought Experiment” to more clearly tie it to the appropriate section of text. We also recommend explicitly stating in the box that the described study addresses one Aim of PCOR, namely to understand how treatment choices perform in “real life,” but fails to address a second Aim, which is to understand how well a treatment can work.3) We very much like the four categories detailed in Table 3.1. The questions are very intuitive, and we find this to be a helpful way to organize the approach to the 60 standards. It is a manageable context for the list in Appendix A. 4) It might be helpful if Table 3.1 denoted who would be undertaking the various activities (i.e., investigators, reviewers, funding organizations, etc.). Alternatively, it will be helpful for the text to note that some activities are more applicable to certain stakeholders in PCOR.5) Page 22 – Section 3.1.5 Measure Outcomes that People in the Population of Interest Notice and Care About. It seems that “treatment burden or cost” should be added to the list of examples of such outcomes.6) The Standards for Formulating Research Questions feel “tagged on” in this chapter. Most of Chapter 3 provides an overview of all 60 standards and the process/considerations that went into developing them, yet near the end of Chapter 3 we are presented with the 5 standards for formulating research questions. Then the chapter concludes with a section on transparency that is once again, appropriate for an overview.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

1) We think that the interest in patient engagement is well done – but would like to see mentioned that engaging a handful of patients in ones study design and oversight does not necessarily represent the heterogeneity of patients with whatever condition(s) is being studied. For example, one may see a particularly motivated, interested patient signing up to be on an advisory board. Thus, studies that look at large samples of people with a condition and assess variability in preferences are essential, as well as other modes of engagement of patients in research that will practically be on a smaller scale in terms of numbers. 2) On page 27, there is a question that is essentially a test for a patient centered outcome. Were it to be the only thing that changed, patients would be willing to undergo a treatment with associated risk cost or inconvenience. This would be true of treatments that ameliorated symptoms or prevented morbidity and mortality. It would not be true of treatments that lowered blood pressure, improved cardiac output improved bone density, or the like, without improving the quality or increasing the length of life.” This comes directly from the users guide to the literature. The full definition from the Journal of the American Medical Association (JAMA) is “Outcomes that patients value directly.” This is in contrast to surrogate, substitute, or physiologic outcomes that clinicians may consider important. One way of thinking about a patient-important outcome is that, were it to be the only thing that changed, patients would be willing to undergo a treatment with associated risk, cost, or inconvenience. This would be true of treatments that ameliorated symptoms or prevented morbidity or mortality. It would not be true of treatments that lowered blood pressure, improved cardiac output, improved bone density, or the like, without improving the quality or increasing the length of life.” We believe this definition provides more clarity, though, it still has some problems (see below). The definition clearly focuses on things that matter to most patients. But it vastly oversimplifies decision-making, without needing to, and we think it should be refined. Examples include: Some patients may value reducing their future risk of a stroke by lowering blood pressure. We think it depends on how tightly linked a surrogate is, and the person’s own values. What we find confusing about the definition is the idea that it assumes patients would value the risks, costs and inconvenience similarly, and arrive at similar decisions about whether or not to pursue a treatment with the same information. It also does not acknowledge the fact that it is not what the outcome is that effects decision making, but it is how effective the treatment is and over what period of time. For example, if a treatment reduces risk of having a hip fracture by 0.01% absolute risk reduction over 5 years, even though a hip fracture is a patient-important outcome, we are not sure we would choose the treatment. Even though this is a somewhat established definition, we suggest considering refining it.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

This chapter outlined the charter of PCORI to set a research agenda and incorporate patient perspectives into the decision-making process as well as balance the needs for evidence with the impact that the evidence could have. Basing these important priority decisions on systematic reviews and gap analysis sounds efficient for those areas in which evidence exits. Additionally, the rationale for doing systematic reviews to improve efficiency in research conduction as well as transparency and accountability in funding priority decisions is good. However, this approach may be biased towards research that is already established or ongoing and may deflect from new or neglected areas of research (e.g. studies in older adults, studies of rehabilitation interventions, etc.). There are so many areas in which evidence does not yet exist and therefore the ability to conduct a systematic review should not be requisite. Rather, the gap analysis should be the core, with assessments of the need for knowledge in a particular area, whether or not there is a pre-existing body of research. Another issue with systematic reviews is that basing decisions on those would avoid taking risks and potentially lead to PCORI funding safe, but very small incremental advancements beyond current knowledge. In addition, the basic science and mechanistic research do not appear to be addressed in this priority statement. This is likely due to the difficulty in dealing with the issue of how science moves and how to balance research with immediate results versus advancing basic understandings of science. The focus on quick results that will have a clinically meaningful impact is certainly a valuable goal. However, science benefits from asking more proximal questions regarding causality and mechanisms. Thus, there is a need to include basic science and translational science in the priorities. The recommendation to develop novel methods of obtaining patient input in research topic generation sounds good, but there is a lack of detail regarding how patients will be appropriately selected or targeted. Will selection involve educational background, age, past medical history, family history or some other factors? How will the needs of older adults be represented? How will geriatric stakeholders, who may have difficulty with travel or mobility, be included? Who will represent prevention? Patients do not benefit from prevention because it is too late by the time that they are patients. Who will represent wellness or healthy aging? A recommendation is to use Value of Information (VOI) methods to determine research prioritization. This sounds like a good approach, but we do not have experience with this method. This chapter on Prioritizing PCOR may not be the location that the PCORI Board of Governors considered knowledge transfer. However, there is a need to address the issues surrounding how will the knowledge get out to the stakeholders and be utilized. Overall, Chapter 5 indicates a reasonable start to approaching the question of how to prioritize research that should be funded. However, there is a lack of detail regarding specifically how this will be approached, possibly due to the lack of evidence regarding how this should be approached. The authors demonstrate this on top of page 34, where they recommend "support for empirical research" to learn how to select research topics (i.e. research on what to research). PCORI could look towards HealthyPeople 2020 or other work that have been considered in great detail by public health researchers. An examination of which problems, facing older adults, are in need of answers and are feasible and which public health work has been completed could be valuable in assigning priorities to research topics pertinent to older adults.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

This chapter begins by saying that the randomized clinical trial (RCT) is the most effective process. However, there is no mention of those groups who are under-represented – namely, older adults with multiple chronic conditions. We believe that the chapter should note how healthier volunteers may not generalize to those with multiple medications/conditions, as seen with many older adults. Chapter 7 also discusses observational studies as an alternative and how these studies are limited and confounding. It mentions that propensity scores are often used but do not address what is unmeasured and confounding. We suggest reviewing the following papers of relevance: • Brookhart MA, Stürmer T, Glynn RJ, Rassen J, Schneeweiss S. Confounding control in healthcare database research: challenges and potential approaches. Med Care. 2010;48(6 suppl):S114–S120. • Bosco JLF, Silliman RA, Thwin SS, Geiger AM, Buist DS, Prout MN, Yood MU, Haque R, Wei F, Lash TL. A most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies. J Clin Epidemiol 2009 May 19. PMID: 19457638 PCORI should also consider adding that confounding issues are not as important in comparative safety studies as they are in comparative effectiveness studies where confounding by indication or disease severity comes in to play. We suggest the additional of these citations:- Klungel OH, Edwin P, Martens EP, Psaty BM, Grobbee DE, Sullivan SD, Stricker BH, Leufkens HG, de Boer A. Methods to assess intended effects of drug treatment in observational studies. J Clin Epidemiol 2004;57(12):1223-31. And Psaty BM, Koepsell TD, Lin D, Weiss NS, Siscovick DS, Rosendaal FR, Pahor M, Furberg CD. Assessment and control for confounding by indication in observational studies. J Am Geriatr Soc 1999;47(6):749-54. And Tinetti ME, Studenski SA. Comparative effectiveness research and patients with multiple chronic conditions. N Eng J Med 2011;364:2478-81.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

We believe this chapter is well constructed and that the description of the issues and the recommendations made are very reasonable. Important outcomes that we want to emphasize are functional outcomes, which are acknowledged: “Identify and select outcomes the population of interest notices and cares about (e.g., survival, function, symptoms, health-related quality of life) and that inform an identified health decision.”


Author

Medical Device Manufacturers Association

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

MDMA supports the principles of evidence‐based medicine and comparative effectiveness research. We believe that patients and physicians should have access to the best information and data on which treatments work best in addition to which treatments are less effective. The availability of this information will be in the best interest of the patient and all stakeholders. As Congress and the Administration work to build upon the efforts on comparative effectiveness research, it is important to also examine other areas of the health care system including delivery system reform and prevention and wellness programs. The PCORI would be amiss if it were not to study all factors that are contributing to increased costs within the healthcare system, including the examination of the inherent root causes. For example, we are all sadly aware of the growing obesity epidemic among the nation’s adolescent population. Our children are living less‐healthier lifestyles than in generations past; they are less physically active and are consuming more unhealthy foods. If our country continues on this path, we can only begin to try to speculate what the related costs will be for treating obesity related conditions, such as diabetes, pulmonary hypertension and heart disease in general. The positive news is that this is a controllable condition, and by increasing education and awareness, we can make an impact. Tackling these issues early‐on will likely have a far greater impact on cost savings in the future than our current thinking on CER. To this end, efforts on comparative effectiveness research should not be focused exclusively on efforts such as technology assessment. Rather, focusing research efforts on wellness and prevention should be complimentary to the current thinking on CER. As the old adage goes, “an ounce of prevention is worth a pound of cure.” In this sense, a primary focus of CER and the PCORI should be to examine the root causes of increased health care spending. Comparative effectiveness research should also be used to address disparities in the health care system. Too often is the case that varying patient populations receive disproportionate health care treatment and coverage. PCORI has a tremendous opportunity to conduct studies and use the research to address these disparities to ensure that all Americans are receiving the appropriate care. Finally, the manner in which the PCORI and related agencies conducts its work and generates data must be as transparent as possible. As is apparent, the work produced by the PCORI will likely have a significant impact on numerous entities, including patients, payers and the industry. In keeping with President Obama’s pledge for transparency in government, it is absolutely critical that the work of the PCORI remain open and transparent for all stakeholders. This includes allowing stakeholders to submit public comment on the PCORI’S processes and methodologies for comparative studies as well as its results. To this end, it is important that expanded CER initiatives include a formal infrastructure to ensure public input on the work of the PCORI and related agencies. Furthermore, PCORI should not engage in any activities that restrict physician choice or patient access to therapies.


Author

Integrated Benefits Institute

Date

Friday, September 14, 2012

Zip Code

94105

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

No

1 Comments

There should be more outcomes-related content on quality of life and work function in particular. For working age individuals being able to attend work and function effectively at work is a vital outcome. Returning to work in a healthy fashion as soon as one is able including in a modified arrangement should be emphasized.

2. What methodological gaps should be addressed in future reports?

Assessments of work function and healthy and timely return to work.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Yes, however a broader range of patient outcomes should be incorporated. For those of working age it is critical that ability to attend and perform well at work be assessed. This is at vital aspect of quality of life.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

If the right outcomes can be assessed then this initiative could greatly impact the health and quality of life of many individuals. If broader outcomes including work function were included, we have many channels to suggest for distribution.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

To the extent that outcomes measured can include a critical aspect of quality of life, work function, we would be able to suggest many avenues for dstribution.


Author

Center on Adherence and Self-Determination

Date

Friday, September 14, 2012

Zip Code

6492

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Although this is consistent with much of the work of the CASD, I suspect that this will be a controversial issues for many medical researchers, who are likely to view self-report measures as inferior to more objective outcomes.There are legitimate concerns with reliance on self-report measures, including concerns about socially desirable responding or common method bias, that lead many people to dismiss self-report methods. It would be useful for the standards to provide guidance on how to address these concerns.

2. What methodological gaps should be addressed in future reports?

Greater details about the decision model and how to incorporate into grant.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No


Author

Thomas Nolan

Date

Friday, September 14, 2012

Zip Code

20910

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Yes but incomplete. See general comments document.

2. What methodological gaps should be addressed in future reports?

Provide methods accessible to a large group of knowledge developers

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

But as configured would be limiting.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments-on-the-PCORI-Draft-Methodology-Report.docx


Author

Indiana Center for Evidence Based Nursing Practice

Date

Friday, September 14, 2012

Zip Code

456323

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

The standards and recommendations are welcomed and aligned with the mission and values of PCORI

2. What methodological gaps should be addressed in future reports?

Future reports need to address the valuable role for qualitative research and systematic reviews for addressing the meaning of the patient/family/provider experience. Questions like, what is the experience of I phenomenon of Interest in Co context and P population will provide systematic approaches to understanding patient experiences that quantitative approaches cannot provide. Qualitative research and systematic reviews of qualitative primary research add to our understanding of how patients experience healthcare and the meaning of outcomes.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

The standards and translation tables will be more easily used if they can be distilled into more consise forms.

Please provide your comments on Chapter 1 of the Draft Methodology Report.

The scenarios are particularly helpful to illustrate the issue of the key questions related to patient engagement and patient centeredness.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

The methods were appropriate adn public comment important. However, a plain language, short version would be much more appetizing and useful to illicit consumer feedback.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

the typical PICOT approach would need to be adapted to address questions of meaning. Specifically, PICO can be adapted to P for population, I for the phenomenon of Interest and Co for context. For example, a PICo question could be: what is the experience of acute dyspnea in patients who are in the midst of an acutte exacerbation of COPD. This type of question is important to understand how pts experience dyspnea. Qualitative, narrative data from 15 patients may provide a much richer data source than say--a unidimensional intensity ratiing of 200 pts experiencing data.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Bravo for the plea for transparency. Translation tables are very helpful. Please consider engaging the knowledge users as well as the consumers (ie, those who will use the knowledge generated from the systematic review or primary research)

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Systematic reviews certainly should inform the gaps. However, they are also part of the synthesis, translation and utilization cycle that will speed the spread of evidence uptake--perhaps, if we also study their role in the cycle. Are there plans to include the incorporatin of systematic review in the completion of the cycle (beyond gap analysis)?

Please provide your comments on Chapter 6 of the Draft Methodology Report.

translation table are a very helpful


Author

American Association of Neurological Surgeons-Congress of Neurological Surgeons

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Clinician

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AANS-CNS-PCORI-Methodology-Comments-9.14.12.pdf


Author

Kaiser Permanente

Date

Friday, September 14, 2012

Zip Code

91101

Stakeholder Category

Researcher

1 Comments

Please see attached PDF

2. What methodological gaps should be addressed in future reports?

Please see attached PDF

3 Comments

Please see attached PDF

5. Please enter any additional comments or concerns here.

Please see attached PDF

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI_Methodology_Report_KP_Comments.pdf

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Standard 7.1.5The focus of this standard appears to be exclusively on guidelines for ensuring internal validity. We recommend that the PCORI Methodology Committee expand this standard to include guidelines for external validity, such as the REAIM framework (http://www.rwjf.org/pr/product.jsp?id=15569) that balances considerations of internal and external validity.Standard 7.2.4When the interventions of interest are time-varying, relying solely on confounders measured before the start of exposure could be inappropriate; time-dependent covariates measured after the first exposure may be risk factors for the outcome and affect subsequent exposures of interest. Standard 7.2.4 implies that most PCOR questions will involve point treatment/exposure – an assumption that is too restrictive. In fact, the effects of exposures experienced over time are more often of interest in practice. This Standard also contradicts other statements in the Report, e.g., promoting observational methods using the example of a randomized trial with informative loss to follow-up and non-adherence (see Page 61). In this example, control for non-adherence and informative loss to follow-up will often require the research to account for confounders collected after randomization using marginal structural modeling (MSM). Use of MSM appears to be restricted or prohibited by this Standard. Standard 7.2.5This standard seems to assume that all analyses based on propensity scoring will involve a point treatment/exposure. But in cases of time-varying exposures, where propensity scores are used as weights (marginal structural modeling), this standard does not apply. “While stratified analysis is valid for estimating stratified treatment effects, it is incorrect for inferring HTE. To correctly test for HTE, investigators need to test whether the difference between the two stratified treatment effects is zero using an interaction test, assuming the variability is similar between the treatment groups” (see Page 66).The proposed test for heterogeneity of treatment effect (HTE) could be implemented from the two stratified effect estimates that are criticized in the first part of the statement: one could use the delta method to determine the asymptotic distribution of the difference of the two estimators and thus conduct the HTE test without using a model with an interaction term. The way the standard is written does not appear to allow for this non-parametric approach (see Page 64). Standard 7.4.3 “Single imputation methods like last observation carried forward and baseline observation carried forward are discouraged as the primary approach for handling missing data in the analysis” (see Page 72). This standard too appears focused on point treatment/exposure. It is not clear that it applies to PCOR questions involving time-varying exposures (We question whether the literature on which the standard is based applies to problems with time-varying exposures, e.g., MSM approaches. We are concerned that the Report mentions instrumental variable methods and propensity score methods but not MSM, especially because MSM can be used to handle 1) baseline and time-varying confounding (e.g., to control for non-adherence in the trial example; see Page 61) and missing data (e.g., to control for informative loss to follow-up; see Pages72-73) and in the example (see Page 62).Standard 7.5.1 This Standard should clarify some additional important issues. As written, it does not address the level of data integration with institutional operations data or enumerate data governance rules between the entity supplying the data and the research group. The Standard should outline procedures and practices for decision making and communications across multiple institutions, the proper level of involvement with clinical and scientific experts and information technology staff in confirming that the data extracted from operational data stores is clinically and scientifically useful and accurate. Researchers should be able to identify data sources and have clear methods for identifying the data transformations and pooling and using data from multiple institutions. Standard 7.5.2 This Standard should also address the necessary understanding a researcher must demonstrate regarding data privacy and security, including knowledge of the various datasets - unlimited, limited, de-identified and anonymized, and which methods of protection apply to each type. The researcher should also be able to distinguish the necessary safeguards to protect data in transit as well as data at rest, such as requirements for the use and disclosure of data, both for research and for activities preparatory to research. Standard 7.5.3 This Standard should address role-based access necessary to ensure appropriate access to data, including an understanding of how processes for local control and responsibility for identity management and authentication ensure appropriate access. This Standard should ensure that researchers are familiar with the policies and procedures used to govern identity management and authentication. Standard 7.5.4 Researchers should demonstrate an awareness of the agreements and procedures that must be in place to establish and maintain appropriate intellectual property protections. Standard 7.5.5Researchers should understand of the need for syntactical and semantic consistencies of the data across multiple institutions, and be able to demonstrate prior successful mastery of the techniques used to ensure data consistency. Researchers must demonstrate an understanding of the various coding standards and how they vary across the health care data domains. Standard 7.5.6 Researchers should characterize the quality of the metadata available within an individual institution and across multiple institutions. Researchers should also address the need for multiple levels of metadata that must be maintained for the needs of IT staff, analytical staff, and clinical and scientific staff. Standard 7.5.7 Researchers should express a clear understanding of the need for a Common Data Model and provide a schematic of the Common Data Model across all pooling institutions. The schematic should provide the keys for relational joins used to construct analysis files. This schematic should also describe the original data stores from which extractions are made and the subsequent loads into the Common Data Model. Researchers should be able to demonstrate reasonable experience with data model work, either by the researchers or by other investigative teams across the network.


Author

Population Health Impact Institute

Date

Friday, September 14, 2012

Zip Code

45140

Stakeholder Category

Organizational Provider

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

We were certainly pleased to see in “PCORI Methodology Committee Draft Methodology Report: “Our Questions, Our Decisions: Standards for Patient--‐centered Outcomes Research” July 23, 2012” a discussion of the importance of “dissemination;” and that it be “effective” and done “in a manner understandable to each target audience.” However, we were very surprised to see in a document related to methodology that the methods used to generate the results of PCORI studies was not part of the dissemination requirement. This is especially relevant to standard 4.1.4, “Develop and Implement a Dissemination Assessment to Achieve Broad Awareness of Study Results.”We believe this lack of a requirement to ensure the disclosure of methods used to achieve results must be corrected.

2. What methodological gaps should be addressed in future reports?

include in requirements the need to disseminate in an "effective" way to stakeholders (and the public) the methods used to generate results. A simple 'strength of evidence' should be replaced with a more robust discussion or score of the bias found in virtually all studies (see attached)

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Yes, if you include guidelines to disseminate methods used in studies, not just research results.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Include a discussion of the need to effectively disseminate methods used in studies to stakeholders (and public) , not just findings of studies.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/20120914-PHII-TO-PCORI-%2B-EXHIBITS-A-B.pdf

Please provide your comments on Chapter 4 of the Draft Methodology Report.

This is a follow-up to our comment to the Federal Coordinating Council for Comparative Effectiveness Research on May 11, 2009 on dissemination of methods where we stated that: “A structured approach to method disclosures, such as that developed by the Population Health Impact Institute (www.PHIinstitute.org), should be considered.” (See Exhibit A in our attachment). Section 1: We were certainly pleased to see in “PCORI Methodology Committee Draft Methodology Report: “Our Questions, Our Decisions: Standards for Patient--‐centered Outcomes Research” July 23, 2012” a discussion of the importance of “dissemination;” and that it be “effective” and done “in a manner understandable to each target audience.” However, we were very surprised to see in a document related to methodology that the methods used to generate the results of PCORI studies was not part of the dissemination requirement. This is especially relevant to standard 4.1.4, “Develop and Implement a Dissemination Assessment to Achieve Broad Awareness of Study Results.”We believe this lack of a requirement to ensure the disclosure of methods used to achieve results must be corrected. Section 2: Bias: The results of most, if not all, comparative effectiveness studies funded by PCORI (and others) will not be definitive. One major reason for that will be related to the internal (or external) validity of the studies being conducted. It is essential for all stakeholders, and the public in general, to be aware of the incompleteness of findings and therefore of a deeper appreciation of the key sources of bias that exist in all studies.The complexity of methods in many studies would seem to preclude a simple and effective dissemination strategy. A simple rating system of methods used in the past showing Randomized Clinical Trials (RCT) at the top and other studies below it, does little to help stakeholders (and the public) appreciate the ubiquity of bias in the projects the public is funding. Your document rightly pointed out the superiority of RCTs to observational studies, that both are subject to bias. And the former is often impossible to implement and when implemented on highly select population, to extrapolate to the real world. Bias must be not only be disclosed, it must be disseminated to stakeholders and the public.We encourage PCORI to design and implement a system to better communicate the essential sources of bias in all studies and make this a requirement for dissemination of all projects. To put in another way, PCORI must develop an effective method for the dissemination of results and methods to all stakeholders.Section 3: The Population Health Impact Institute is a 501c3 from the “real world” and has found a system (Healthcare Transparency and Attribution or H-TAP) related to assessing equivalence in the baseline/pre and follow-up/post period among four types of metrics between an intervention group and a reference (or comparator in your language) may be a good framework for PCORI moving forward. It is useful for assessing both internal validity and external validity, where in the latter the ‘referent’ is the population where the extrapolation is made. These ideas are expressed in more detail in the attached (Exhibit B)Extracted from H-TAP GlossaryMetrics: These are quantitative expressions of factors of interest. There are four types:• Type I (the intervention itself, the “cause”); • Type II (a proximate or intermediate outcome, an “effect”) ;• Type III (an ultimate outcome, also an “effect”); and • Type IV metrics (confounding factors that can independently impact Type II and Type III metrics) Below is conceptual diagram of the interrelationship of these metrics:Figure (in attachment)Method Evaluation Process (MEP) Measurement Periods / Comparisons: The act of comparing between two time segments and/or two different groups as highlighted below. Figure in attachment)Comparison #1: Comparing Measurement Periods: A to C (H-TAP Standard 16)• Equivalence of the “Pre” baseline-periods comparing the intervention and referent groups. • Assessment of confounding factors and outcome variables between the intervention and referent groups prior to the intervention.Comparison #2: Comparing Measurement Periods: A to B. (H-TAP Standard 23.1)• An examination of any “Pre/Post” change in intervention group. • Assessment of outcome factors over time in intervention groupComparison #3: Comparing Measurement Periods: C to D (H-TAP Standard 23.2) • An examination of any “Pre/Post” change in referent group.• Assessment of outcome factors over time in referent group. Comparison #4: Comparing Measurement Periods: B to D (H-TAP Standard 23.3) • Assessment of confounding factors and outcome variables between the intervention and referent groups after to the intervention.• Comparison of confounding variables and outcome metrics in the Post/follow-up-periods between the intervention and referent groups. These ideas are expressed in more detail in the attached (Exhibit B)Thank you for the opportunity to respond.


Author

American Clinical Laboratory Association

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see attached comments.

2. What methodological gaps should be addressed in future reports?

Please see attached comments.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Please see attached comments.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Please see attached comments.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/2012-0914-ACLA-Comments-to-PCORI-Methodology-Comm-FINAL.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see attached comments.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see attached comments.


Author

Ashley Slagle

Date

Friday, September 14, 2012

Zip Code

20993

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/Comments-on-Draft-Report_signed.pdf


Author

National Women's Health Network

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Other

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Comments_Final.pdf


Author

NORD

Date

Friday, September 14, 2012

Zip Code

20036

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/NORD-comments-PCORI-methods-report.pdf


Author

American Association for Cancer Research

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Researcher

5. Please enter any additional comments or concerns here.

See attached letter

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AACRLetterOnPCORIMethReport_Final.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

See attached

Please provide your comments on Chapter 2 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 3 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 4 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 5 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 6 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 7 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 8 of the Draft Methodology Report.

See attached letter

Please provide your comments on Chapter 9 of the Draft Methodology Report.

See attached letter


Author

Anonymous

Date

Friday, September 14, 2012

Zip Code

77096

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

In general they are however; I had hoped that they would open the topic up to more methodological approaches. This still seems very biomedically based and is somewhat limited in getting a better understanding of patient’s (individuals, families and communities) perspectives on health and health behaviors.

2. What methodological gaps should be addressed in future reports?

I would like to see more emphasis on mixed methods and other research approaches such as Action based approaches and utilizing some more sociological approaches and complexity science. I would like to see more exploration on approaches to health promotion and cultural issues.. I would also like to see more specifics on research methods for tailored interventions. A lot of interventions , especially behavioral ones are much more effective when they can be individually tailored to the individual or specific cultural groups and the dogmatic orientation to RCTs requires standardization to the intervention and this is off-putting to researchers and clinicians who effectively use a tailoring approach. I think the document makes an attempt to do this, but the underltyig fidelity to RCT research makes this less inviting for researchers.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

But some attention to above would be helpful


Author

Michelle Rogers

Date

Friday, September 14, 2012

Zip Code

19104

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

The issue of dissemination is addressed as one function without further comment. Often reports and results of studies are "disseminated" but patient involvment in the action is limited. The same thought that goes into the development of the outcomes, methods and processes should go into the development of the dissemination plans. That is minimally addressed in this document

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

Please provide your comments on Chapter 3 of the Draft Methodology Report.

More clarity should be provided as to if this list is linear or just what should be included overall. If it is to be followed step-by-step then step one (section 3.11) needs to be clarified. At our institution, "a Formal Study Protocol" is the the document that guides the study implementation. That would be the final step in formulating research questions. It may just be a wording issue.


Author

David Kent

Date

Friday, September 14, 2012

Zip Code

2139

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

I have not thoroughly reviewed the who docuument

3 Comments

I have not thoroughly reviewed the whole document

Please provide your comments on Chapter 7 of the Draft Methodology Report.

I read with interest PCORI’s recommendation for the testing of heterogeneity of treatment effect. Such standards are long overdue, and the recommendations made by PCORI are in general thoughtful and supported by a large body of literature and a consensus of investigators (as ably summarized in the review of guidance documents by Varadhan et al). In particular, clearly distinguishing between hypothesis-testing (confirmatory) subgroup analysis and other analyses (i.e. exploratory analyses) is critical if subgroup analyses are to yield credible results that can drive practice. With this in mind, I was puzzled as to why the new concept of descriptive analysis was adopted as a standard that future PCORI researchers will have to adhere to. While the concept of descriptive subgroup analysis as a distinct category is an interesting and novel idea worthy of additional exploration and discussion, it seems to fall far short of PCORIs declared criteria for a standard. The only known appearance of this concept is its brief description in the review of guidance documents by Varadhan et al, available online. After a thorough literature review, the authors propose this category as a novel concept, that had not been previously suggested, and for which the need was somehow totally overlooked. Meanwhile, the available description of this category does not address some very basic questions, such that the concept of what would constitute a descriptive subgroup analysis as a practical matter is likely to be confusing, even to the tiny group of investigators who are aware of the existence of this newly proposed category. The questions I raise below are a brief and non-exhaustive list of questions that the proposal of requiring “descriptive subgroups” raises.According to Varadhan et al, the essential distinction of descriptive subgroup analysis is that they are intended for future meta-analysis. This begs the question: intended by whom? If this is a PCORI standard, should all PCORI investigators now—on their own-- be anticipating which of the subgroup analyses that they are planning might be appropriate for future meta-analysis”? Since uniformity of these analyses across studies appears to be the hallmark of the descriptive analyses, it makes little sense for individual investigators to be determining this on their own. Yet, if the set of “descriptive variables” is determined (only) by some outside authority then by what authority (regulatory agencies , funding agencies, journal editors, etc.?), and why are investigators now required to define these in the studies proposed to PCORI when there has yet been no determination of what variables might be included in such a list, or how that list might even be determined. PCORI should clarify these issues before proposing this as a standard.Further, if regulatory agencies, funding agencies, journal editors, professional societies are the arbiters of the set of descriptive variables (which is only suggested but not explicitly stated and unmentioned in the table), then how is the key distinction between descriptive and exploratory variables that descriptive variables are always determined a priori (as Varadhan et al maintain)—since it is not even the investigators own responsibility to determine this set of variables and the set of descriptive variables may change at any point during the trial or even after the initial analysis of the trial, and might depend on what specific journal or regulatory body the trial is submitted to, or to which funding agency sponsors the research or to which professional society the investigators belong.Another basic question about this new category of descriptive subgroups is whether it describes a distinct category of subgroup analysis (i.e. one that is mutually exclusive from confirmatory and exploratory analysis). This is the position described by Varadhan et al (and clarified to me in personal communication). Does this make sense? To take an example adapted from Varadhan et al, if the FDA produces a guidance that gender is included in their set of descriptive variables for all heart failure agents, this would be non-informative to an investigator deciding whether to consider a subgroup analysis based on this variable as confirmatory or exploratory (or to a clinician, considering whether the subgroup results might be actionable). For some studies, investigators may choose to consider this variable exploratory (e.g. a new beta-blocker with no anticipated gender interaction); for some, it might be considered confirmatory (e.g. a sex hormone- based therapy with strong preclinical evidence of a gender interaction). In either case, it seems to me that the variable would remain on the list of descriptive variables.To my mind, the key potential contribution of having a “descriptive” category, as a practical matter, is that it defines the set of compulsory subgroup analyses. Having some consensus about an agreed upon list of compulsory variables that should always be reported (possibly disease-specific or therapeutic-class specific) is important, and not just for future evidence synthesis (as the PCORI guidelines suggest). Presenting overall results alone is tantamount to under-disclosing trial results, yet current peer reviewers have little authority to require from investigators to report specific subgroup analyses (for example risk-based analyses which some investigators, including myself, have suggested should be done routinely.) Yet now this key distinguishing feature is relatively obscure in the current guidance, and excluded from Table 7.1. Instead, the term is introduced as part of a now complexified statistical taxonomy.Indeed, calling this category of SGA compulsory, or “compulsory descriptive SGA”— may help clarify (and make more explicit) this important aspect of this new category of subgroup (now implicit)—and also clarify other characteristics of this new category of SGA. For example, it would also clarify the fact that the dimension described by this newly proposed category may be thought of as (largely) orthogonal to the (well accepted) confirmatory-exploratory dimension. While the confirmatory-exploratory dimension is determined by the scientific hypothesis that the investigators themselves propose to test, based on prior empirical evidence or highly compelling pathophysiology, the compulsory-discretionary dimension is determined externally, by stakeholders, and might be based on a number of factors (e.g. sub-populations of “high interest” or by compelling methodological justification, such as risk-based analyses). Finally, it would clarify that the major task in establishing this new category is not to compel individual investigators to fit their variables into some taxonomy, but to determine how to reach consensus about which variables all investigators (e.g. in a given disease area) might be compelled to report. Indeed, arguing about the precise definition of “descriptive SGAs” seems overall a less important project than (1) developing consensus that some SGA should be compulsory; (2) developing consensus about what criteria and processes might be used to consensually determine what those compulsory SGAs should be and (3) to actually develop such a consensus around these variables. This seems to be at the heart of what Varadhan et al mean by descriptive analysis, but for some reason these points are lost in the taxonomy. To summarize my own view: there is a practical need for investigators to declare and end users to know which subgroups are actionable (i.e. hypothesis confirming) and which are “fun to look at” and informative to future research (exploratory,). There is also a practical need for investigators to know which subgroups are compulsory to present, based on a community consensus, and which are purely discretionary. While facilitating unbiased meta-analysis, the need for compulsory analyses is not limited to the fact that some subgroups are intended for evidence synthesis (and meta-analysts may still be interested in synthesizing the results of confirmatory or exploratory analyses). I anticipate that there may be aspects of my discussion that both Varadhan et al, and members of the PCORI board, would disagree with. Regardless, I would hope that we could all agree that the descriptive category is brand new to the literature, unsupported by empirical testing of its feasibility or value, and that a fully developed consensus about what this category is and how it can be implemented has not emerged--even among the very small handful of like-minded investigators who are aware of this concept, and have given it some thought. It thus falls far short of something that can be described as a standard by which all PCORI proposals should be held.—and it may interfere with the uptake and impact of the other important (and well supported) recommendations that PCORI makes on this critical issue.


Author

Donald AQbrams

Date

Friday, September 14, 2012

Zip Code

94115

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Use of registries in integrative medicine/health to define best practices

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Nice work. Would ask to pay attention to integrative health reistries to help define best practices


Author

American Association of Colleges of Osteopathic Medicine

Date

Friday, September 14, 2012

Zip Code

20815

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AACOM-Comments-on-PCORI-Draft-Methodology-Report-9-14-12.pdf


Author

Margaret Winker

Date

Friday, September 14, 2012

Zip Code

60093

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Regarding data standards, the datasets should be placed in data repositories where they are freely available for researchers to access to conduct and replicate studies.


Author

Manpreet Sidhu

Date

Friday, September 14, 2012

Zip Code

7960

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

This report is a very good step forward to integration of patients relevant endpoints into the treatment paradigm, illustrating the often wide disparity between what is scientifically relevant and what this translates to in the clinical and humanistic aspect. However, it is important to note that in order to develop evidence that is both relevant to the real-world whilst providing clear evidence of a product's value, there needs to be a change in the way health technologies are approved, so that these large pragmatic "real-life" trials can deliver the value and information required without being bound by the initial regulatory hurdle - that requires the product to show it is as good as what is already available and does no harm. There are issues of size and patient recruitment that would constrain these trials, since larger numbers of patients would be needed to assess a difference in treatment. Finally, the "trust" issue implies that the industry is developing technologies that are harmful to patients, only to further their own agenda (increased sales), whilst it may be the case that a pharma company may want to show their product in the best posisble light, it is highly unfair to suggest that the company would do so at the detriment to the lives of patients. Furthermore, the industry is the largest provider of funding and scientific evidence, if it is felt that the studies provided by industry are not to trusted, then other means of assessing the scientific value of therapies is required, through governmennt or charitable funding.


Author

National Health Council

Date

Friday, September 14, 2012

Zip Code

20036

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/NHC-Comment-Letter-Draft-Methodology-Report.pdf


Author

Jeffery Dusek

Date

Friday, September 14, 2012

Zip Code

55410

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

The legislation lists 7 catergoies of interventions including integrative health practices also known by the term complementary and alternative medicine (CAM). The 2007 National Health Interview Survey (NHIS), which included a comprehensive survey of CAM use by Americans, showed that approximately 38 percent of adults use CAM. Despite this widespread usage in clinical settings across the country, there are no laarge scale data registries focused on use of integrative health or CAM. There is a unique opportunity for PCORI to support a large scale registry of integrative health or CAM practices that neither AHRQ nor NCCAM presently support.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

There is a unique opportunity for PCORI to support a large scale registry of integrative health or CAM practices that neither AHRQ nor NCCAM presently support.


Author

Association for Molecular Pathology

Date

Friday, September 14, 2012

Zip Code

20814

Stakeholder Category

Other

1 Comments

Please see attached letter.

2. What methodological gaps should be addressed in future reports?

Please see attached letter.

3 Comments

Please see attached letter.

5. Please enter any additional comments or concerns here.

Please see attached letter.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI_Comments_AMP_Final.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see attached letter.


Author

National Alliance on Mental Illness

Date

Friday, September 14, 2012

Zip Code

22203

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/nami-pcori-methods-comments.pdf


Author

American Hospital Association

Date

Friday, September 14, 2012

Zip Code

20004

Stakeholder Category

Organizational Provider

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/AHA-Comments-methodology-cmte-report-9_14_12.pdf


Author

Boehringer Ingelheim Pharmaceuticals, Inc.

Date

Friday, September 14, 2012

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/20120914-BIPI-Comments-on-PCORI-Draft-Methodology-Report.pdf


Author

PhRMA

Date

Friday, September 14, 2012

Zip Code

20004

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PhRMA-comments-PCORI-Methods-Final-Sept-2012.pdf


Author

American College of Emergency Physicians & Society for Academic Emergency Medicine

Date

Friday, September 14, 2012

Zip Code

27599

Stakeholder Category

Organizational Provider

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ACEP-SAEM-PCORI-09-14-12.pdf


Author

Biotechnology Industry Organization

Date

Friday, September 14, 2012

Zip Code

20024

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/BIO-Comment-Letter-PCORI-Draft-Methodology-Report_9-14-12.pdf.pdf


Author

AAPCHO, APIAHF, NAPAFASA

Date

Friday, September 14, 2012

Zip Code

92078

Stakeholder Category

Patient/Caregiver Advocacy Organization

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Methodology-Comments-9-14-12-final.pdf


Author

California Chronic Care Coalition

Date

Friday, September 14, 2012

Zip Code

95814

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see the attached comments.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/CCCC-Comments-PCORI-Methodology-Commitee-Report.pdf


Author

National Working Group on Evidence-Based Health Care

Date

Friday, September 14, 2012

Zip Code

22311

Stakeholder Category

Patient/Caregiver Advocacy Organization

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

See attached comments related to increasing focus on patient-centeredness in the process of the report develompent and use and report standards.

2. What methodological gaps should be addressed in future reports?

See attached comments related to specificity and patient-centeredness of the standards.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

The report requires much more specificity and operationalization of the standards in order for it to be useful for grant reviews and submissions. Please see the attached comments for more detail.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Please see the attached comments for more details, but we recommend that PCORI work on a more accessible executive summary or summary document geared at patients and other lay people.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/EBH-Working-Group-Comments-PCORI-Methodology-Committee-Report.pdf


Author

GlaxoSmithKline

Date

Friday, September 14, 2012

Zip Code

19426

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

See attached

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-9-14-12.pdf


Author

Pfizer Inc

Date

Friday, September 14, 2012

Zip Code

10017

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see attached letter.

2. What methodological gaps should be addressed in future reports?

Please see attached letter.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

No

3 Comments

Please see attached letter.

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

Yes

5. Please enter any additional comments or concerns here.

Please see attached letter.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/2012-Sept-14-Pfizer-Comments-on-PCORI-Draft-Methodology-Report.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see attached letter.

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see attached letter.


Author

The Society of Thoracic Surgeons

Date

Friday, September 14, 2012

Zip Code

20001

Stakeholder Category

Other

5. Please enter any additional comments or concerns here.

The Society of Thoracic Surgeons (STS) is providing comments on the various chapters of the Patient-Centered Outcomes Research Institute (PCORI) Draft Methodology Report: “Our Questions, Our Decisions: Standards for Patient-centered Outcomes. STS represents more than 6,500 surgeons, researchers and allied health care professionals worldwide who are dedicated to ensuring the best possible outcomes for surgeries of the heart, lung, and esophagus, as well as other surgical procedures within the chest. We appreciate the opportunity to provide comments on this draft proposal. STS appreciates the thoughtful deliberations of the PCORI Methodology Committee on this matter and appreciate the opportunity to offer comments. We look forward to working closely with the PCORI to ensure that physicians remain engaged, enthusiastic, and involved stakeholders in this complex and very important process. Again, thank you for the opportunity to comment on this important methodology report.


Author

The Society of Thoracic Surgeons

Date

Friday, September 14, 2012

Zip Code

20001

Stakeholder Category

Other

Please provide your comments on Chapter 4 of the Draft Methodology Report.

4.1.1 Engage patient informants, persons representative of the population of interest, in all phases of PCORComments: STS agrees that it is critical to seek patient input in particular from those persons representative of the population of interest when designing and formulating PCOR research studies. However, we have concerns with the requirements set forth for the research proposals to involve patients in all phases of PCOR. We believe these requirements would probably be too cumbersome to researchers. STS recommends as an alternative, a requirement that would have patients serve as part of a multi-stakeholder advisory committee established by the researcher to achieve the defined objectives - to engage patients in formulating research questions; defining essential characteristics of study participants, comparators, and outcomes; monitoring study conduct and progress; and disseminating results, etc.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

7.2.4 Measure Confounders before Start of ExposureComments: Section 7.2.4 Measure Confounders before Start of Exposure states that variables for use in confounding adjustment should be ascertained and measured prior to the first exposure to the therapy (or therapies) under study. STS recommends that this requirement to measure confounders prior to first exposure is not always feasible and should not be a requirement. PCORI might consider making a recommendation to encourage researchers to make best efforts to achieve this objective, if possible.

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Chapter 8. Design-- Specific MethodsStandards for Data RegistriesComments: STS supports the PCORI Board’s findings that data from randomized controlled studies may not be ideal and are not always feasible to obtain. We agree that effective PCOR will require opportunities to incorporate data from alternative sources, such as clinical registries. Clinical data registries allow health care stakeholders to more clearly observe patterns of care and the effectiveness of various interventions over time and in different populations.Expansion of existing clinical registries and databases would provide a strong foundation when conducting PCOR and at the same time these registries would also provide an excellent beginning point for PCOR. Utilizing, expanding, or integrating existing clinical registries would constitute an invaluable investment in the much needed infrastructure for accurately comparing clinical outcomes based on “real life” conditions where care delivery settings vary, patients may have numerous co-morbidities, and the patient populations are diverse. In turn, the clinical registries are not identical and may, to a greater or lesser extent, be able to promote a learning health care environment. Thus, evaluating the relative clinical effectiveness of various clinical registry models and alternatives to them remains a vital priority. Further, building the PCOR infrastructure and capacity, in part, upon registries and clinical data networks will leverage PCOR resources and boost the capacity of the system as a whole to learn and adapt in real-time.STS is eager to participate in efforts to demonstrate the utility of clinical registries to develop an infrastructure to accelerate PCOR and methodological research. We look forward to opportunities to assist PCORI in this regard – to help to build clinical registries that effectively and efficiently produce PCOR. PCOR has the potential to have a profoundly positive impact on the quality of the information available to physicians and patients and, when used appropriately and with care, may help address escalating health care costs.


Author

March of Dimes

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Other

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI-Research-Methodology-Comments.pdf


Author

ACRO

Date

Friday, September 14, 2012

Zip Code

20005

Stakeholder Category

Industry

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/ACRO_PCORI_Methodology_Comment-9.14.12.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

The draft Methodology Report contains a section titled “Problems that PCORI Hopes to Address.” Frankly, we do not believe this section is at all relevant to the work of the Methodology Committee and are unsure why it would appear in the draft report.That said, to the extent any of these “problems” exist, they would be largely addressed through the execution of a well-designed research project agenda.Most disturbing is the discussion of “a trust problem” in this report. There is an important fundamental and entirely legitimate difference in the research approach taken by the biopharmaceutical industry when developing a new drug and that taken to study comparative effectiveness. The study objectives and endpoints are entirely different. Further, for a Committee comprised of researchers to make such assertions of manipulation or bias without any supporting evidence is highly problematic and calls into question the objectivity and integrity of the Committee. This discussion of “a trust” problem” impinges on PCORI’s credibility as a research organization by exposing its own potential bias. This discussion is beneath the research standards to which PCORI should adhere and should be removed from the final version of the report.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Chapters 6, 7 and 8 of the draft report, which outline the translation table and research methods, comprise the foundation of the Methodology Committee’s work and, by statute, the extent of the Committee’s mandate.In general, the Committee has provided a comprehensive overview and analysis of research methods relevant to patient centered outcomes research, including, importantly, the need to include subgroups in research projects. We are also pleased to see the Committee acknowledge that flexibility of methods will be needed in each research project, as, by definition, the goals and objectives of the research will determine the specific methodology employed.

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Similarly, the Committee’s desire to promote “crosscutting” and innovative research methods should be commended as comparative clinical effectiveness research with this high level of patient engagement is unprecedented. The use of adaptive and Bayesian trial designs will clearly come into play within this new research paradigm. ACRO has no further specific comments on research methodology at this point as we see this as an evolving exercise. We look forward to lending the expertise that resides within our member companies to the Committee and to provide representatives to any expert panels that may be convened by the Institute.


Author

Genentech

Date

Friday, September 14, 2012

Zip Code

22030

Stakeholder Category

Industry

Please provide your comments on Chapter 3 of the Draft Methodology Report.

• Genentech is generally supportive of the standards described in Chapter 3, which emphasize good research practices as outlined in the GRACE principles. These include developing a formal study protocol, identifying populations and assessing patient subgroups that may respond differently, using appropriate interventions and comparators, and measuring outcomes that people care about. Guidelines also exist for reporting literature reviews and meta-analyses, including PRISMA and MOOSE, and Genentech recommends PCORI include a reference to those as well.• In Table 3.1, we recommend PCORI emphasize the declaration of conflict of interests, transparency, and the peer review process. The existence or even perception of undisclosed conflicts can limit the utility of research findings by limiting their uptake, and those studies funded by PCORI will be closely scrutinized by the public. Genentech supports every effort to ensure trustworthiness of research findings, and believes that transparency can best be achieved through scientific collaborations. • Genentech further supports PCORI’s effort to achieve transparency through public notification of research awards and subsequent publication of PCORI funded projects.

Please provide your comments on Chapter 6 of the Draft Methodology Report.

• In Figure 6.2, Genentech is concerned that there is potential for confusion among study designs with regard to the term intention to treat (ITT). To improve clarity, PCORI should consider alternative language, such as using ITT for randomized controlled trials and treatment as observed (TAO) for observational studies.


Author

Hospital Sisters Health System

Date

Friday, September 14, 2012

Zip Code

62707-9797

Stakeholder Category

Organizational Provider

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

Please see Hospital Sisters Health System Comment letter (attached)

2. What methodological gaps should be addressed in future reports?

Please see Hospital Sisters Health System Comment letter (attached)

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

3 Comments

Please see Hospital Sisters Health System Comment letter (attached)

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

Please see Hospital Sisters Health System Comment letter (attached)

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/PCORI_Draft_Method_Comment_HSHS_9-14-2012.pdf

Please provide your comments on Chapter 1 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 2 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 3 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 4 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 5 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 6 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 7 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 8 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)

Please provide your comments on Chapter 9 of the Draft Methodology Report.

Please see Hospital Sisters Health System Comment letter (attached)


Author

American College Of Chest Physicians

Date

Friday, September 14, 2012

Zip Code

60062-2348

Stakeholder Category

Organizational Provider

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

1) It is not clear how the translation table is to be used and by whom. It may be that this concept is new and seems to have some heuristic value. Some further clarity on this would be helpful.2) Is the use of the translation table an expectation for developing a methodology for every PCOR question?3) Should cost of care be a factor requiring a methodological consideration? Patient preference issues seemed more oriented to benefits and non-economic burdens.4) Missing is any discussion about the cost and hence cost-effectiveness or value of comparative clinical effectiveness research and its impact on study design. 5) There also remains some ambiguity of the details of the definition of PCOR, despite the length and specific attempts to do so.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes

4. Do you have any ideas about how PCORI might distribute the final report to increase awareness/application of the standards?

No

5. Please enter any additional comments or concerns here.

See uploaded letter

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/120914-ACCP-Response-to-PCORI-Draft-Methods-Paper3.pdf

Please provide your comments on Chapter 7 of the Draft Methodology Report.

See attached letter


Author

Anonymous

Date

Friday, September 14, 2012

Stakeholder Category

Researcher

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

2. What methodological gaps should be addressed in future reports?

Healthcare Delivery Systems Research Methodology References for inclusion consideration:•Jeffrey Alexander, University of Michigan, Methods and Metrics Issues in Delivery System Research (subsequently published as Alexander JA, Hearld LR. Methods and metrics challenges of delivery-system research. Implement Sci 2012 Mar 12;7(1):15. Available at http://www.implementationscience.com/content/pdf/1748-5908-7-15.pdf)•Creswell JW, Klassen AC, Plano Clark VL, Smith KC for the Office of Behavioral and Social Sciences Research. Best practices for mixed methods research in the health sciences. August 2011. National Institutes of Health. Available athttp://obssr.od.nih.gov/mixed_methods_research•Opportunities for Advancing Delivery System Research. AHRQ Expert Meeting on the Challenge and Promise of Delivery System Research: Meeting Summary. Available athttp://www.ahrq.gov/qual/deliverysys/2011mtg/mtgsumm.htm#messagesThank you for your consideration.

3. Can this report be used to inform PCORI policy regarding grant reviews and submissions?

Yes


Date

Friday, September 14, 2012

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/BCBSA-Comment-Letter-on-PCORI-Draft-Methodology-Report.pdf


Author

National Pharmaceutical Council

Date

Friday, September 14, 2012

Zip Code

20006

Stakeholder Category

Industry

1. Are the standards and recommended actions consistent with PCORI's mission and vision?

Yes

1 Comments

The standards and recommended areas represent a portion of the research, interventions, and stakeholders whom PCORI is meant to research or inform. To conduct comparative clinical effectiveness research on the medical treatments, services, and health care interventions outlined in the Affordable Care Act legislation, a broad set of research methodologies will need to be utilized. For example, in order to evaluate various protocols for treatment, care management, or delivery systems, comparisons among groups of patients standards on cluster randomized trials will be needed. Limiting the types of study designs in which standards are identified may result in over-emphasis among areas best studied in traditional parallel randomized trials or existing data sources, and few studies in other areas without standards such as healthcare interventions and delivery systems. Without standards, those who review PCORI proposals are unlikely to have the tools and resources to review, approve, or recommend funding for questions that offer a unique opportunity for PCORI to improve the nation’s health.

2. What methodological gaps should be addressed in future reports?

1) Standards for systematic reviews should be included either wholly or in part in the Committee Report. 2) If standards do not exist for all interventions outlined in the legislative mandate or study designs outlined in Table 6.1, expanding the list of research standards should be a priority of the methodology committee and the public. 3) All stakeholders, not just patients should be addressed. Without consistency of standards across all stakeholders, there will be limited uptake and coordination. One of the unique aspects of PCORI is the multi-stakeholder board and broad expertise. By segmenting these standards by stakeholders, one of the unique benefits of PCORI surrounding stakeholder dialogue and collaboration is diminished.

3 Comments

Despite a wealth of studies to inform PCOR as outlined in Chapter 6, the limited number of study designs included in the report may not adddress the standards needed to review all grants and submissions. We suggest expansion of the types of studies as a priority for the PCORI methodology committee. Without standards, those who review PCORI proposals are unlikely to have the tools and resources to review, approve, or recommend funding for questions that offer a unique opportunity for PCORI to improve the nation’s health.

5. Please enter any additional comments or concerns here.

We welcome the opportunity to comment on the Draft Methodology Report “Our Questions, Our Decisions: Standards for Patient-centered Outcomes Research” published for public comment on July 23, 2012. In the first report, the Methodology Committee’s initial 60 standards to guide patient-centered outcomes research represents an important first step to improve the quality, conduct, and applicability of research to inform treatment decisions by patients and their providers. We stand ready to provide additional information, engage in further dialogue, and advance the development of high quality methodological standards. We firmly believe that research should be rigorous and transparent, and conducted in accordance with a clear set of methods guidelines.

6. You may upload a document with any additional general comments here.

https://pcori.org/sites/default/files/public-comments/methodology-report/NPC-Comments-on-the-Methodology-Committee-Report-09142012.pdf


 

What's Happening at PCORI?

The Patient-Centered Outcomes Research Institute sends weekly emails about opportunities to apply for funding, newly funded research studies and engagement projects, results of our funded research, webinars, and other new information posted on our site.

Subscribe to PCORI Emails

Image

Hand pointing to email icon