The PCORI Board of Governors approved this proposed category of Methodology Standards for a public comment period in July 2018. The public comment period took place between July 24 and September 21, 2018, and the final standards were adopted by the Board in February 2019.

View the Final Standards

View the Draft Proposed Standards for the Public Comment Period

 

Submitted Comments

Submitted

8/1/18 14:26

Name and Organization

Peggy Shannon-Baker, Georgia Southern University

Community

Stakeholder

---

Submitted

8/1/18 15:49

Principal 1: This is placement text for the title

Design should be supported by appropriate reference to methodological literature.

Name and Organization

Anonymous

Community

Stakeholder

---

Submitted

8/2/18 21:34

Principal 1: This is placement text for the title

The language used here implies that qualitative design primarily involves choosing a specific approach or strategy from a defined list of possibilities (e.g., "state the qualitative approach," "identify which approach," "state the types of data," "select and justify an appropriate . . .sampling strategy"). Certainly some amount of selection is involved in most qualitative research, but a great deal of qualitative design (particularly by experienced qualitative researchers) involves the creation of a strategy that draws from multiple sources and is not easily pigeonholed or labeled. The term primarily used in the following standards, "describe," is generally more appropriate for qualitative studies. What is critical in assessing a proposed qualitative study is not its use of standardized components, but the overall coherence of the design. For a detailed presentation of such an approach to design, see Joseph Maxwell, Qualitative Research Design: An Interactive Approach, 3rd ed, 2013, Sage Publications.

Principal 4: This is placement text for the title

The designation of only 3 ways of establishing credibility is far too limiting, and doesn't address the most important issue for credibiolity/validity: ruling out alternative plausible conclusionss or interpretations of the data. Negative case analysis and member checking are both important for many qualitative studies, but aren't always appropriate or even possible, and I'm not sure how "reflexivity" is a way of establishing credibility; do you mean addressing researcher reactivity/reflexivity?

General feedback on the Proposed Principles for the Consideration of the Full Range of Outcomes Data

Although much of these standards is appropriate, the overall tone seems better suited to quantitative research, in which standardization is important for assuring the comparability and aggregatability of the data, than to qualitative research. I'm not challenging the importance of explicitly addressing threats to credibility/validity, but qualitative research has rather different ways of dealing with these. See Joseph Maxwell, The Validity and Reliability of Research, pp. 116-140 in D. Wyse, N. Selwyn, E. Smith, and L. E. Suter (Eds.), The BERA/SAGE Handbook of Educational Research, London: SAGE Publications, 2017.

Name and Organization

Joseph Maxwell

Community

Stakeholder

---

Submitted

9/10/18 16:13

Principal 1: This is placement text for the title

1. I strongly recommend you use the term “method” rather than “approach”. Authots write that they used a “grounded theory approach”, and on examination you see they used some grounded theory strategies, such as “constant comparision” but did NOT use grounded theory. Also focus groups may be used as a method and as a strategy. When used as a methods, the design should be different than is used as a strategy (increased number of groups, and so forth). Confusing approach with method is problematic. 2. Similarly, use STRATEGIES for data collection rather than TYPES of data. Interview data (i.e., a type) has more than 15 strategies of collection (i.e., informal, unstructured, guided, semi-structured, focus group, “man-on-the-street’, recorded, video, group, family, dyadic, child, with drawings, photovoice, confirmatory, inductive, deductive, and so forth ). 3. Use Pacing of data collection. Pacing refers to the use of strategies in relation to the analysis. Timing refers to clock time. Sometimes it is necessary to pace interviews, so that “retrospective” data are collected after present, ongoing observations. Data sets may be moved around in the analysis to make a cohesive and compelling narrative. 4. Why are you asking about computer programs? Are you going to evaluate the appropriateness of one over another? Using a program does not ensure rigor, in fact, in my experience, used inappropriately, it may make the results shallow and trite.

Principal 2: This is placement text for the title

ange approach to method (See above). Sampling may initially fit the research question, but it the research is working reflexivity, sampling may also change in the course of the project. Sample size is tricky, as you know. But recently I notice something sacred is happening with 10—which is almost universally disasterous. Can this be written less precisely? Perhaps, “explain your anticipated sample size, and rationale for variation in sampling that may occur during your study. “Saturation” is the most misused and misunderstood term—how will the committee evaluates a project that defines saturation as replication?

Principal 3: This is placement text for the title

Data analysis. Case study or group analyses? Change “coding of themes” to “thematic or categorical analysis” Have the research question been modified in the process of analysis? Describe? Add “categories and/or themes”. Remove: “State how this process is congruent with the chosen qualitative approach and its methodology.” That should be inherent above. I am worried that “conclusions will be derived and how they relate to interpretations and content of the original data” will interfere with inductive processes. Can this be written more speculatively?

Principal 4: This is placement text for the title

Questions 4. There is a difference between descriptive and interpretative research that is not addressed here. “Rigor. reflexivity, negative case analysis, and/or member checking.” These are processes for determining that the investigator is on the “right track” during analysis. For the reviewer, the final result must reveal new findings, be strong, and convince. That means that the results must be rich, adequately documented, logical and innovative, even if it moves beyond the status quo. It must be theoretically bold, and “stand on the shoulders of giants” to show how it moves the field forward. This should appear in the "implications" section of the proposal.

General feedback on the Proposed Principles for the Consideration of the Full Range of Outcomes Data

How does the committee review a proposal that has not yet been conducted, and the investigator cannot promise the results (as with quan) research. One the of the most important criteria to review is the researchers vita. Has the investigator had previous experience and produced excellent qualitative inquiry? If the investigators is new to qualitative inquiry, do they have a mentor/consultant with regular contact? One of the biggest mistakes is the new investigators think that qual inquiry is quick and easy and do not allow enough time for analysis and conceptualization. [Where is conceptualization in the above criteria?}

Name and Organization

Janice Morse, University of Utah, editor Qualitative Health Research, Ed Bd JMMR

Community

Stakeholder

---

Submitted

9/11/18 4:28

Name and Organization

Alicia O'Cathain, ScHARR, University of Sheffield, UK

Community

Stakeholder

---

Submitted

9/16/18 9:10

Principal 1: This is placement text for the title

last sentence - perhaps add 'if any' in parentheses after programs, as not all researchers use computer assisted data analysis software.

General feedback on the Proposed Principles for the Consideration of the Full Range of Outcomes Data

Helpful and clear standards.

Name and Organization

Catherine Pope University of Southampton UK

Community

Stakeholder

---

Submitted

9/18/18 13:57

Principal 1: This is placement text for the title

Need to add strategy for subject language preferences. Too many times non-English speakers are excluded because of researcher lack of capacity.

Principal 2: This is placement text for the title

Need to add strategy for subject language preferences.

Principal 3: This is placement text for the title

Inferences / themes - in particular with respect to language / culture. Can also include age as a culture.

General feedback on the Proposed Principles for the Consideration of the Full Range of Outcomes Data

Need to add strategy for subject language /cultural preferences throughout. Need to be explicit about this.

Name and Organization

Paul Meissner, Montefiore Health Systems

Community

Stakeholder

---