PCORI funded the Pilot Projects to explore how to conduct and use patient-centered outcomes research in ways that can better serve patients and the healthcare community. Learn more.

Background

There is little research on best practices or clinical outcomes of psychiatric inpatient treatment or general use tools to assess the quality of inpatient psychiatric settings from the patients’ perspective. Specifically, it is important to understand what experiences are important to patients their recovery.

Project Purpose

The purpose of this study was to develop and conduct initial testing on a measure to evaluate safety and person-centered care in inpatient psychiatric settings. The measure is predicated on the idea that if optimal care is to be achieved, then all major stakeholders (patients and staff) need to experience a positive environment.

Study Design

This study employed an instrument development design to create the patient and staff nurse versions of the Combined Assessment of Psychiatric Environments (CAPE) (phase one) and to test dimensions and items of both versions (phase two). In preliminary work, dimensions for the CAPE patient and staff nurse versions were derived from the qualitative literature on patient and staff nurse experiences, and items were generated from focus groups with individuals who had been hospitalized and staff nurses who were working in psychiatric units. Phase one of the project included event judgment, expert panel review, and cognitive interviewing.

• Event Judgment: From the previously conducted staff nurse interviews and consumer focus groups, a list of items describing important experiences of inpatient psychiatric treatment was generated and, informed by existing literature on the psychiatric inpatient and staff nurse experience, organized into five dimensions. Following item reduction, 30 former inpatients and 30 nursing staff were asked to judge the importance of the nominated events to establish which were most valued. Judgments were made on each statement by rating the importance of events on a four-point scale. From these data, potential items for both versions of the CAPE measure were generated.
• Expert Panel Review: Items nominated as most important and relevant during event judgment were organized into CAPE patient and staff nurse pilot versions. These pilot versions were then reviewed by an expert panel of eight peer specialists (patient version) and eight expert staff nurses (staff nurse version). Each member of the expert panel was mailed the pilot version and asked to evaluate the dimensions and items from the perspectives of patients (peer specialist panel) and psychiatric nurses (expert nurses panel). Items were evaluated based on their importance, relevance, and fit within dimensions. Participants were also asked to respond to an open-ended question about whether any important quality dimensions of the psychiatric inpatient setting were missing.
• Cognitive Interviewing: The purpose of the cognitive interviews was to determine whether items were understandable, answerable, and evoked the responses anticipated by the investigators. Following the expert panel review, 25 items were selected for beta versions of the surveys. From the group of participants in the event judgment procedure, 10 participants in each group (staff and consumer) were asked to elaborate on their understanding of these items to ascertain how they interpreted items, their rationale for the importance of items, and whether the item fit in the dimension. From these data, potential items for both versions of the CAPE measure were generated.

This summary will focus on phase two of the study, testing the pilot instruments.

Participants, Interventions, Settings, and Outcomes

Because several staff in one unit did not submit completed forms, the research team ended with a sample of 113 staff and 150 patients. The inclusion criteria for patients required that they be (1) at least 21 years old, (2) in the unit for at least four days, (3) within 24 hours of discharge, and (4) English speakers. Nursing staff were either registered nurses or mental health counselors who worked at least half time in the study unit. Patients and staff members for the study were recruited in collaboration with the managers of the units.

The pilot patient and staff nurse versions of the CAPE were administered to convenience samples in six inpatient psychiatric units with an average bed size of 20. These facilities included three psychiatric units in community hospitals, two psychiatric units in medical centers, and a unit in a free-standing psychiatric hospital. All are designed to provide brief psychiatric inpatient treatment to patients who, because of mental illness, were deemed dangerous to themselves or others or unable to care for themselves.

Data Sources

To test the pilot versions of the CAPE, patients and staff nurses completed their respective versions of the pilot instrument. In addition, patients completed the Perceptions of Care (POC) survey, a 16-item, clinical-care-oriented, self-report satisfaction rating scale. Nurses and mental health workers completed the Practice Environment Scale of the Nursing Work Index (PES-NWI), a 31-item instrument used to measure the quality of the nursing work environment.

Data Analysis

Psychometric evaluation of the patient and staff versions of the CAPE was conducted in successive phases. First, internal consistency of the two forms (patient and staff nurse) was assessed to estimate Cronbach’s alpha. Internal reliability was established with Cronbach’s alpha and test-retest reliability, using 20 percent of former inpatient and nurse samples. Convergent validity was established with the PES-NWI (staff nurses) and the POC survey (patients). The dimensionality was explored using a confirmatory factor analysis—specifically, structural equation modeling—to examine whether the five quality dimensions described in the proposal were reflected in the empirical data.

Findings

The overall alpha for the tools was .91 (staff nurses) and .91 (patients). The subscales ranged from .654 to .804 (patients) and .535 to .846 (staff nurses), but particular items had a negative impact on the subscale alphas. Thus, the tools had good internal consistency but the researchers had questions about several of the subscales for both patients and staff nurses. The alphas for the test-retest were .82 (staff nurses) and .76 (patients), indicating adequate internal reliability. Correlations between the PES-NWI (staff nurses) and the POC survey (patients) confirmed convergent validity, and correlations between the subscales of the instruments were in the expected direction. The tool measures two distinct dimensions for the patient version and five dimensions for the staff nurse version. For patients, the dimensions were:

• Consumer perceptions of staff competence
• Consumer perceptions of treatment efficacy.

For staff nurses, dimensions were:

• A sense of personal competence or their own effectiveness
• Staff perception of whether there were adequate resources and staffing on the unit
• Management involvement with care
• How well staff in the unit worked together as a team
• The amount of autonomous control the staff have in the unit.

Limitations

The patient group tended to rate almost all items high, meaning they believed they frequently experienced the events depicted in the study items. Thus, although the staff saw differences in dimensions of their experiences in a particular unit, such as their view of the safety of the unit, these differences were not apparent in the patient data. It could be that the questions on the tool need to be reworded to capture differences, or it could be that the tool follows the known pattern of patients rating treatment experiences relatively positively.

Conclusions

The CAPE patient and staff nurse versions appear to be instruments with good internal consistency and internal reliability as well as convergent validity. There appear to be underlying dimensions that inform what patients believe are important aspects of treatment. There is a growing interest in patient-reported outcome measures and patient-centered care. Until recently, patients treated in inpatient psychiatric units were excluded from standard hospital measures (e.g., Press-Ganey).

Future Research

Future research will include hospitals using the CAPE pre- and post-culture change and testing to determine whether the CAPE is sensitive to restraint use. The researchers are also in the process of creating a shorter version of the CAPE.