- Email: [email protected]
Consumer perspectives on quality of inpatient mental health services
Consumer Perspectives on Quality Of Inpatient Mental Health Services Patricia B. Howard, Peggy El-Mallakh, Mary Kay Rayens, and James J. Clark This study, conducted at two public-sector psychiatric hospitals in a south-eastern state, investigated satisfaction with inpatient services and treatment outcomes among 204 hospitalized mental health consumers. A simple survey design with nonrandom sampling technique was used; instruments included the KY-CSI, the 21-item MHSIP Consumer Survey, and the CSQ-8. Respondents reported satisfaction with time available to be with other patients, staff availability, and their degree of comfort talking to staff. Areas of dissatisfaction included lack of client input into treatment planning, lack of family involvement, and lack of medication education. Providers are encouraged to include clients in all phases of treatment planning and to continue to evaluate consumer perspectives of quality services. © 2003 Elsevier Inc. All rights reserved.
C
HANGES WITHIN THE health care system have prompted health care providers and researchers to re-evaluate and modify the process of service system evaluation. Evaluation research conducted throughout the 1980s focused primarily on the structure and processes of health care. In contrast, evaluation research shifted its focus to the measurement of treatment outcomes during the 1990s (Jennings, Staggers, & Brosch, 1999; Sederer, Dickey, & Hermann, 1996). Issues that drove the outcomes measurement movement included escalating health care costs, consumer advocacy for needed and appropriate services, increased focus on evidence-based practice, implementation of managed care in both the private and public mental health care systems, and development of new service delivery systems (Auslander et al., 1998; Campbell, 1998; Chisholm et al., 1997; Kolb, Race, & Seibert, 2000; Rosswurm & Larrabee, 1999; Teague, Ganju, Hornik, Johnson, & McKinney, 1997). Buckley (1993) conceptualizes the evolution of evaluation research as a decreased focus on the service delivery organization and a simultaneous emergence of a person-centered focus. This evolution represents a shift away from “persons served within individual specialty mental health organiza-
tions towards persons with significant needs for mental health services and supports, regardless of the number or type of organizations” (p. 1). Likewise, Campbell (1998) notes that early research efforts evaluated mental health service systems according to “measures of output and volume” (p. 13), but states that current evaluation research increasingly emphasizes patients’ values and “broadened measurements of outcomes that encompass not only symptom reduction, but functional status and quality of life” (p. 13). Basic research in the fields of neuroscience and genetics are crucial for understanding the etiology From the College of Nursing, University of Kentucky, Lexington, KY and College of Social Work, University of Kentucky, Lexington, KY. Supported by the Kentucky Department for Mental Health and Mental Retardation Services (KDMHMRS) and, in part, with a grant from the Department of Health and Human Services, Public Health Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services (4 HRI SM52058-02-4). Address reprint requests to Patricia B. Howard, PhD, RN, CNAA, College of Nursing, University of Kentucky, 760 Rose Street, Lexington, KY 40536-0232. E-mail: [email protected] 䊚 2003 Elsevier Inc. All rights reserved. 0883-9417/03/1705-0003$30.00/0 doi:10.1053/S0883-9417(03)00096-7
Archives of Psychiatric Nursing, Vol. XVII, No. 5 (October), 2003: pp 205-217
205
206
HOWARD ET AL.
of mental illness and for developing new treatment approaches (U. S. Department of Health and Human Services [USDHHS], 1999). Furthermore, the person-centered, humanitarian domain of clinical service research is important for understanding what consumers expect in treatment. Outcomes measurement can illuminate the humanistic aspects of service research by acknowledging the importance of the consumers’ opinions in the process of service system evaluation. The purposes of this report are to describe the findings of a study that investigated consumers’ perceptions of service quality in public-sector psychiatric hospitals, report on the feasibility of using recently developed instruments to measure consumer satisfaction and outcomes of mental health services, and determine whether patient characteristics were predictive of satisfaction with mental health services. LITERATURE REVIEW
Docherty and Streeter (1996) suggest that outcome measurement should include several dimensions, including symptoms, social and interpersonal functioning, work functioning, rates of treatment utilization, health status and perceptions of global well-being, health-related quality of life, and satisfaction with services. Service satisfaction is the outcome dimension most commonly used to determine consumers’ opinions about the services they receive. In their work with policy makers and researchers at the national level, consumers have advocated to include measurement of satisfaction as an indicator of service quality. Definitions of Satisfaction With Services According to Ruggeri (1994), the subjective nature of satisfaction makes it difficult to define; however, he acknowledges that this subjectivity can provide a “unique opportunity in service evaluation to take account of and examine the perspective of consumers” (p. 215). In general, a person’s perceptions of satisfaction are based on expectations, experiences, and results of overall treatment (Ruggeri, 1994). Lebow (1982) defined satisfaction as the “extent to which treatment gratifies the wants, wishes, and desires of clients for services” (p. 244). Likewise, Pascoe (1983) defined satisfaction as a cognitive and affective reaction to the context, process and results of a service encounter. Teague (2000) maintains that satisfaction research
addresses both the process and outcomes of care; the process dimension is measured by the respondents’ assessments of the interactions between patients and providers, and the outcomes dimension is measured by patients’ summary judgments of an overall episode of care. Ware and Snyder (1975) propose that satisfaction is a multidimensional construct that encompasses five domains: quality of care, convenience, finances, physical environment, and availability. Current researchers concur that satisfaction is a multidimensional construct, but further maintain that antecedent factors such as personal values, expectations, perceptions and previous experiences can influence levels of satisfaction and dissatisfaction (Mahon, 1996). Others suggest that service performance is an important indicator of service quality and that service quality is an antecedent to service satisfaction (deRuyter, Bloemer, & Peeters, 1997). National Initiatives to Measure Service Quality Movement towards a person-centered approach to evaluation research has led researchers and policy-makers to expand the concept of service quality beyond satisfaction and thereby identify and define broader domains that are important to consumers. Key activity in this movement took place in 1993, when the Center for Mental Health Services (CMHS) and the Mental Health Statistics Improvement Program (MHSIP) created a Task Force that consisted of consumers, family members, representatives of federal, state, and local mental health and substance abuse systems, advocacy groups, policy analysts, and service researchers (Center for Mental Health Services [CMHS], 1996). The primary focus of the Task Force was to develop a consumer-oriented report card with domains, indicators, and measures for adults with serious mental illness who receive treatment in community settings (CMHS, 1996; Teague et al., 1997). The report card was intended to help consumers, state agencies, and corporate purchasers compare the quality and cost of mental health and substance abuse services across healthcare organizations. The Task Force identified the domains of access to services, appropriateness, outcomes, and prevention as the major concerns of consumers and key indicators of service quality (Buckley, 1993;
CONSUMER PERSPECTIVES ON QUALITY
CMHS, 1996; Eisen, Shaul, Leff, Stringfellow, Clarridge, & Cleary, 2001; Manderscheid, Henderson, & Brown, 2000). The 40-item MHSIP Consumer Survey was developed as part of the report card to promote consumer assessment of these domains (Teague et al., 1997). Access refers to the degree to which services are quickly and readily obtainable, the responsiveness of the mental health system to individual and cultural needs, the availability of a wide array of relevant services, and lack of financial barriers to care. Appropriateness refers to the degree to which services are individualized to address a particular consumer’s strengths and weaknesses, cultural context, service preferences, and recovery goals. The broad domain of outcomes, which encompasses symptom reduction and functioning, addresses the extent to which services provided to individuals with emotional and behavioral disorders have a positive or negative effect on their well being, life circumstances, and capacity for self-management and recovery. According to the Task Force, satisfaction with services is directly related to consumer concerns about access, appropriateness, and outcomes, and is therefore implicit within those domains (CMHS, 1996). STUDY OVERVIEW
This study, which was conducted in a southeastern state of the U.S., is part of a larger collaborative research program to determine consumers’ perceptions of mental health service quality in a variety of the state’s public-sector treatment settings. Research team members have continuously evaluated and modified the study objectives and methods to better reflect the needs of consumers for quality improvement in service provision. Initial phases of the study (Howard et al., 2001) focused solely on consumer satisfaction as a dimension of service quality, and consumers functioned as data collectors in face-to-face taped interviews with participants (Howard & El-Mallakh, 2001). This initial study was replicated at a hospital in the central part of the state (Howard, Clark, Rayens, & Coyle, 2000). The replication study provided information from consumers about the ease of use and the reading level of the Kentucky Consumer Satisfaction Instrument (KY-CSI). Based on this information, tool modifications were made; the reading level was simplified to a 4th to 5th grade level while
207
retaining substantive content (see Table 1). In addition, the Likert scale was modified to include a 6-choice response set throughout the entire tool, and a 5-item qualitative comment section was added to the tool. The 5-item qualitative section provided participants with an opportunity to write comments about satisfaction, dissatisfaction, helpful and nonhelpful service experiences, and services not addressed in the scaled items. Subsequent phases of the study, conducted at the two hospitals reported here, used the modified KY-CSI (Table 1) and piloted the MHSIP 21-item Consumer Survey (Table 2). The MHSIP Consumer Survey used in the studies reported here was the 21-item version developed in the early stages of the CMHS-funded Five-State Feasibility Study (Ganju, 1999; Smith & Hopkins, 2000). Research Staff In the study reported here, two levels of research staff were at both research sites. Primary research staff consisted of recovered consumers of mental health services, who were recruited, hired and trained to work as Research Associates (RAs). A comprehensive report describing the RA training program’s didactic and lab components, quantitative and qualitative data collection techniques, reliability and validity issues, and fieldwork stages has been published (Howard & El-Mallakh, 2001). As described by Howard and El-Mallakh (2001), mental health consumers, rather than providers, were used as data collectors to enhance the validity of the study findings. The RAs’ primary responsibility was data collection on the inpatient units. In addition, they participated in regular research team meetings, met with research team members to discuss study progress and problems in day-to-day research activities, and worked with the Research Coordinators to review strategies for data collection and participant recruitment. The RAs also assumed responsibility for records management at both sites. Secondary research staff varied slightly between the two study sites. At hospital #1, secondary research staff included two Research Coordinators; one was a graduate student at the sponsoring university’s College of Nursing, and the other was a certified registered nurse. Secondary staff at hospital #2 consisted of two Research Coordinators; one was a staff member of the hospital’s Risk Management Department and the other was the
208
HOWARD ET AL.
Table 1. KY Consumer Satisfaction Instrument (KY-CSI)* Ordered by Section; the Number in Each Category is Presented, as Well as the Percentage of the Total (N ⴝ 204) Responses Section/Item
Environment The facility was clean. Staff explained my rights as a patient. I felt safe while I was in treatment. I had privacy. When I had a physical illness, I received care promptly. I received the services I needed to help me get better. Affiliation/esteem I understood what was expected of me during treatment. I was able to talk to staff about my problems. I was treated with respect. Staff were available to talk with me about my problems. I was able to easily contact my friends and family. I feel better about myself as a result of treatment. Growth/self-actualization I understood my treatment plan. Staff asked me what I thought would help me get better. Staff explained the effects of my medication. I had time to be with other patients. My friends and family were included in treatment planning when needed. I can deal with my problems better as a result of treatment. Overall Satisfaction I would come back to this facility if I needed treatment again.
Positive
Neutral
Negative
188 (92.2%) 158 (77.5%) 166 (81.4%) 151 (74.0%) 150 (73.5%) 172 (84.3%)
9 (4.4%) 25 (12.3%) 19 (9.3%) 19 (9.3%) 30 (14.7%) 20 (9.8%)
7 (3.4%) 21 (10.3%) 19 (9.3%) 34 (16.7%) 24 (11.8%) 12 (5.9%)
178 (87.3%) 173 (84.8%) 167 (81.9%) 170 (83.3%) 162 (79.4%) 172 (84.3%)
16 (7.8%) 12 (5.9%) 19 (9.3%) 17 (8.3%) 20 (9.8%) 19 (9.3%)
10 (4.9%) 19 (9.3%) 18 (8.8%) 17 (8.3%) 22 (10.8%) 13 (6.4%)
167 (82.3%) 151 (74.4%) 158 (77.8%) 191 (93.6%) 142 (70.0%) 171 (84.2%)
21 (10.3%) 24 (11.8%) 17 (8.4%) 8 (3.9%) 27 (13.3%) 23 (11.3%)
15 (7.4%) 28 (13.8%) 28 (13.8%) 5 (2.5%) 34 (16.8%) 9 (4.4%)
156 (76.9%)
14 (6.9%)
33 (16.3%)
*Item response options: 5, strongly agree; 4, agree; 3, I am neutral; 2, disagree; 1, strongly disagree; 0, not applicable.
Hospital’s Patient Advocate. The Research Coordinators supervised on-site study activities, assisted the RAs, participated in research team meetings, and were responsible for communicating with the Principal Investigator and the technical staff of the research team. METHODS
Study Design A simple survey design with nonrandom sampling technique was utilized at the two study sites. Hospital #1 serves a metropolitan area and a few rural counties; hospital #2 serves primarily a rural area. The study was conducted over a 5-month period at hospital #1 (January to May 1999), and a 3-month period at hospital #2 (March to May 1999). The sponsoring university’s Medical Institutional Review Board approved all study protocols and procedures for informed consent at both sites. Sample The sample consisted of eligible consumers hospitalized during the study period who agreed to
participate in the study. Inclusion criteria were: (1) 18 years of age and older; (2) currently hospitalized; and (3) identification by the treatment team as symptomatically stable and near discharge from the hospital. Exclusion criteria were: (1) identification by the treatment team as incapable of giving informed consent, (2) physical condition that precluded participation, and (3) forensic classification. At hospital #1, 152 eligible consumers were recruited, and of those, 107 agreed to participate, resulting in a response rate of 74%. At hospital #2, 167 were recruited, and 108 participants responded to the survey, resulting in a response rate of 65%. A validation item was included in the questionnaire to determine response set biases. Based on this validation item, a total of 204 surveys were determined to be acceptable for analysis. This included 103 surveys from hospital #1, and 101 surveys from hospital # 2. The analyses described here were conducted on the combined samples from the two state hospital sites. Combining the samples from the two sites is warranted for several reasons: (1) identical inclusion criteria and data collection procedures were
CONSUMER PERSPECTIVES ON QUALITY
209
Table 2. Mental Health Statistics Improvement Program (MHSIP) 21-Item Consumer Survey.* Ordered by Section; the Number in each Category is Presented as Well as the Percentage of the Total (N ⴝ 204) Responses Section/Item
Satisfaction I liked the services that I received. If I had other options, I would still choose to get services from this health plan. I would recommend this service to a friend or family member. Access The location of services was convenient. Staff caregivers were willing to help as often as I felt it was necessary.† My calls were returned within 24 hours. Services were available at times that were good for me. I was able to get all the services I thought I needed. Appropriateness Staff caregivers believed that I could grow, change, and recover.† I felt free to complain. I was told what side effects to watch for. My wishes about who is and is not to be given information about my treatment were respected. Staff caregivers were sensitive to my cultural/ethnic background.† Staff caregivers helped me obtain the information needed so I could take charge of managing my illness.† Outcomes I deal more effectively with daily problems. I am better able to control my life. I am better able to deal with crisis. I am getting along better with my family. I do better in social situations. I do better in school and/or work. My symptoms are not bothering me as much.
Positive
Neutral
Negative
166 (81.4%) 142 (69.6%) 157 (77.3%)
18 (8.8%) 22 (10.8%) 21 (10.3%)
20 (9.8%) 40 (19.6%) 25 (12.3%)
136 (67.3%) 162 (79.8%) 143 (71.9%) 161 (79.3%) 154 (76.2%)
31 (15.4%) 19 (9.4%) 38 (19.1%) 20 (9.9%) 21 (10.4%)
35 (17.3%) 22 (10.8%) 18 (9.1%) 22 (10.8%) 27 (13.4%)
174 (86.1%) 148 (73.3%) 149 (74.1%)
16 (7.9%) 20 (9.9%) 17 (8.5%)
12 (5.9%) 34 (16.8%) 35 (17.4%)
161 (80.1%) 161 (79.7%)
23 (11.4%) 25 (12.4%)
17 (8.5%) 16 (7.9%)
172 (85.6%)
14 (7.0%)
15 (7.5%)
163 (81.5%) 165 (82.1%) 153 (76.5%) 148 (73.3%) 149 (74.9%) 139 (69.5%) 161 (80.1%)
19 (9.5%) 21 (10.5%) 35 (17.5%) 33 (16.3%) 29 (14.6%) 41 (20.5%) 19 (9.5%)
18 (9.0%) 15 (7.5%) 12 (6.0%) 21 (10.4%) 21 (10.6%) 20 (10.0%) 21 (10.5%)
*Response options for all items: 5, strongly agree; 4, agree; 3, I am neutral; 2, disagree; 1, strongly disagree; 0, not applicable. †The word “Staff” was substituted for the word “My.”
used at the two sites, (2) relatively modest differences were found between the two samples, with no significant differences in any of the outcome variables; and (3) the combined sample provided a broader cross-section of the inpatient mental health consumers in this state. Sample Characteristics As described in Table 3, the majority of the 204 respondents in the combined sample were male (60.8%); 84.8% were European-American and the remainder was African-American. The average age of respondents was 37.4 years (SD⫽11.1). More than half of the respondents had at least a high school education (57.7%), and nearly one-fourth of respondents had some education beyond high school. The majority of respondents were unemployed (86.8%); of those who were employed, 10.8% had full- time employment, and 2.5% had part-time employment.
There were a variety of diagnoses on the DSM-IV Axis I in the combined sample (see Table 3); the most prevalent were mood disorder, schizophrenia, and substance abuse. Less frequent diagnoses included delusion/other psychoses, Alzheimer’s/organic brain disorders, impulse control disorders, anxiety disorders, attention deficit disorder, and other miscellaneous disorders. The majority of respondents (63.7%) were admitted to the hospitals on an involuntary basis. The average length of stay (LOS) was 31.0 days (SD⫽ 76.6) but an extremely long stay of nearly 3 years skewed the distribution. The median LOS for all respondents in the combined sample was 19.5 days. Slightly more than one-fourth of the combined sample reported that the current hospitalization was either their first or second; the remaining 71.1% reported at least two previous hospitalizations. Like LOS, the number of previous hospitalizations was influenced by a few observations on
210
HOWARD ET AL.
Table 3. Demographic Characteristics of the Sample (N ⴝ 204) Variable
Gender Male Female Ethnicity European-American African-American Education Less than HS diploma High School diploma At least 1 year post HS Employment status Employed full-time Employed part-time Unemployed Primary diagnosis (DSM-IV) Mood disorders Schizophrenia Substance abuse Delusion/other psychoses Alz/organic brain Impulse control disorder Anxiety Attention deficit disorder Other disorder Admission status Voluntary Involuntary Previous hospitalizations 0 1 2 ⬎2
Frequency (n)
Percent (%)
124 80
60.8 39.2
173 31
84.8 15.2
86 73 44
42.4 36.0 21.7
22 5 177
10.8 2.5 86.8
87 51 24 10 8 7 5 3 9
42.7 25.0 11.8 4.9 3.9 3.4 2.5 1.5 4.4
74 130
36.3 63.7
25 34 35 110
12.3 16.7 17.2 53.9
The sample was similar to the overall population at the two sites on gender, race, admission status, and diagnosis of substance abuse. However, compared with the hospital population, the sample was over represented with respondents with mood disorders and under represented with respondents with schizophrenia and psychotic disorders. Instruments Self-Administered Questionnaire The self-administered questionnaire was organized around the 19-item KY-CSI, the 21-item MHSIP Consumer Survey, the 8-item Consumer Satisfaction Questionnaire (CSQ-8), and a single Quality of Mental Health Care (QMHC) item. The Kentucky Consumer Satisfaction Instrument (KY-CSI)
the high end. The median number of hospitalizations prior to the current one was 2, with a range of 0 to 60.
The KY-CSI, which is based on Maslow’s motivation and human needs theory, contains 19 scaled-response items (Table 1). Response choices for the 19 items range from 5 (“strongly agree”) to 1 (“strongly disagree”); a “not applicable” response choice is also included. As indicated in Table 1, the tool contains six items related to the physical environment, six related to affiliation, and six related to goal attainment and self-actualization. One item questions whether the respondent would return to the facility if services were needed in the future. Potential scores for the KY-CSI range from 19 to 95 (Table 4), with higher scores indicating higher levels of satisfaction. The potential ranges for the sub-scales are summarized in Table 4.
Table 4. Descriptive Analysis and Internal Consistency of the Satisfaction Scales, Subscales, and Single Item (N ⴝ 204) Scale
Number of Items
Mean (SD)
Actual Range
Potential Range
Cronbach’s alpha
KY-CSI Environment Affiliation Growth Would return CSQ-8 MHSIP Satisfaction Accessibility Appropriateness Outcome QMHC
19 6 6 6 1 8 21 3 5 6 7 1
77.3 (13.2) 24.4 (4.5) 24.6 (4.5) 24.2 (4.3) 4.0 (1.3) 25.7 (5.7) 83.2 (15.4) 11.7 (3.0) 19.5 (4.2) 24.0 (4.8) 27.9 (5.4) 7.7 (2.4)
28-95 7-30 10-30 10-30 1-5 10-32 35-105 3-15 5-25 8-30 13-35 0-10
19-95 6-30 6-30 6-30 1-5 8-32 21-105 3-15 5-25 6-30 7-35 0-10
0.95 0.86 0.87 0.84 0.94 0.96 0.88 0.87 0.89 0.91 –
CONSUMER PERSPECTIVES ON QUALITY
The Mental Health Statistics Improvement Program (MHSIP) 21-Item Consumer Survey Since its inception, the original 40-item Consumer Survey developed by the MHSIP Task Force has undergone continuous psychometric testing, evaluation and revision. Psychometric evaluation of the 40-item tool conducted by members of the CMHS-funded Five-State Feasibility Study and the Behavioral Health Measurement Advisory Panel (MAP) of the National Committee on Quality Assurance (NCQA) resulted in the 21-item version that was used in this study (Ganju, 1999). As illustrated in Table 2, the instrument measures consumers’ perceptions of access, appropriateness, outcomes of service, and general satisfaction (Teague et al., 1997). Response choices range from 5 (“strongly agree”) to 1 (“strongly disagree”), with an additional response choice of “not applicable.” Potential scores for the MHSIP 21-Item Consumer Survey range from 21 to 105 (Table 4), with higher scores indicating a higher rating of the quality of services. The range of possible subscale scores is summarized in Table 4. The MHSIP was used in this study to initiate hospital-based psychometric testing and determine the feasibility of using the tool to evaluate outcomes of inpatient care. The Client Satisfaction Questionnaire (CSQ-8) The CSQ-8 is a unidimensional survey that measures global satisfaction with mental health services (Attkinsson & Greenfield, 1996). It contains eight scaled-response items that measure the quality of services received, the degree to which the respondent received services that were needed and wanted, satisfaction with the amount of services received, the degree to which treatment was effective, and whether the respondent would recommend services to a friend or return for services if needed in the future. Response choices range from 4 (“highest satisfaction with services”) to 1 (“lowest satisfaction with services”); the total score has a potential range of 8 to 32, with higher scores indicating a higher level of satisfaction (Table 4). Evidence for high internal consistency of the CSQ-8 is suggested by Cronbach’s alphas of 0.83 to 0.93 (Sederer & Dickey, 1996). The CSQ-8 was included in this study for the purposes of psychometric evaluation of the KY-CSI and MHSIP surveys.
211
Single-Item Quality Rating In addition to the three instruments, a single Quality of Mental Health Care (QMHC) item was added to measure respondents’ overall perception of quality of mental health care. Response choices for the QMHC ranged from 0 (“worst possible care”) to 10 (“best possible care”). Procedures Hospitalized consumers who met study inclusion criteria were approached by RAs to determine their interest in participating in the study. Once verbal consent was obtained, RAs reviewed the consent form with consumers and obtained signatures indicating informed consent. Participants were given a copy of the consent form. After informed consent was obtained, RAs met with the participants in a private area on the unit to give them the survey packet and review the instructions for completing the self-administered questionnaire. RAs read the text of the consent form and questionnaire to participants who were unable to read. It took approximately 15 to 20 minutes for participants to independently complete the self-administered questionnaire. Data Analysis Total scores for each of the three satisfaction instruments were obtained by summing the corresponding items contained in each scale. For each scale, missing data were filled in using mean substitution in cases where the number of items missing for a given respondent was at most 25% of the items in the scale. Mean substitution is the practice of substituting the average value from nonmissing items for missing items. The typical rule of thumb is that this method should only be used to fill in a limited number of missing values and that if the number of missing values is more than 25% of the total items for the scale on a given survey, the summary score should not be calculated for that respondent. In addition, subscale scores were obtained by summing responses on the appropriate items. Because of the lack of normality of the distributions of hospital length of stay and number of previous hospitalizations, these variables were transformed using the square root transformation before being included in the bivariate and multiple regression analyses. This transformation is appropriate for both of these variables since it minimizes
212
the impact of outliers on the high end of the distribution. Univariate Analysis The satisfaction measures and the sociodemographic characteristics were summarized using univariate analysis by calculating means and standard deviations for continuous variables and frequency distributions for categorical variables. Bivariate Analyses Comparison of the two hospital sites on demographic and satisfaction variables was accomplished using two-sample t-tests and chi-square tests of association. In the case of small expected cell counts within the chi-square test of association, Fisher’s exact test was used as an appropriate alternative. Bivariate analyses were also conducted to ascertain the relationships among and between the satisfaction measures and the sociodemographic variables. These analyses consisted of Pearson’s product moment correlations for pairs of continuous variables, chi-square tests of association or Fisher’s exact test for pairs of categorical variables, and one-way analysis of variance (ANOVA) to assess the relationships between continuous and categorical variables. In the case of one-way ANOVA, Fisher’s least significant difference procedure was used to determine whether pair-wise comparisons were significant. Cronbach’s alphas were calculated for the satisfaction scales and subscales to assess internal consistency. Multiple Regression Multiple regression was used to determine whether any of the measures of satisfaction could be predicted using respondent characteristics. A backward stepwise procedure was used to reduce the full set of regressors to only those that were significant predictors. For each satisfaction measure, the initial model contained the potential predictors of age, ethnicity (European-American versus African-American), gender, admission status (voluntary versus involuntary), education (less than high school versus high school and above), employment status (employed versus unemployed), the square root of length of stay, the square root of number of previous hospitalizations, and hospital site (hospital # 1 versus hospital # 2).
HOWARD ET AL.
RESULTS
Instruments Findings indicate good internal consistency of the instruments used in the study (Table 4). Cronbach’s alpha for the overall KY-CSI was .95, and subscale Cronbach’s alphas were .84 and above. For the 21-item MHSIP, the Cronbach’s alpha for the overall scale was .96, and subscale alphas were .88 and above. Cronbach’s alpha for the CSQ-8 was .94. These findings suggest that reliable tools were used in the study. Descriptive Analysis of the Satisfaction Measures All of the satisfaction scales and subscales, the QMHC item, and the item questioning the likelihood of choosing the same facility in the future, showed relatively high mean values when averaged across respondents. For example, the QMHC item, which measured respondents’ perceptions of the overall quality of their care, had an average score of 7.7 out of a possible score of 10. Table 4 displays the mean and standard deviation for each of these measures of satisfaction and overall quality of care. The actual range and potential range of satisfaction scores are also displayed, along with the Cronbach’s alphas for the multi-item scales and subscales. Satisfaction With Services For items on the KY-CSI and MHSIP, 70% or more respondents indicated that they were satisfied with the services they had received during their hospitalization (Tables 1 and 2). Specifically, areas of greatest satisfaction included the time available to be with other patients (94%) and cleanliness of the facilities (92%). In addition, staff-client relationships were a major source of satisfaction among respondents. Staff members were available to speak with patients (83%), and respondents were comfortable talking to staff about their problems and issues (85%). In the majority of cases, staff members provided information that respondents needed to “take charge” of their illnesses (86%), and respondents received the services that they needed to get better (84%). The majority of respondents indicated that they understood what was expected of them during treatment (87%), and they perceived that staff had confidence in their ability to grow, change, and recover (86%). Furthermore,
CONSUMER PERSPECTIVES ON QUALITY
84% of respondents reported that they felt better about themselves as a result of treatment. Despite the overall high ratings of satisfaction, up to 20% of respondents reported some dissatisfaction with the services they received (Tables 1 and 2). For example, on the KY-CSI, 33 (16.3%) respondents indicated that they would not return to the facility if they needed services in the future. Similarly, on the MHSIP, 40 (19.6%) respondents indicated that if they had other options they would not choose this health plan in the future. On the KY-CSI, 28 (13.8%) respondents reported that they did not receive medication education. This is supported by findings of the MHSIP, in which 35 (17.4%) respondents reported that they were not told what side effects to look for. Treatment planning issues were also a source of dissatisfaction for some respondents. Almost 14% reported that staff did not ask them what they thought would help them get better (Table 1) and almost 17% indicated that they did not feel free to complain (Table 2). Respondents also reported dissatisfaction with family involvement in their care; almost 17% indicated that their family and friends were not included in treatment planning when needed (Table 1). An additional area of dissatisfaction (Table 1) included lack of privacy in the inpatient setting (16.7%). Furthermore, access to care was a concern; 35 (17.3%) respondents stated that the location of services was inconvenient (Table 2). Bivariate Relationships Between Satisfaction and Demographic Characteristics Correlation analysis indicated a high degree of association among the various measures of satisfaction. For all scales and subscales, correlation coefficients ranged from r ⫽ 0.5 to r ⫽ 1.0 (p⬍ .0001), and the majority of the coefficients were in the 0.7 to 0.8 range. None of the satisfaction measures were significantly related to age, gender, education, DSM-IV Axis I diagnosis, or the square root of the length of stay. The number of previous hospitalizations was significantly negatively correlated with several measures of satisfaction, including the Affiliation subscale of the KY-CSI (r ⫽ -.18; p ⫽ .01), the CSQ-8 (r ⫽ -.15; p ⫽ .04), the Appropriateness subscale of the MHSIP (r ⫽ -.15;
213
p ⫽ .04), and the single item QMHC (r ⫽ -.18; p ⫽ .01). ANOVA calculations indicated that the Outcome subscale of the MHSIP differed by ethnicity, with African-Americans rating these items higher than European-Americans (the means were 30.1 and 27.7 respectively, with t ⫽ 2.5 and p ⫽ .02). The Satisfaction subscale of the MHSIP also differed according to admission status, with significantly higher scores among voluntary admissions (the means for voluntary and involuntary admissions were 12.4 and 11.2 respectively, with t ⫽ 2.9; p ⫽ .004). Employed respondents rated the Growth/Self-Actualization subscale of the KY-CSI higher than those who were unemployed (the means were 5.9 and 4.7 respectively, with t ⫽ 2.1 and p ⫽ .04). Predictors of Satisfaction The models for the Environment and Affiliation subscales of the KY-CSI contained no significant predictors of consumer satisfaction. Similarly, there were no significant predictors of the CSQ-8, the Access subscale of the MHSIP, and the single item QMHC. The final models for each of the other measures of satisfaction are displayed in Table 5. Although employed respondents rated their satisfaction with services higher than unemployed respondents in the ANOVA calculations, employment was a significant negative predictor of satisfaction in the regression model. Other significant negative predictors of satisfaction included male gender and European-American ethnicity (Table 5). In contrast, a voluntary admission status may bolster satisfaction. The low R2 values (Table 5) for the regression models indicated that although some of these patient characteristics are significant predictors, they account for a very small portion of the variability in satisfaction. For example, as indicated in Table 5, only 2% of the variability in satisfaction, as scored on the MHSIP Appropriateness Subscale is accounted for by European-American ethnicity. In addition, the small values for R2 may be an artifact of the relatively low variability in satisfaction scores as a result of high satisfaction among nearly all participants.
214
HOWARD ET AL. Table 5. Final Models for Each of the Satisfaction Measures With a Significant Regression (N ⴝ 204) Satisfaction Measure
KY-CSI KY-CSI growth KY-CSI would return MHSIP MHSIP satisfaction MHSIP appropriateness MHSIP outcome
Significant Predictors
Standardized Betas
p Value
Model R2
Male Employed Male Employed Voluntary admit European American Voluntary admit Employed European American European American
⫺.15 ⫺.15 ⫺.20 ⫺.18 .19 ⫺.16 .24 ⫺.16 ⫺.15 ⫺.21
.04 .04 .005 .01 .01 .03 .0009 .03 .04 .005
0.04
DISCUSSION
The Consumer Research Associate Program Research Associates, who were recovered mental health consumers, assumed major responsibility for data collection in this study. Several findings support this role in service system evaluation research. In this study, respondents independently completed the self-administered survey in about 20 minutes, whereas in the pilot study, the face-toface interviews lasted about one hour (Howard et al., 2001). The reduced time for survey completion suggests that the RAs were thorough in giving participants instructions and that the self-report method of data collection imposed less burden on respondents compared with the face-to-face interview. Shorter completion time may also reduce costs for providers and institutions in outcomes assessment programs. Moreover, survey research typically yields low response rates (Levy & Lemeshow, 1999) yet the response rate for this study was approximately 70%. Furthermore, whereas 215 surveys were administered, a total of 204 were retained in the analysis. This high rate of correct completion of the surveys is an additional indication that RAs provided respondents with clear instructions about the study purpose and procedures for survey completion. Both “unit nonresponse” and “item nonresponse” are “very difficult to avoid in the sampling of populations” (Levy & Lemeshow, 1999, p. 393). RAs helped offset potential study limitations by boosting both the unit and item response rates and by contributing to the collection of quality data. The nonrandom sampling technique possibly contributed to the high response rate but it was also a study limitation. In our sample, respondents with
0.07 0.04 0.02 0.08 0.02 0.04
mood disorders were over represented and respondents with schizophrenia and psychotic disorders were under represented when compared with the hospital populations. Study findings must be interpreted in light of this limitation. Service Satisfaction and Dissatisfaction In general, whether findings were viewed within the context of one instrument or collectively, the degree of satisfaction with mental health services appeared to be quite high. Viewed collectively, findings suggest the presence of a therapeutic, trusting, reciprocal relationship between staff and respondents. In addition, respondents reported high levels of satisfaction with staff-client relationships, staff availability, and respondents’ ability to talk to staff members about their problems. Respondents were also highly satisfied with several aspects of service delivery. Given the high degree of satisfaction with staff-client relationships and delivery of needed services, it is not surprising to find that the majority of respondents reported feeling better about themselves as a result of treatment. Service Dissatisfaction A limitation of research about consumer satisfaction concerns lack of attention to areas of dissatisfaction. Moreover, it is important to understand reasons for dissatisfaction in the public mental health systems since it serves as the “safety net” for most individuals with mental health problems. We explored items in which respondents indicated that they “strongly disagreed” or “disagreed” to determine areas that might warrant improvement. Some items related to service dissatisfaction were noteworthy. As many as 40 respondents indicated that they would not return to the facility or choose the health
CONSUMER PERSPECTIVES ON QUALITY
plan in the future. Contributing factors may have been related to perceptions about exclusion from treatment planning. For instance, despite the current emphasis on consumer-driven care, findings suggest that respondents had limited input into their treatment planning. Specifically, several respondents indicated that they were not asked what would help them get better. Perceptions of satisfaction are based on the consumers’ expectations and desires, as well as their assessment of their interactions with providers (Lebow, 1982; Ruggeri, 1994; Teague, 2000). More important, providers who are not aware of what consumers expect in treatment may not maintain a focus on client-centered treatment goals, despite their good intentions. Other items of concern were lack of education about medications and treatment side effects (Tables 1 & 2). Lack of education and information are cause for even greater concern when coupled with lack of involvement of family and friends in the process of treatment planning. A primary goal of treatment is to promote recovery and self-care in the least restrictive setting. When consumers and their family caregivers do not understand treatment or do not know about medication side effects, recovery is compromised and the consumer is at risk for relapse and rehospitalization. Finally, the findings about areas of dissatisfaction raise questions about a continued emphasis on the provider’s perspectives in the process of making decisions about the clients’ plan of care. Domains of Satisfaction Results of the multiple regression analysis provide some evidence to suggest that satisfaction with mental health services may be lower among males, European-Americans, and those who are employed. Prior satisfaction research has shown a lack of consistency in elucidating the relationships between sociodemographic characteristics and satisfaction with mental health services. However, Rosenheck, Wilson, and Merketo (1997) conducted a descriptive study to investigate patient satisfaction in a sample of veterans with a primary psychiatric diagnosis who had been recently discharged from an inpatient setting. Their findings indicated that African-Americans reported less satisfaction than European-Americans on a measure of courtesy, but they reported more satisfaction compared with European-Americans on measures
215
of emotional support, physical comfort, involvement, and practical orientation. Culture and ethnicity will dominate concerns in the delivery of mental health services in the U.S. in the future because of changing demographics. Therefore, research about the relationships among and between these variables is crucial. CONCLUSIONS
This study shows how the person-centered focus of the consumer movement was incorporated in a collaborative project between University researchers and a state’s Department for Mental Health and Mental Retardation Services. Findings were promising in several ways. First, the consumers represented by the respondents of this study indicated that public-sector psychiatric hospitals provide quality services. Second, the consumer RAs were valuable members of the research teams at both hospital sites. Third, preliminary findings suggest that both the KY-CSI and 21-item MHSIP are reliable instruments and are feasible for use in inpatient settings. Additional psychometric studies to ascertain the reliability and validity of the 21item version and any other versions of the MHSIP Consumer Survey are recommended. Similar studies about the KY-CSI are warranted not only in inpatient but also in community settings. To offset limitations related to representative samples, we recommend stratified random sampling techniques, when feasible, in evaluation research on consumer perspectives of services. The person-centered focus in mental health services can be facilitated by conducting evaluation studies that explore the consumers’ perspective and by using instruments that were developed with consumer input. Furthermore, inclusion of consumers in the research process is crucial for understanding ways to improve public and private systems of care. ACKNOWLEDGMENT The authors thank those who kindly volunteered to participate in the study and the Research Associates who collected the data. They also thank Commissioner Margaret Pennington of the KDMHMRS; the administrators and staff of the hospitals; the Research Coordinators, Clara Furlong and Connie Smedley; the Research Assistants, Deborah S. Armstrong and Barbara Slater; W. Russell Coyle for technical assistance; Ronald W. Manderscheid, of the Survey and Analysis Branch, Center for Mental Health Services, Rif S. El-Mallakh, M.D. for reviewing the manuscript; and Jewel D. Begley for manuscript preparation.
216
HOWARD ET AL.
REFERENCES Attkinsson, C.C., & Greenfield, T.K. (1996). The Client Satisfaction Questionnaire (CSQ) Scales and the Service Satisfaction Scale-30 (SSS-30). In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp.120-127). Baltimore, MD: Williams and Wilkins. Auslander, M., Moses, D.J., Granger, D., Henderson, M.J., Johnson, J., & Johnson, J.R., et al. (1998). Consumer issues in managed behavioral health care. Journal of the Washington Academy of Sciences, 85(1), 15-27. Buckley, S.M. (1993). Moving MHSIP towards a person-centered paradigm: A concept paper submitted to CMHS and the MHSIP ad hoc advisory group. Available at: http://www.mhsip.org/documents/moving.htm. Accessed May 16, 2001. Campbell, J. (1998). Consumerism, outcomes, and satisfaction: A review of the literature. In R.W. Manderscheid & M.J. Henderson (Eds.), Mental Health United States, 1998 (pp. 11-28). Center for Mental Health Services, DHHS Pub. No. (SMA) 99-3285. Washington, DC: Supt. Of Docs., U. S. Govt. Print. Off. Center for Mental Health Services (1996). The MHSIP Consumer-Oriented Mental Health Report Card: Final report of the Mental Health Statistics Improvement Program (MHSIP) Task Force on a consumer-oriented mental health report card. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service, Substance Abuse and Mental Health Services Administration. Chisholm, M., Howard, P.B., Boyd, M.A., Clement, J.A., Hendrix, M.J., & Reiss-Brennan, B. (1997). Quality indicators for primary mental health within managed care: A public health focus. Archives of Psychiatric Nursing, 11(4), 167-181. deRuyter, K, Bloemer, J, & Peeters, P. (1997). Merging service quality and service satisfaction: An empirical test of an integrative model. Journal of Economic Psychology, 18, 387-406. Docherty, J.P., & Streeter, M.J. (1996). Measuring outcomes. In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp. 8-18). Baltimore, MD: Williams & Wilkins. Eisen, S.V., Shaul, J.A., Leff, H.S., Stringfellow, V., Clarridge, B.R., & Cleary, P.D. (2001). Toward a national consumer survey: Evaluation of the CABHS and MHSIP instruments. Journal of Behavioral Health Services and Research, 28(3), 347-369. Ganju, V. (1999). The MHSIP consumer survey: History, development, revisions, applications, commonly-asked questions. Available at: http://www.mhsip.org/documents/ MHSIPConsumerSurvey.pdf. Accessed May 3, 2001. Howard, P.B., Clark, J.J., Rayens, M.K., & Coyle, R. (2000). Satisfaction as an outcome indicator of quality in mental health services at Central State Hospital. (KDMHMRS Technical Report Series). Lexington, KY: Author. Howard, P.B., Clark, J.J., Rayens, M.K., Hines-Martin, V., Weaver, P., & Littrell, R. (2001). Consumer satisfaction
with services in a regional psychiatric hospital: A collaborative research project in Kentucky. Archives of Psychiatric Nursing, 15(1), 10-23. Howard, P.B., & El-Mallakh, P. (2001). Training consumers to collect data in mental health service system evaluation research. Journal of Psychosocial Nursing, 39(5), 3039. Jennings, B.M., Staggers, N., & Brosch, L.R. (1999). A classification scheme for outcome indicators. Image: Journal of Nursing Scholarship, 31(4), 381-388. Kolb, S.J., Race, K.E., & Seibert, J.H. (2000). Psychometric evaluation of inpatient psychiatric care consumer satisfaction survey. The Journal of Behavioral Health Services and Research, 27(1), 75-86. Lebow, J. (1982). Consumer satisfaction with mental health treatment. Psychological Bulletin, 91(2), 244-259. Levy, P.S., & Lemeshow, S. (1999). Sampling of populations: Methods and applications (3rd ed). New York, NY: John Wiley & Sons. Mahon, P.Y. (1996). An analysis of the concept ‘patient satisfaction’ as it relates to contemporary nursing care. Journal of Advanced Nursing, 24, 1241-1248. Manderscheid, R.W., Henderson, M.J., & Brown, D.Y. (2001). Status of national accountability efforts at the millennium. In R.W. Manderscheid & M.J. Henderson (Eds.), Mental Health United States, 2000 (pp.45-54). DHHS Supt. of Docs. Publication No. (SMA) 01-3537. Washington, DC: U. S. Government Printing Office. Pascoe, G.C. (1983). Patient satisfaction in primary health care: A literature review and analysis. Evaluation and Program Planning, 6, 185-210. Rosenheck, R., Wilson, N.J., & Meterko, M. (1997). Influence of patient and hospital factors on consumer satisfaction with inpatient mental health treatment. Psychiatric Services, 48(12), 1553-1561. Rosswurm, M.A., & Larrabee, J.H. (1999). A model for change to evidence-based practice. Image: Journal of Nursing Scholarship, 31(4), 317-322. Ruggeri, M. (1994). Patients’ and relatives’ satisfaction with psychiatric services: The state of the art of its measurement. Social Psychiatry and Psychiatric Epidemiology, 29, 212-227. Sederer, L.I., Dickey, B., & Hermann, R.C. (1996). The imperative of outcomes assessment in psychiatry. In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp. 1-7). Baltimore, MD: Williams & Wilkins. Smith, M.E., & Hopkins, C. (2000). Report on the MHSIP Survey Workgroup Meeting. Paper presented at the 49th Annual Conference of the Center for Mental Health Services and Mental Health Statistics Improvement Program, Washington, DC. Teague, G.B. (2000). Patient perceptions of care measures. In Handbook of Psychiatric Measures (pp. 169-194). Washington, DC: American Psychiatric Association. Teague, G.B., Ganju, V., Hornik, J.A., Johnson, J.R., & McKinney, J. (1997). The MHSIP Mental Health Report Card: A consumer-oriented approach to monitoring the quality of mental health plans. Evaluation Review, 21(3), 330-341.
CONSUMER PERSPECTIVES ON QUALITY
U. S. Department of Health and Human Services. (1999). Mental Health: A Report of the Surgeon General-Executive Summary. Rockville, MD: U. S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental
217
Health Services, National Institutes of Health, National Institute of Mental Health. Ware, J.E., & Snyder, M.K. (1975). Dimensions of patient attitudes regarding doctors and medical care services. Medical Care, 13(8), 669-682.
C
HANGES WITHIN THE health care system have prompted health care providers and researchers to re-evaluate and modify the process of service system evaluation. Evaluation research conducted throughout the 1980s focused primarily on the structure and processes of health care. In contrast, evaluation research shifted its focus to the measurement of treatment outcomes during the 1990s (Jennings, Staggers, & Brosch, 1999; Sederer, Dickey, & Hermann, 1996). Issues that drove the outcomes measurement movement included escalating health care costs, consumer advocacy for needed and appropriate services, increased focus on evidence-based practice, implementation of managed care in both the private and public mental health care systems, and development of new service delivery systems (Auslander et al., 1998; Campbell, 1998; Chisholm et al., 1997; Kolb, Race, & Seibert, 2000; Rosswurm & Larrabee, 1999; Teague, Ganju, Hornik, Johnson, & McKinney, 1997). Buckley (1993) conceptualizes the evolution of evaluation research as a decreased focus on the service delivery organization and a simultaneous emergence of a person-centered focus. This evolution represents a shift away from “persons served within individual specialty mental health organiza-
tions towards persons with significant needs for mental health services and supports, regardless of the number or type of organizations” (p. 1). Likewise, Campbell (1998) notes that early research efforts evaluated mental health service systems according to “measures of output and volume” (p. 13), but states that current evaluation research increasingly emphasizes patients’ values and “broadened measurements of outcomes that encompass not only symptom reduction, but functional status and quality of life” (p. 13). Basic research in the fields of neuroscience and genetics are crucial for understanding the etiology From the College of Nursing, University of Kentucky, Lexington, KY and College of Social Work, University of Kentucky, Lexington, KY. Supported by the Kentucky Department for Mental Health and Mental Retardation Services (KDMHMRS) and, in part, with a grant from the Department of Health and Human Services, Public Health Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services (4 HRI SM52058-02-4). Address reprint requests to Patricia B. Howard, PhD, RN, CNAA, College of Nursing, University of Kentucky, 760 Rose Street, Lexington, KY 40536-0232. E-mail: [email protected] 䊚 2003 Elsevier Inc. All rights reserved. 0883-9417/03/1705-0003$30.00/0 doi:10.1053/S0883-9417(03)00096-7
Archives of Psychiatric Nursing, Vol. XVII, No. 5 (October), 2003: pp 205-217
205
206
HOWARD ET AL.
of mental illness and for developing new treatment approaches (U. S. Department of Health and Human Services [USDHHS], 1999). Furthermore, the person-centered, humanitarian domain of clinical service research is important for understanding what consumers expect in treatment. Outcomes measurement can illuminate the humanistic aspects of service research by acknowledging the importance of the consumers’ opinions in the process of service system evaluation. The purposes of this report are to describe the findings of a study that investigated consumers’ perceptions of service quality in public-sector psychiatric hospitals, report on the feasibility of using recently developed instruments to measure consumer satisfaction and outcomes of mental health services, and determine whether patient characteristics were predictive of satisfaction with mental health services. LITERATURE REVIEW
Docherty and Streeter (1996) suggest that outcome measurement should include several dimensions, including symptoms, social and interpersonal functioning, work functioning, rates of treatment utilization, health status and perceptions of global well-being, health-related quality of life, and satisfaction with services. Service satisfaction is the outcome dimension most commonly used to determine consumers’ opinions about the services they receive. In their work with policy makers and researchers at the national level, consumers have advocated to include measurement of satisfaction as an indicator of service quality. Definitions of Satisfaction With Services According to Ruggeri (1994), the subjective nature of satisfaction makes it difficult to define; however, he acknowledges that this subjectivity can provide a “unique opportunity in service evaluation to take account of and examine the perspective of consumers” (p. 215). In general, a person’s perceptions of satisfaction are based on expectations, experiences, and results of overall treatment (Ruggeri, 1994). Lebow (1982) defined satisfaction as the “extent to which treatment gratifies the wants, wishes, and desires of clients for services” (p. 244). Likewise, Pascoe (1983) defined satisfaction as a cognitive and affective reaction to the context, process and results of a service encounter. Teague (2000) maintains that satisfaction research
addresses both the process and outcomes of care; the process dimension is measured by the respondents’ assessments of the interactions between patients and providers, and the outcomes dimension is measured by patients’ summary judgments of an overall episode of care. Ware and Snyder (1975) propose that satisfaction is a multidimensional construct that encompasses five domains: quality of care, convenience, finances, physical environment, and availability. Current researchers concur that satisfaction is a multidimensional construct, but further maintain that antecedent factors such as personal values, expectations, perceptions and previous experiences can influence levels of satisfaction and dissatisfaction (Mahon, 1996). Others suggest that service performance is an important indicator of service quality and that service quality is an antecedent to service satisfaction (deRuyter, Bloemer, & Peeters, 1997). National Initiatives to Measure Service Quality Movement towards a person-centered approach to evaluation research has led researchers and policy-makers to expand the concept of service quality beyond satisfaction and thereby identify and define broader domains that are important to consumers. Key activity in this movement took place in 1993, when the Center for Mental Health Services (CMHS) and the Mental Health Statistics Improvement Program (MHSIP) created a Task Force that consisted of consumers, family members, representatives of federal, state, and local mental health and substance abuse systems, advocacy groups, policy analysts, and service researchers (Center for Mental Health Services [CMHS], 1996). The primary focus of the Task Force was to develop a consumer-oriented report card with domains, indicators, and measures for adults with serious mental illness who receive treatment in community settings (CMHS, 1996; Teague et al., 1997). The report card was intended to help consumers, state agencies, and corporate purchasers compare the quality and cost of mental health and substance abuse services across healthcare organizations. The Task Force identified the domains of access to services, appropriateness, outcomes, and prevention as the major concerns of consumers and key indicators of service quality (Buckley, 1993;
CONSUMER PERSPECTIVES ON QUALITY
CMHS, 1996; Eisen, Shaul, Leff, Stringfellow, Clarridge, & Cleary, 2001; Manderscheid, Henderson, & Brown, 2000). The 40-item MHSIP Consumer Survey was developed as part of the report card to promote consumer assessment of these domains (Teague et al., 1997). Access refers to the degree to which services are quickly and readily obtainable, the responsiveness of the mental health system to individual and cultural needs, the availability of a wide array of relevant services, and lack of financial barriers to care. Appropriateness refers to the degree to which services are individualized to address a particular consumer’s strengths and weaknesses, cultural context, service preferences, and recovery goals. The broad domain of outcomes, which encompasses symptom reduction and functioning, addresses the extent to which services provided to individuals with emotional and behavioral disorders have a positive or negative effect on their well being, life circumstances, and capacity for self-management and recovery. According to the Task Force, satisfaction with services is directly related to consumer concerns about access, appropriateness, and outcomes, and is therefore implicit within those domains (CMHS, 1996). STUDY OVERVIEW
This study, which was conducted in a southeastern state of the U.S., is part of a larger collaborative research program to determine consumers’ perceptions of mental health service quality in a variety of the state’s public-sector treatment settings. Research team members have continuously evaluated and modified the study objectives and methods to better reflect the needs of consumers for quality improvement in service provision. Initial phases of the study (Howard et al., 2001) focused solely on consumer satisfaction as a dimension of service quality, and consumers functioned as data collectors in face-to-face taped interviews with participants (Howard & El-Mallakh, 2001). This initial study was replicated at a hospital in the central part of the state (Howard, Clark, Rayens, & Coyle, 2000). The replication study provided information from consumers about the ease of use and the reading level of the Kentucky Consumer Satisfaction Instrument (KY-CSI). Based on this information, tool modifications were made; the reading level was simplified to a 4th to 5th grade level while
207
retaining substantive content (see Table 1). In addition, the Likert scale was modified to include a 6-choice response set throughout the entire tool, and a 5-item qualitative comment section was added to the tool. The 5-item qualitative section provided participants with an opportunity to write comments about satisfaction, dissatisfaction, helpful and nonhelpful service experiences, and services not addressed in the scaled items. Subsequent phases of the study, conducted at the two hospitals reported here, used the modified KY-CSI (Table 1) and piloted the MHSIP 21-item Consumer Survey (Table 2). The MHSIP Consumer Survey used in the studies reported here was the 21-item version developed in the early stages of the CMHS-funded Five-State Feasibility Study (Ganju, 1999; Smith & Hopkins, 2000). Research Staff In the study reported here, two levels of research staff were at both research sites. Primary research staff consisted of recovered consumers of mental health services, who were recruited, hired and trained to work as Research Associates (RAs). A comprehensive report describing the RA training program’s didactic and lab components, quantitative and qualitative data collection techniques, reliability and validity issues, and fieldwork stages has been published (Howard & El-Mallakh, 2001). As described by Howard and El-Mallakh (2001), mental health consumers, rather than providers, were used as data collectors to enhance the validity of the study findings. The RAs’ primary responsibility was data collection on the inpatient units. In addition, they participated in regular research team meetings, met with research team members to discuss study progress and problems in day-to-day research activities, and worked with the Research Coordinators to review strategies for data collection and participant recruitment. The RAs also assumed responsibility for records management at both sites. Secondary research staff varied slightly between the two study sites. At hospital #1, secondary research staff included two Research Coordinators; one was a graduate student at the sponsoring university’s College of Nursing, and the other was a certified registered nurse. Secondary staff at hospital #2 consisted of two Research Coordinators; one was a staff member of the hospital’s Risk Management Department and the other was the
208
HOWARD ET AL.
Table 1. KY Consumer Satisfaction Instrument (KY-CSI)* Ordered by Section; the Number in Each Category is Presented, as Well as the Percentage of the Total (N ⴝ 204) Responses Section/Item
Environment The facility was clean. Staff explained my rights as a patient. I felt safe while I was in treatment. I had privacy. When I had a physical illness, I received care promptly. I received the services I needed to help me get better. Affiliation/esteem I understood what was expected of me during treatment. I was able to talk to staff about my problems. I was treated with respect. Staff were available to talk with me about my problems. I was able to easily contact my friends and family. I feel better about myself as a result of treatment. Growth/self-actualization I understood my treatment plan. Staff asked me what I thought would help me get better. Staff explained the effects of my medication. I had time to be with other patients. My friends and family were included in treatment planning when needed. I can deal with my problems better as a result of treatment. Overall Satisfaction I would come back to this facility if I needed treatment again.
Positive
Neutral
Negative
188 (92.2%) 158 (77.5%) 166 (81.4%) 151 (74.0%) 150 (73.5%) 172 (84.3%)
9 (4.4%) 25 (12.3%) 19 (9.3%) 19 (9.3%) 30 (14.7%) 20 (9.8%)
7 (3.4%) 21 (10.3%) 19 (9.3%) 34 (16.7%) 24 (11.8%) 12 (5.9%)
178 (87.3%) 173 (84.8%) 167 (81.9%) 170 (83.3%) 162 (79.4%) 172 (84.3%)
16 (7.8%) 12 (5.9%) 19 (9.3%) 17 (8.3%) 20 (9.8%) 19 (9.3%)
10 (4.9%) 19 (9.3%) 18 (8.8%) 17 (8.3%) 22 (10.8%) 13 (6.4%)
167 (82.3%) 151 (74.4%) 158 (77.8%) 191 (93.6%) 142 (70.0%) 171 (84.2%)
21 (10.3%) 24 (11.8%) 17 (8.4%) 8 (3.9%) 27 (13.3%) 23 (11.3%)
15 (7.4%) 28 (13.8%) 28 (13.8%) 5 (2.5%) 34 (16.8%) 9 (4.4%)
156 (76.9%)
14 (6.9%)
33 (16.3%)
*Item response options: 5, strongly agree; 4, agree; 3, I am neutral; 2, disagree; 1, strongly disagree; 0, not applicable.
Hospital’s Patient Advocate. The Research Coordinators supervised on-site study activities, assisted the RAs, participated in research team meetings, and were responsible for communicating with the Principal Investigator and the technical staff of the research team. METHODS
Study Design A simple survey design with nonrandom sampling technique was utilized at the two study sites. Hospital #1 serves a metropolitan area and a few rural counties; hospital #2 serves primarily a rural area. The study was conducted over a 5-month period at hospital #1 (January to May 1999), and a 3-month period at hospital #2 (March to May 1999). The sponsoring university’s Medical Institutional Review Board approved all study protocols and procedures for informed consent at both sites. Sample The sample consisted of eligible consumers hospitalized during the study period who agreed to
participate in the study. Inclusion criteria were: (1) 18 years of age and older; (2) currently hospitalized; and (3) identification by the treatment team as symptomatically stable and near discharge from the hospital. Exclusion criteria were: (1) identification by the treatment team as incapable of giving informed consent, (2) physical condition that precluded participation, and (3) forensic classification. At hospital #1, 152 eligible consumers were recruited, and of those, 107 agreed to participate, resulting in a response rate of 74%. At hospital #2, 167 were recruited, and 108 participants responded to the survey, resulting in a response rate of 65%. A validation item was included in the questionnaire to determine response set biases. Based on this validation item, a total of 204 surveys were determined to be acceptable for analysis. This included 103 surveys from hospital #1, and 101 surveys from hospital # 2. The analyses described here were conducted on the combined samples from the two state hospital sites. Combining the samples from the two sites is warranted for several reasons: (1) identical inclusion criteria and data collection procedures were
CONSUMER PERSPECTIVES ON QUALITY
209
Table 2. Mental Health Statistics Improvement Program (MHSIP) 21-Item Consumer Survey.* Ordered by Section; the Number in each Category is Presented as Well as the Percentage of the Total (N ⴝ 204) Responses Section/Item
Satisfaction I liked the services that I received. If I had other options, I would still choose to get services from this health plan. I would recommend this service to a friend or family member. Access The location of services was convenient. Staff caregivers were willing to help as often as I felt it was necessary.† My calls were returned within 24 hours. Services were available at times that were good for me. I was able to get all the services I thought I needed. Appropriateness Staff caregivers believed that I could grow, change, and recover.† I felt free to complain. I was told what side effects to watch for. My wishes about who is and is not to be given information about my treatment were respected. Staff caregivers were sensitive to my cultural/ethnic background.† Staff caregivers helped me obtain the information needed so I could take charge of managing my illness.† Outcomes I deal more effectively with daily problems. I am better able to control my life. I am better able to deal with crisis. I am getting along better with my family. I do better in social situations. I do better in school and/or work. My symptoms are not bothering me as much.
Positive
Neutral
Negative
166 (81.4%) 142 (69.6%) 157 (77.3%)
18 (8.8%) 22 (10.8%) 21 (10.3%)
20 (9.8%) 40 (19.6%) 25 (12.3%)
136 (67.3%) 162 (79.8%) 143 (71.9%) 161 (79.3%) 154 (76.2%)
31 (15.4%) 19 (9.4%) 38 (19.1%) 20 (9.9%) 21 (10.4%)
35 (17.3%) 22 (10.8%) 18 (9.1%) 22 (10.8%) 27 (13.4%)
174 (86.1%) 148 (73.3%) 149 (74.1%)
16 (7.9%) 20 (9.9%) 17 (8.5%)
12 (5.9%) 34 (16.8%) 35 (17.4%)
161 (80.1%) 161 (79.7%)
23 (11.4%) 25 (12.4%)
17 (8.5%) 16 (7.9%)
172 (85.6%)
14 (7.0%)
15 (7.5%)
163 (81.5%) 165 (82.1%) 153 (76.5%) 148 (73.3%) 149 (74.9%) 139 (69.5%) 161 (80.1%)
19 (9.5%) 21 (10.5%) 35 (17.5%) 33 (16.3%) 29 (14.6%) 41 (20.5%) 19 (9.5%)
18 (9.0%) 15 (7.5%) 12 (6.0%) 21 (10.4%) 21 (10.6%) 20 (10.0%) 21 (10.5%)
*Response options for all items: 5, strongly agree; 4, agree; 3, I am neutral; 2, disagree; 1, strongly disagree; 0, not applicable. †The word “Staff” was substituted for the word “My.”
used at the two sites, (2) relatively modest differences were found between the two samples, with no significant differences in any of the outcome variables; and (3) the combined sample provided a broader cross-section of the inpatient mental health consumers in this state. Sample Characteristics As described in Table 3, the majority of the 204 respondents in the combined sample were male (60.8%); 84.8% were European-American and the remainder was African-American. The average age of respondents was 37.4 years (SD⫽11.1). More than half of the respondents had at least a high school education (57.7%), and nearly one-fourth of respondents had some education beyond high school. The majority of respondents were unemployed (86.8%); of those who were employed, 10.8% had full- time employment, and 2.5% had part-time employment.
There were a variety of diagnoses on the DSM-IV Axis I in the combined sample (see Table 3); the most prevalent were mood disorder, schizophrenia, and substance abuse. Less frequent diagnoses included delusion/other psychoses, Alzheimer’s/organic brain disorders, impulse control disorders, anxiety disorders, attention deficit disorder, and other miscellaneous disorders. The majority of respondents (63.7%) were admitted to the hospitals on an involuntary basis. The average length of stay (LOS) was 31.0 days (SD⫽ 76.6) but an extremely long stay of nearly 3 years skewed the distribution. The median LOS for all respondents in the combined sample was 19.5 days. Slightly more than one-fourth of the combined sample reported that the current hospitalization was either their first or second; the remaining 71.1% reported at least two previous hospitalizations. Like LOS, the number of previous hospitalizations was influenced by a few observations on
210
HOWARD ET AL.
Table 3. Demographic Characteristics of the Sample (N ⴝ 204) Variable
Gender Male Female Ethnicity European-American African-American Education Less than HS diploma High School diploma At least 1 year post HS Employment status Employed full-time Employed part-time Unemployed Primary diagnosis (DSM-IV) Mood disorders Schizophrenia Substance abuse Delusion/other psychoses Alz/organic brain Impulse control disorder Anxiety Attention deficit disorder Other disorder Admission status Voluntary Involuntary Previous hospitalizations 0 1 2 ⬎2
Frequency (n)
Percent (%)
124 80
60.8 39.2
173 31
84.8 15.2
86 73 44
42.4 36.0 21.7
22 5 177
10.8 2.5 86.8
87 51 24 10 8 7 5 3 9
42.7 25.0 11.8 4.9 3.9 3.4 2.5 1.5 4.4
74 130
36.3 63.7
25 34 35 110
12.3 16.7 17.2 53.9
The sample was similar to the overall population at the two sites on gender, race, admission status, and diagnosis of substance abuse. However, compared with the hospital population, the sample was over represented with respondents with mood disorders and under represented with respondents with schizophrenia and psychotic disorders. Instruments Self-Administered Questionnaire The self-administered questionnaire was organized around the 19-item KY-CSI, the 21-item MHSIP Consumer Survey, the 8-item Consumer Satisfaction Questionnaire (CSQ-8), and a single Quality of Mental Health Care (QMHC) item. The Kentucky Consumer Satisfaction Instrument (KY-CSI)
the high end. The median number of hospitalizations prior to the current one was 2, with a range of 0 to 60.
The KY-CSI, which is based on Maslow’s motivation and human needs theory, contains 19 scaled-response items (Table 1). Response choices for the 19 items range from 5 (“strongly agree”) to 1 (“strongly disagree”); a “not applicable” response choice is also included. As indicated in Table 1, the tool contains six items related to the physical environment, six related to affiliation, and six related to goal attainment and self-actualization. One item questions whether the respondent would return to the facility if services were needed in the future. Potential scores for the KY-CSI range from 19 to 95 (Table 4), with higher scores indicating higher levels of satisfaction. The potential ranges for the sub-scales are summarized in Table 4.
Table 4. Descriptive Analysis and Internal Consistency of the Satisfaction Scales, Subscales, and Single Item (N ⴝ 204) Scale
Number of Items
Mean (SD)
Actual Range
Potential Range
Cronbach’s alpha
KY-CSI Environment Affiliation Growth Would return CSQ-8 MHSIP Satisfaction Accessibility Appropriateness Outcome QMHC
19 6 6 6 1 8 21 3 5 6 7 1
77.3 (13.2) 24.4 (4.5) 24.6 (4.5) 24.2 (4.3) 4.0 (1.3) 25.7 (5.7) 83.2 (15.4) 11.7 (3.0) 19.5 (4.2) 24.0 (4.8) 27.9 (5.4) 7.7 (2.4)
28-95 7-30 10-30 10-30 1-5 10-32 35-105 3-15 5-25 8-30 13-35 0-10
19-95 6-30 6-30 6-30 1-5 8-32 21-105 3-15 5-25 6-30 7-35 0-10
0.95 0.86 0.87 0.84 0.94 0.96 0.88 0.87 0.89 0.91 –
CONSUMER PERSPECTIVES ON QUALITY
The Mental Health Statistics Improvement Program (MHSIP) 21-Item Consumer Survey Since its inception, the original 40-item Consumer Survey developed by the MHSIP Task Force has undergone continuous psychometric testing, evaluation and revision. Psychometric evaluation of the 40-item tool conducted by members of the CMHS-funded Five-State Feasibility Study and the Behavioral Health Measurement Advisory Panel (MAP) of the National Committee on Quality Assurance (NCQA) resulted in the 21-item version that was used in this study (Ganju, 1999). As illustrated in Table 2, the instrument measures consumers’ perceptions of access, appropriateness, outcomes of service, and general satisfaction (Teague et al., 1997). Response choices range from 5 (“strongly agree”) to 1 (“strongly disagree”), with an additional response choice of “not applicable.” Potential scores for the MHSIP 21-Item Consumer Survey range from 21 to 105 (Table 4), with higher scores indicating a higher rating of the quality of services. The range of possible subscale scores is summarized in Table 4. The MHSIP was used in this study to initiate hospital-based psychometric testing and determine the feasibility of using the tool to evaluate outcomes of inpatient care. The Client Satisfaction Questionnaire (CSQ-8) The CSQ-8 is a unidimensional survey that measures global satisfaction with mental health services (Attkinsson & Greenfield, 1996). It contains eight scaled-response items that measure the quality of services received, the degree to which the respondent received services that were needed and wanted, satisfaction with the amount of services received, the degree to which treatment was effective, and whether the respondent would recommend services to a friend or return for services if needed in the future. Response choices range from 4 (“highest satisfaction with services”) to 1 (“lowest satisfaction with services”); the total score has a potential range of 8 to 32, with higher scores indicating a higher level of satisfaction (Table 4). Evidence for high internal consistency of the CSQ-8 is suggested by Cronbach’s alphas of 0.83 to 0.93 (Sederer & Dickey, 1996). The CSQ-8 was included in this study for the purposes of psychometric evaluation of the KY-CSI and MHSIP surveys.
211
Single-Item Quality Rating In addition to the three instruments, a single Quality of Mental Health Care (QMHC) item was added to measure respondents’ overall perception of quality of mental health care. Response choices for the QMHC ranged from 0 (“worst possible care”) to 10 (“best possible care”). Procedures Hospitalized consumers who met study inclusion criteria were approached by RAs to determine their interest in participating in the study. Once verbal consent was obtained, RAs reviewed the consent form with consumers and obtained signatures indicating informed consent. Participants were given a copy of the consent form. After informed consent was obtained, RAs met with the participants in a private area on the unit to give them the survey packet and review the instructions for completing the self-administered questionnaire. RAs read the text of the consent form and questionnaire to participants who were unable to read. It took approximately 15 to 20 minutes for participants to independently complete the self-administered questionnaire. Data Analysis Total scores for each of the three satisfaction instruments were obtained by summing the corresponding items contained in each scale. For each scale, missing data were filled in using mean substitution in cases where the number of items missing for a given respondent was at most 25% of the items in the scale. Mean substitution is the practice of substituting the average value from nonmissing items for missing items. The typical rule of thumb is that this method should only be used to fill in a limited number of missing values and that if the number of missing values is more than 25% of the total items for the scale on a given survey, the summary score should not be calculated for that respondent. In addition, subscale scores were obtained by summing responses on the appropriate items. Because of the lack of normality of the distributions of hospital length of stay and number of previous hospitalizations, these variables were transformed using the square root transformation before being included in the bivariate and multiple regression analyses. This transformation is appropriate for both of these variables since it minimizes
212
the impact of outliers on the high end of the distribution. Univariate Analysis The satisfaction measures and the sociodemographic characteristics were summarized using univariate analysis by calculating means and standard deviations for continuous variables and frequency distributions for categorical variables. Bivariate Analyses Comparison of the two hospital sites on demographic and satisfaction variables was accomplished using two-sample t-tests and chi-square tests of association. In the case of small expected cell counts within the chi-square test of association, Fisher’s exact test was used as an appropriate alternative. Bivariate analyses were also conducted to ascertain the relationships among and between the satisfaction measures and the sociodemographic variables. These analyses consisted of Pearson’s product moment correlations for pairs of continuous variables, chi-square tests of association or Fisher’s exact test for pairs of categorical variables, and one-way analysis of variance (ANOVA) to assess the relationships between continuous and categorical variables. In the case of one-way ANOVA, Fisher’s least significant difference procedure was used to determine whether pair-wise comparisons were significant. Cronbach’s alphas were calculated for the satisfaction scales and subscales to assess internal consistency. Multiple Regression Multiple regression was used to determine whether any of the measures of satisfaction could be predicted using respondent characteristics. A backward stepwise procedure was used to reduce the full set of regressors to only those that were significant predictors. For each satisfaction measure, the initial model contained the potential predictors of age, ethnicity (European-American versus African-American), gender, admission status (voluntary versus involuntary), education (less than high school versus high school and above), employment status (employed versus unemployed), the square root of length of stay, the square root of number of previous hospitalizations, and hospital site (hospital # 1 versus hospital # 2).
HOWARD ET AL.
RESULTS
Instruments Findings indicate good internal consistency of the instruments used in the study (Table 4). Cronbach’s alpha for the overall KY-CSI was .95, and subscale Cronbach’s alphas were .84 and above. For the 21-item MHSIP, the Cronbach’s alpha for the overall scale was .96, and subscale alphas were .88 and above. Cronbach’s alpha for the CSQ-8 was .94. These findings suggest that reliable tools were used in the study. Descriptive Analysis of the Satisfaction Measures All of the satisfaction scales and subscales, the QMHC item, and the item questioning the likelihood of choosing the same facility in the future, showed relatively high mean values when averaged across respondents. For example, the QMHC item, which measured respondents’ perceptions of the overall quality of their care, had an average score of 7.7 out of a possible score of 10. Table 4 displays the mean and standard deviation for each of these measures of satisfaction and overall quality of care. The actual range and potential range of satisfaction scores are also displayed, along with the Cronbach’s alphas for the multi-item scales and subscales. Satisfaction With Services For items on the KY-CSI and MHSIP, 70% or more respondents indicated that they were satisfied with the services they had received during their hospitalization (Tables 1 and 2). Specifically, areas of greatest satisfaction included the time available to be with other patients (94%) and cleanliness of the facilities (92%). In addition, staff-client relationships were a major source of satisfaction among respondents. Staff members were available to speak with patients (83%), and respondents were comfortable talking to staff about their problems and issues (85%). In the majority of cases, staff members provided information that respondents needed to “take charge” of their illnesses (86%), and respondents received the services that they needed to get better (84%). The majority of respondents indicated that they understood what was expected of them during treatment (87%), and they perceived that staff had confidence in their ability to grow, change, and recover (86%). Furthermore,
CONSUMER PERSPECTIVES ON QUALITY
84% of respondents reported that they felt better about themselves as a result of treatment. Despite the overall high ratings of satisfaction, up to 20% of respondents reported some dissatisfaction with the services they received (Tables 1 and 2). For example, on the KY-CSI, 33 (16.3%) respondents indicated that they would not return to the facility if they needed services in the future. Similarly, on the MHSIP, 40 (19.6%) respondents indicated that if they had other options they would not choose this health plan in the future. On the KY-CSI, 28 (13.8%) respondents reported that they did not receive medication education. This is supported by findings of the MHSIP, in which 35 (17.4%) respondents reported that they were not told what side effects to look for. Treatment planning issues were also a source of dissatisfaction for some respondents. Almost 14% reported that staff did not ask them what they thought would help them get better (Table 1) and almost 17% indicated that they did not feel free to complain (Table 2). Respondents also reported dissatisfaction with family involvement in their care; almost 17% indicated that their family and friends were not included in treatment planning when needed (Table 1). An additional area of dissatisfaction (Table 1) included lack of privacy in the inpatient setting (16.7%). Furthermore, access to care was a concern; 35 (17.3%) respondents stated that the location of services was inconvenient (Table 2). Bivariate Relationships Between Satisfaction and Demographic Characteristics Correlation analysis indicated a high degree of association among the various measures of satisfaction. For all scales and subscales, correlation coefficients ranged from r ⫽ 0.5 to r ⫽ 1.0 (p⬍ .0001), and the majority of the coefficients were in the 0.7 to 0.8 range. None of the satisfaction measures were significantly related to age, gender, education, DSM-IV Axis I diagnosis, or the square root of the length of stay. The number of previous hospitalizations was significantly negatively correlated with several measures of satisfaction, including the Affiliation subscale of the KY-CSI (r ⫽ -.18; p ⫽ .01), the CSQ-8 (r ⫽ -.15; p ⫽ .04), the Appropriateness subscale of the MHSIP (r ⫽ -.15;
213
p ⫽ .04), and the single item QMHC (r ⫽ -.18; p ⫽ .01). ANOVA calculations indicated that the Outcome subscale of the MHSIP differed by ethnicity, with African-Americans rating these items higher than European-Americans (the means were 30.1 and 27.7 respectively, with t ⫽ 2.5 and p ⫽ .02). The Satisfaction subscale of the MHSIP also differed according to admission status, with significantly higher scores among voluntary admissions (the means for voluntary and involuntary admissions were 12.4 and 11.2 respectively, with t ⫽ 2.9; p ⫽ .004). Employed respondents rated the Growth/Self-Actualization subscale of the KY-CSI higher than those who were unemployed (the means were 5.9 and 4.7 respectively, with t ⫽ 2.1 and p ⫽ .04). Predictors of Satisfaction The models for the Environment and Affiliation subscales of the KY-CSI contained no significant predictors of consumer satisfaction. Similarly, there were no significant predictors of the CSQ-8, the Access subscale of the MHSIP, and the single item QMHC. The final models for each of the other measures of satisfaction are displayed in Table 5. Although employed respondents rated their satisfaction with services higher than unemployed respondents in the ANOVA calculations, employment was a significant negative predictor of satisfaction in the regression model. Other significant negative predictors of satisfaction included male gender and European-American ethnicity (Table 5). In contrast, a voluntary admission status may bolster satisfaction. The low R2 values (Table 5) for the regression models indicated that although some of these patient characteristics are significant predictors, they account for a very small portion of the variability in satisfaction. For example, as indicated in Table 5, only 2% of the variability in satisfaction, as scored on the MHSIP Appropriateness Subscale is accounted for by European-American ethnicity. In addition, the small values for R2 may be an artifact of the relatively low variability in satisfaction scores as a result of high satisfaction among nearly all participants.
214
HOWARD ET AL. Table 5. Final Models for Each of the Satisfaction Measures With a Significant Regression (N ⴝ 204) Satisfaction Measure
KY-CSI KY-CSI growth KY-CSI would return MHSIP MHSIP satisfaction MHSIP appropriateness MHSIP outcome
Significant Predictors
Standardized Betas
p Value
Model R2
Male Employed Male Employed Voluntary admit European American Voluntary admit Employed European American European American
⫺.15 ⫺.15 ⫺.20 ⫺.18 .19 ⫺.16 .24 ⫺.16 ⫺.15 ⫺.21
.04 .04 .005 .01 .01 .03 .0009 .03 .04 .005
0.04
DISCUSSION
The Consumer Research Associate Program Research Associates, who were recovered mental health consumers, assumed major responsibility for data collection in this study. Several findings support this role in service system evaluation research. In this study, respondents independently completed the self-administered survey in about 20 minutes, whereas in the pilot study, the face-toface interviews lasted about one hour (Howard et al., 2001). The reduced time for survey completion suggests that the RAs were thorough in giving participants instructions and that the self-report method of data collection imposed less burden on respondents compared with the face-to-face interview. Shorter completion time may also reduce costs for providers and institutions in outcomes assessment programs. Moreover, survey research typically yields low response rates (Levy & Lemeshow, 1999) yet the response rate for this study was approximately 70%. Furthermore, whereas 215 surveys were administered, a total of 204 were retained in the analysis. This high rate of correct completion of the surveys is an additional indication that RAs provided respondents with clear instructions about the study purpose and procedures for survey completion. Both “unit nonresponse” and “item nonresponse” are “very difficult to avoid in the sampling of populations” (Levy & Lemeshow, 1999, p. 393). RAs helped offset potential study limitations by boosting both the unit and item response rates and by contributing to the collection of quality data. The nonrandom sampling technique possibly contributed to the high response rate but it was also a study limitation. In our sample, respondents with
0.07 0.04 0.02 0.08 0.02 0.04
mood disorders were over represented and respondents with schizophrenia and psychotic disorders were under represented when compared with the hospital populations. Study findings must be interpreted in light of this limitation. Service Satisfaction and Dissatisfaction In general, whether findings were viewed within the context of one instrument or collectively, the degree of satisfaction with mental health services appeared to be quite high. Viewed collectively, findings suggest the presence of a therapeutic, trusting, reciprocal relationship between staff and respondents. In addition, respondents reported high levels of satisfaction with staff-client relationships, staff availability, and respondents’ ability to talk to staff members about their problems. Respondents were also highly satisfied with several aspects of service delivery. Given the high degree of satisfaction with staff-client relationships and delivery of needed services, it is not surprising to find that the majority of respondents reported feeling better about themselves as a result of treatment. Service Dissatisfaction A limitation of research about consumer satisfaction concerns lack of attention to areas of dissatisfaction. Moreover, it is important to understand reasons for dissatisfaction in the public mental health systems since it serves as the “safety net” for most individuals with mental health problems. We explored items in which respondents indicated that they “strongly disagreed” or “disagreed” to determine areas that might warrant improvement. Some items related to service dissatisfaction were noteworthy. As many as 40 respondents indicated that they would not return to the facility or choose the health
CONSUMER PERSPECTIVES ON QUALITY
plan in the future. Contributing factors may have been related to perceptions about exclusion from treatment planning. For instance, despite the current emphasis on consumer-driven care, findings suggest that respondents had limited input into their treatment planning. Specifically, several respondents indicated that they were not asked what would help them get better. Perceptions of satisfaction are based on the consumers’ expectations and desires, as well as their assessment of their interactions with providers (Lebow, 1982; Ruggeri, 1994; Teague, 2000). More important, providers who are not aware of what consumers expect in treatment may not maintain a focus on client-centered treatment goals, despite their good intentions. Other items of concern were lack of education about medications and treatment side effects (Tables 1 & 2). Lack of education and information are cause for even greater concern when coupled with lack of involvement of family and friends in the process of treatment planning. A primary goal of treatment is to promote recovery and self-care in the least restrictive setting. When consumers and their family caregivers do not understand treatment or do not know about medication side effects, recovery is compromised and the consumer is at risk for relapse and rehospitalization. Finally, the findings about areas of dissatisfaction raise questions about a continued emphasis on the provider’s perspectives in the process of making decisions about the clients’ plan of care. Domains of Satisfaction Results of the multiple regression analysis provide some evidence to suggest that satisfaction with mental health services may be lower among males, European-Americans, and those who are employed. Prior satisfaction research has shown a lack of consistency in elucidating the relationships between sociodemographic characteristics and satisfaction with mental health services. However, Rosenheck, Wilson, and Merketo (1997) conducted a descriptive study to investigate patient satisfaction in a sample of veterans with a primary psychiatric diagnosis who had been recently discharged from an inpatient setting. Their findings indicated that African-Americans reported less satisfaction than European-Americans on a measure of courtesy, but they reported more satisfaction compared with European-Americans on measures
215
of emotional support, physical comfort, involvement, and practical orientation. Culture and ethnicity will dominate concerns in the delivery of mental health services in the U.S. in the future because of changing demographics. Therefore, research about the relationships among and between these variables is crucial. CONCLUSIONS
This study shows how the person-centered focus of the consumer movement was incorporated in a collaborative project between University researchers and a state’s Department for Mental Health and Mental Retardation Services. Findings were promising in several ways. First, the consumers represented by the respondents of this study indicated that public-sector psychiatric hospitals provide quality services. Second, the consumer RAs were valuable members of the research teams at both hospital sites. Third, preliminary findings suggest that both the KY-CSI and 21-item MHSIP are reliable instruments and are feasible for use in inpatient settings. Additional psychometric studies to ascertain the reliability and validity of the 21item version and any other versions of the MHSIP Consumer Survey are recommended. Similar studies about the KY-CSI are warranted not only in inpatient but also in community settings. To offset limitations related to representative samples, we recommend stratified random sampling techniques, when feasible, in evaluation research on consumer perspectives of services. The person-centered focus in mental health services can be facilitated by conducting evaluation studies that explore the consumers’ perspective and by using instruments that were developed with consumer input. Furthermore, inclusion of consumers in the research process is crucial for understanding ways to improve public and private systems of care. ACKNOWLEDGMENT The authors thank those who kindly volunteered to participate in the study and the Research Associates who collected the data. They also thank Commissioner Margaret Pennington of the KDMHMRS; the administrators and staff of the hospitals; the Research Coordinators, Clara Furlong and Connie Smedley; the Research Assistants, Deborah S. Armstrong and Barbara Slater; W. Russell Coyle for technical assistance; Ronald W. Manderscheid, of the Survey and Analysis Branch, Center for Mental Health Services, Rif S. El-Mallakh, M.D. for reviewing the manuscript; and Jewel D. Begley for manuscript preparation.
216
HOWARD ET AL.
REFERENCES Attkinsson, C.C., & Greenfield, T.K. (1996). The Client Satisfaction Questionnaire (CSQ) Scales and the Service Satisfaction Scale-30 (SSS-30). In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp.120-127). Baltimore, MD: Williams and Wilkins. Auslander, M., Moses, D.J., Granger, D., Henderson, M.J., Johnson, J., & Johnson, J.R., et al. (1998). Consumer issues in managed behavioral health care. Journal of the Washington Academy of Sciences, 85(1), 15-27. Buckley, S.M. (1993). Moving MHSIP towards a person-centered paradigm: A concept paper submitted to CMHS and the MHSIP ad hoc advisory group. Available at: http://www.mhsip.org/documents/moving.htm. Accessed May 16, 2001. Campbell, J. (1998). Consumerism, outcomes, and satisfaction: A review of the literature. In R.W. Manderscheid & M.J. Henderson (Eds.), Mental Health United States, 1998 (pp. 11-28). Center for Mental Health Services, DHHS Pub. No. (SMA) 99-3285. Washington, DC: Supt. Of Docs., U. S. Govt. Print. Off. Center for Mental Health Services (1996). The MHSIP Consumer-Oriented Mental Health Report Card: Final report of the Mental Health Statistics Improvement Program (MHSIP) Task Force on a consumer-oriented mental health report card. Rockville, MD: U.S. Department of Health and Human Services, Public Health Service, Substance Abuse and Mental Health Services Administration. Chisholm, M., Howard, P.B., Boyd, M.A., Clement, J.A., Hendrix, M.J., & Reiss-Brennan, B. (1997). Quality indicators for primary mental health within managed care: A public health focus. Archives of Psychiatric Nursing, 11(4), 167-181. deRuyter, K, Bloemer, J, & Peeters, P. (1997). Merging service quality and service satisfaction: An empirical test of an integrative model. Journal of Economic Psychology, 18, 387-406. Docherty, J.P., & Streeter, M.J. (1996). Measuring outcomes. In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp. 8-18). Baltimore, MD: Williams & Wilkins. Eisen, S.V., Shaul, J.A., Leff, H.S., Stringfellow, V., Clarridge, B.R., & Cleary, P.D. (2001). Toward a national consumer survey: Evaluation of the CABHS and MHSIP instruments. Journal of Behavioral Health Services and Research, 28(3), 347-369. Ganju, V. (1999). The MHSIP consumer survey: History, development, revisions, applications, commonly-asked questions. Available at: http://www.mhsip.org/documents/ MHSIPConsumerSurvey.pdf. Accessed May 3, 2001. Howard, P.B., Clark, J.J., Rayens, M.K., & Coyle, R. (2000). Satisfaction as an outcome indicator of quality in mental health services at Central State Hospital. (KDMHMRS Technical Report Series). Lexington, KY: Author. Howard, P.B., Clark, J.J., Rayens, M.K., Hines-Martin, V., Weaver, P., & Littrell, R. (2001). Consumer satisfaction
with services in a regional psychiatric hospital: A collaborative research project in Kentucky. Archives of Psychiatric Nursing, 15(1), 10-23. Howard, P.B., & El-Mallakh, P. (2001). Training consumers to collect data in mental health service system evaluation research. Journal of Psychosocial Nursing, 39(5), 3039. Jennings, B.M., Staggers, N., & Brosch, L.R. (1999). A classification scheme for outcome indicators. Image: Journal of Nursing Scholarship, 31(4), 381-388. Kolb, S.J., Race, K.E., & Seibert, J.H. (2000). Psychometric evaluation of inpatient psychiatric care consumer satisfaction survey. The Journal of Behavioral Health Services and Research, 27(1), 75-86. Lebow, J. (1982). Consumer satisfaction with mental health treatment. Psychological Bulletin, 91(2), 244-259. Levy, P.S., & Lemeshow, S. (1999). Sampling of populations: Methods and applications (3rd ed). New York, NY: John Wiley & Sons. Mahon, P.Y. (1996). An analysis of the concept ‘patient satisfaction’ as it relates to contemporary nursing care. Journal of Advanced Nursing, 24, 1241-1248. Manderscheid, R.W., Henderson, M.J., & Brown, D.Y. (2001). Status of national accountability efforts at the millennium. In R.W. Manderscheid & M.J. Henderson (Eds.), Mental Health United States, 2000 (pp.45-54). DHHS Supt. of Docs. Publication No. (SMA) 01-3537. Washington, DC: U. S. Government Printing Office. Pascoe, G.C. (1983). Patient satisfaction in primary health care: A literature review and analysis. Evaluation and Program Planning, 6, 185-210. Rosenheck, R., Wilson, N.J., & Meterko, M. (1997). Influence of patient and hospital factors on consumer satisfaction with inpatient mental health treatment. Psychiatric Services, 48(12), 1553-1561. Rosswurm, M.A., & Larrabee, J.H. (1999). A model for change to evidence-based practice. Image: Journal of Nursing Scholarship, 31(4), 317-322. Ruggeri, M. (1994). Patients’ and relatives’ satisfaction with psychiatric services: The state of the art of its measurement. Social Psychiatry and Psychiatric Epidemiology, 29, 212-227. Sederer, L.I., Dickey, B., & Hermann, R.C. (1996). The imperative of outcomes assessment in psychiatry. In L.I. Sederer & B. Dickey (Eds.), Outcomes Assessment in Clinical Practice (pp. 1-7). Baltimore, MD: Williams & Wilkins. Smith, M.E., & Hopkins, C. (2000). Report on the MHSIP Survey Workgroup Meeting. Paper presented at the 49th Annual Conference of the Center for Mental Health Services and Mental Health Statistics Improvement Program, Washington, DC. Teague, G.B. (2000). Patient perceptions of care measures. In Handbook of Psychiatric Measures (pp. 169-194). Washington, DC: American Psychiatric Association. Teague, G.B., Ganju, V., Hornik, J.A., Johnson, J.R., & McKinney, J. (1997). The MHSIP Mental Health Report Card: A consumer-oriented approach to monitoring the quality of mental health plans. Evaluation Review, 21(3), 330-341.
CONSUMER PERSPECTIVES ON QUALITY
U. S. Department of Health and Human Services. (1999). Mental Health: A Report of the Surgeon General-Executive Summary. Rockville, MD: U. S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental
217
Health Services, National Institutes of Health, National Institute of Mental Health. Ware, J.E., & Snyder, M.K. (1975). Dimensions of patient attitudes regarding doctors and medical care services. Medical Care, 13(8), 669-682.