The development and construct validation of a consumer satisfaction questionnaire for psychiatric inpatients

0149-7189189 $3.00 + .oo Copyright 0 1989 Pergamon Pressplc

Evaluation and Program Planning, Vol. 12, pp. 189-194, 1989 Printed in the USA. All rights reserved.

THE DEVELOPMENT AND CONSTRUCT VALIDATION OF A CONSUMER SATISFACTION QUESTIONNAIRE FOR PSYCHIATRIC INPATIENTS

WILLIAMR. HOLCOMB Mid-Missouri Mental Health Center

NICHOLAS

A. ADAMS

University of Missouri-Columbja

HOWARD M. PONDER Columbia, Missouri

ROBERT REITZ Osawatomie State Hospital

ABSTRACT A consumer satisfaction scale was developed for psychiatric inpatients. The scale was administered to a state-wide sampie of (n = 366) patients discharged from acute psychiatric units throughout Missouri. The scale was factor analyzed and given to another sample of (n = 390) discharges. Three out of five factors that were found in the first sample were replicated in a factor analysis of data from the second sample. Acceptable coefficient alphas were obtained on all three factors, demonstrating internal reliability. The three factors were labeled Treatment Effectiveness, Trust of Staff, and Hospital Environment. Comparable levels of satisfaction with prior studies were found on the first general satisfaction factor of Treatment Effectiveness. Subjects discriminated areas of satisfaction by expressing higher levels of dissatisfaction on the factors of Trust of Staff and Hospital Environment.

In recent years there has been an increasing emphasis on consumer satisfaction as a measure of treatment outcome for psychiatric patients. Several measurement scales have been developed for use in both inpatient and outpatient settings (Lebow, 1983; Weinstein, 1979). In general, the results of research on patient satisfaction have not been comparable across studies with evident problems in sampling procedures and the use of varied and unstandardized measurement instruments (Kalman, 1983; Lebow, 1982; Pascoe, 1983). Many of

the measures used are not published with supporting reliability and validity data. Definitional problems also appear in the content of some of these scales in determining what is to be included in “Satisfaction.” Most reviewers have argued that the concept of consumer satisfaction is complex and should be measured with multidimensional scales (Lebow, 1984; Pascoe, 1983). On the other hand, some have argued that patients are unable to discriminate subareas of satisfaction and thus respond in a unidi-

Requests for reprints should be sent to William K. Holcomb, F&ton State Hospital, 600 East 5th Street, F&on. 189

MO 65251.

190

W.R. HOLCOMB,

N.A. ADAMS,

mensional fashion. Larsen, Attkisson, Hargreaves, and Nguyen (1979) reached this conclusion after finding that one general factor in a factor analytic study of outpatients accounted for most of the variance in their sample. However, the principle axis method of factor analysis used by these authors was designed so that the first factor would account for most of the variance (Cattell, 1978). This does not mean that the construct under consideration is unidimensional and that all factors but the first should be discarded. Other factor analytic research with both inpatient and outpatient samples have found that consumer satisfaction is multidimensional (Love, Caid, & Davis, 1979; Tanner, 1982; Weinstein, 1979). There are a number of reasons why an assessment of consumer satisfaction of psychiatric inpatients in public facilities is important. One purpose of mental health treatment is to restore a level of functioning to the patient and the patient has a unique perspective of how well that is accomplished. An assessment of client satisfaction in outcome evaluation seems especially important in light of the fact that prior research has indicated that clients and therapists often disagree on the amount of benefit derived from treatment (El-Guebaly, Toews, Leckie, & Harper, 1983; Larsen et al., 1979; Lebow, 1982; Mayer & Rosenblatt, 1974). A second compelling reason for assessing patient satisfaction is that accreditation standards and/or legislative mandates frequently exist for including such data in program evaluation and quality assurance (Weinstein, 1979). Public psychiatric facilities serve the poor and often socially rejected individuals who do not have the resources to seek alternative treatment in private facilities. These patients are unable to go elsewhere if they are dissatisfied with services. If a patient does decide to disengage from treatment, there may be no financial penalty incurred by the treatment facility, but rather a reduction in perhaps an overburdened workload. Under these conditions in publicly funded institutions, the determination of need for services and

H.M. PONDER,

and R. REITZ

quality of services is left totally in the hands of service providers with little incentive to seek client evaluation of services. Weinstein (1979) has written a review of the literature on consumer satisfaction with services in mental hospitals. Even though the literature is fragmented with little consistency in measurement instruments used, the author concludes that the level of patient satisfaction with inpatient psychiatric services has not substantially improved from the early 1950s to the present. High rates of satisfaction that are consistently found with quantitative measures are contrasted with negative impressions of patient satisfaction from qualitative evaluations. Weinstein (1979) argues that these qualitative evaluations have ignored important considerations of proper sampling, reliability, and validity issues and may have promoted popular misconceptions of the patients’ plight in mental hospitals. A well constructed quantitative instrument can be inexpensively and effectively administered but yet provide feedback that can influence clinical and administrative decisions (Lehman & Zastowny, 1984). A multidimensional measurement of patient satisfaction can help tailor treatment techniques to the needs and expectancies of the patients. An overall level of satisfaction can serve as outcome criteria to compare different treatment facilities, wards, and programs. A comparison of the same program under different conditions could also be made. In spite of a growing body of literature about consumer satisfaction in psychiatric settings, an established measurement scale for inpatients with adequate reliability, validity, and published norms has not been developed. The present report describes the development of a multi-dimensional consumer satisfaction scale for use in psychiatric inpatient settings. Reliability and construct validity data are presented with statewide norms using both state hospitals and state operated community mental health centers with inpatient acute treatment units.

METHOD All patients discharged from the acute treatment units of five state hospitals and three state operated community mental health centers within a two month period were asked to complete a patient satisfaction survey form. All state operated psychiatric inpatient acute treatment units in the state of Missouri are included. Sixty percent of actual discharged patients completed the forms in a valid fashion. About 20% of patients discharged could not complete the forms in a valid fashion because of severe psychopathology or refused to do so. Another 20% left the hospital against medical advice or at times when staff who knew about survey procedures were not available.

The average age of patients was 35.2 with 53% being white and 47% being minorities which were primarily black. Twenty-one percent were committed involuntarily by the courts. The sample had an average of 5.3 prior hospitalizations to state facilities with 38% having no prior hospitalizations for psychiatric problems. The satisfaction form was presented to the patient at time of discharge with instructions indicating that their responses would not influence their treatment or discharge. The purpose of the survey was explained as helping the staff to evaluate their treatment program. Several steps were involved in the development of the satisfaction scale. A number of brainstorming sessions

Consumer

Satisfaction

with groups of patients at one of the five state hospitals was used to compose a list of patient “likes” and “dislikes” with services received. Twenty direct care mental health professionals were asked to edit the original list of patient concerns so that a final list of 90 items resulted. The list was then formated with 5-point Likert scales and administered to 75 additional patients as they were being discharged. An item analysis was done and all items which had an item-total correlation of less than .30 were excluded from the scale. Nunnally (1978) argues for a .20 cutoff for item-total correlations but the authors used the more stringent .30 in order to reduce the total number of items. Nunnally (1978) also presents the rationale for applying item analysis to establish a homogeneous set of items before factor analysis is used. Further, elimination of items which overlapped in content resulted in a final scale of 41 items. A number of negatively worded items were included to help control for response bias. The patient satisfaction forms (n = 366) were then factor analyzed using the principle components method (Ray, Sall, Saffer, Joyner, 8~ Whatley, 1982). This method was chosen according to Nunnally (1978) who

191

Questionnaire

argues that, with 20 or more variables in an exploratory factor analysis, component analysis should be used with unities in the diagonals. A Scree test was applied (Gorsuch, 1983) to determine that five factors should be retained in a varimax rotation procedure. Scores on the items which had factor loadings above .40 on a particular factor became part of that factor. Items were marked on a Likert scale of 1 to 5 with 1 = “Disagree Strongly” to 5 = “Agree Strongly.” To verify the construct validity of this factor structure, the same 41 item questionnaire was again given to all patients admitted and discharged from all acute inpatient psychiatric units in the state public mental health system during another period of two months (n = 390). Principle component factor analysis was again used with varimax rotation and retaining five factors. These five factors were then compared with the five factors obtained on the first state-wide survey using the Coefficient of Congruence and the Cosine method of factor comparison (Cattell, 1978). Cattell (1978) has argued that two quantitative methods should be used to confirm results when comparing factor structures.

RESULTS Table 1 presents the results of two quantitative methods of factor comparisons. The Cosine method computes coefficients that according to Cattell (1978) are roughly equivalent to Pearson-product-moment correlations. Only three of the five factors derived from the first analysis were replicated in the second factor analysis. The Coefficient of Congruence method of factor comparisons is also presented in Table 1. Results show clear replication of the first two factors and modest confirmation of the third factor. When examining items with high factor loadings, there does appear to be considerable overlap between Factors IV and V in the first sample and Factor I in the second sample. There also appears to be considerable overlap between Factor III in the first sample and Factor IV of the second sample. To further explore the reliability of the factor structure, both samples were combined (n = 756) and factor analyzed with the principle components method. Varimax rotation was again used, limiting the factor number to five. As expected, most of the items on Factors IV and V merged into the first three factors. Only two items remained with their highest factor loadings (greater than .40) on Factor IV and only one item remained with a primary loading on Factor V. These three items were eliminated due to their idiosyncratic meaning and varimax rotation was again used limiting the number of factors to three. Table 2 presents the means, standard deviations, and factor loadings of the items which had primary factor loadings of .40 or greater on only one factor. Five additional items with complex meanings or high factor loadings on more

than one factor were eliminated leaving a total of 33 items. The internal reliability of the remaining three factors was tested using Coefficient Alpha. Alpha levels on the factors were .92, .80, and .70 respectively. These internal reliability levels exceed Nunnally’s criteria of accept-

TABLE 1 TWO METHODS (COSINE AND COEFFICIENT OF CONGRUENCE) OF FACTOR COMPARISONS BETWEEN CONSUMER SATISFACTION SCORES IN TWO SAMPLES OF PSYCHIATRIC INPATIENTS Consumer Satisfaction

Cosine Method 1. Treatment effectiveness 2. Trust of staff 3. Hospital environment 4. Staff competence 5. Patient involvement

Scores

1

2

3

4

5

1.18’ .25 -.62 -.52 .68

.04 .88’ .04 -.28 .44

.37 .09 .79* -.09 -.49

-.31 .05 .48 .58 .37

.01 .49 .37 .47 .58

.31 .92* .55 .52 .23

.16 .66 .65’ .03 .21

.34 .30 .81 .34 .I9

.48 .48 .27 .I5 .I8

Coefficient of Congruence Method 1. Treatment effectiveness .94* 2. Trust of staff .27 3. Hospital environment .34 4. Staff competence .83 5. Patient involvement .71 ‘Indicates high levels of congruence the two samples.

between factors obtained in

192

W.R. HOLCOMB,

N.A. ADAMS, H.M. PONDER, and R. REITZ TABLE 2

ITEM MEANS, STANDARD DEVIATIONS, % SATISFIED, % DISSATISFIED, AND FACTOR LOADINGS FOR INPATIENT CONSUMER SATISFACTION SCALE (ICSS)

Factor Loadings

Items

1. 2. 3. 4. 5. 6. 7. 8. 9. IO. 11 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.

Counseling was helpful therapy for me I am satisfied with services I received I feel better as a result of coming to the hospital The nursing staff was willing to listen and talk to me in a helpful way This hospital stay helped me learn how to cope with my problems I am more hopeful about my future after my stay in the hospital It was clearly explained to me why I had to come to the hospital Activities (music, crafts, and exercrse) were a helpful part of my treatment I felt like I could be myself in the hospital The reasons for taking medicatrons were explained to me Talking with my psychologist was helpful to me Talking with my doctor was helpful to me My medication helped me get better I knew what services were available to me I would encourage others needing help to come to the hospital The staff appeared competent and well trained The professional staff spent enough time with me The staff helped me with my physical needs when I needed it I have a higher opinion of mental hospitals as a result of my stay here The staff was consistent in explaining my treatment I helped develop my treatment plan

% Dissatisfied

Trust of Staff

Y

SD

% Satisfied

3.8 3.8 3.9

.9 1.0 1.1

72.2 75.5 75.6

10.3 11.9 12.1

.61 .64 .67

.I 1 .27 .13

-.005 .09 .07

4.0

.9

82.5

8.1

.61

.27

-.09

3.8

1.0

69.7

12.3

.73

.I2

05

3.9

.9

75.5

9.6

.66

.I 1

.002

3.6

1.1

66.6

17.1

.54

.I0

.05

3.7 3.7 3.6 3.7 3.9 3.7 3.6

1.0 1.1 1.0 1.0 .9 1.0 .9

64.5 70.4 67.1 68.1 78.1 63.3 67.8

13.3 17.2 15.1 12.2 9.1 11 .o 16.9

.60 .58 .58 .67 71 .61 53

-.08 .02 .02 IO 14 -.04 .05

3.8 3.9 3.7

1.0 .9 1.0

72.1 78.8 68.4

10.2 8.0 15.5

.64 .66 .61

.I 2 .30 .15

.I0 -.Ol .12

3.9

.9

75.2

8.8

.62

.21

-.02

3.6 3.6 3.4

1.1 1.0 1.1

59.9 62.9 53.2

14.1 14.9 20.8

65 .57 .52

.12 .15 -.I 1

.I0 .Ol -.02

Treatment Effectiveness

Hospital Environment

.I 3 .32 .03 .08 -.03 13 .12

22. Sometimes I felt “put down” by the staff 23. The professional staff (doctors, nurses, psychiatrists, and social workers) generally did not seem concerned about my improvement 24. Ward rules were overly restrictive 25. At times I was not treated with respect 26. I was physically mistreated by staff 27. Professional staff were always too busy to talk with me 28. Generally I didn’t feel I could trust the staff

2.3

1.2

64 2

21.4

.09

.55

.24

2.3 2.6 2.6 2.0 2.3 2.3

1.2 1.1 1.2 .9 1.0 1.1

64.8 54.4 56.4 78.5 66.3 65.1

19.8 23.8 25.3 8.3 13.8 16.8

.13 .09 .09 .14 .13 .16

.59 .49 .63 .70 .67 .68

.08 .37 .34 .03 28 .30

29. While in the hospital I felt like I had no control over my life 30. Other patients on the ward frightened me 31 There was not enough privacy on the ward 32. The hospital was a fnghtening experience 33. The ward was noisy

2.7 2.6 3.1 1.6 3.2

1.3 1.2 1.3 1.2 1.2

53.5 56.0 42.1 56.2 35.9

28.7 27.4 41.4 28.1 44.4

.I3 -.Ol -.02 .I0 04

.34 .22 .28 .24 .17

.46 .66 .55 .65 El

able internal consistency and indicate homogeneity of meaning for each factor (Nunnally, 1978). The intercorrelations between factors were calculated to determine shared variance. The first factor correlates with Factor II, .37, and with Factor III, .22, and therefore

seems to be independent. Factor II correlates .60 with Factor III and thus shows some shared variance (36%). Eigenvalues of the factors were 10.7, 3.9, and 1.6 respectively with total variance explained after rotation being 9.4070, 3.7%, and 3.1%.

Consumer Satisfaction Questionnaire The item content of each factor was examined in order to name the construct being measured. The first factor seems to be measuring an overall satisfaction with services or treatment received. The item with the highest loading was “This hospital stay helped me learn how to cope with my problems.” Several other items addressed how talking with the doctor, psychologist, and receiving medication “helped” the patient. The second factor seems to be concerned with the perceived respect shown by the staff to the patient. An underlying concept being measured may be the patient’s trust of the staff providing services as seems to be reflected in the item: “Generally, I didn’t feel I could trust the staff.” The final factor seems to be measuring the consumer’s satisfaction with the hospital environment, The amount of noise on the ward, how secure the patient feels, the amount of privacy in the hospital, and perceived control are concerns that seem important for this environmental factor. It should be noted that all items on Factor I are positively worded whereas all items on Factors II and III are negatively worded. It is possible that this clustering

193

pattern may represent the operation of response styles. However, the clear content differences between factors, with Factor II being focused on staff issues and Factor III on environmental issues, argues for the importance and meaningfulness of the constructs. Table 2 presents the percent of the combined sample that express satisfaction or dissatisfaction on each item. Percentage of satisfaction was determined by adding the frequencies of those who marked 4 or 5 on the Likert scale for positively worded items and 1 or 2 on negatively worded items. Percent of dissatisfaction was determined by adding frequencies of those who marked 1 or 2 on positively worded items and 4 or 5 on negatively worded items. Sixty-nine percent (69%) of all patients expressed an overall satisfaction on the first factor of Treatment Effectiveness with 12.8% expressing dissatisfaction. Sixty-four percent (64.2%) expressed Trust of Staff (Factor II) with 23.8% expressing dissatisfaction. Finally, only 48.7% expressed satisfaction with the hospital environment with 34% expressing dissatisfaction.

DISCUSSION Even though there has been considerable recent research on the development and validation of consumer satisfaction measurements for psychiatric outpatients, there has been little done toward developing a standardized and valid instrument for inpatients. The objective of this project was to develop a patient satisfaction questionnaire that could be used in public psychiatric inpatient units. Items were initially developed through discussion with psychiatric inpatients. Two samples of all patients being released from all state-operated psychiatric acute treatment units in Missouri were used to further develop and validate the factor structure of the instrument. Several conclusions can be made from these results. 1. Three meaningful factors were identified using the Inpatient Consumer Satisfaction Scale (ICSS). These three factors were replicated in a second state-wide sample of all inpatients discharged from acute treatment units. The first factor seems to be a general factor measuring satisfaction with services. The presence of this one general factor is consistent with prior research with outpatient and inpatient samples (Essex, Fox, & Groom, 1981; Love et al., 1979; Slater, Linn, & Harris, 1982; Weinstein, 1979). Another factor concerns satisfaction with the respect and dignity shown to the patient. The final factor concerns primarily satisfaction with the environment. Factors II and III have also been found in prior research with inpatients (Weinstein, 1979). 2. As in prior studies with psychiatric inpatients, the ICSS reveals high levels of satisfaction in general

with services received. In a meta-analysis of the literature with valid measurement instruments, Lehman and Zastowny (1983) found that approximately 78.27 of subjects representing six programs and 545 nonchronic patients in conventional programs expressed satisfaction, with 15.5% expressing dissatisfaction. These estimates are comparable to the 69.9% expressing satisfaction and 12.8% expressing dissatisfaction on Factor I. On the item, “I am satisfied with services I received,” 75.5% expressed satisfaction and 11.9 expressed dissatisfaction. 3. These overall positive satisfaction scores by acute psychiatric inpatients in public care facilities support Weinstein’s (1979) contention that the negative aspects of psychiatric hospitalization may have been exaggerated by prior qualitative evaluations that have not considered important issues of reliability and validity of measurements. 4. Some authors have indicated that positive response bias negates some of the utility of consumer satisfaction evaluations (Lebow, 1982; LeVois, Nguyen, & Attkisson, 1981; Kalman, 1983). The present results, however, argue that patients can and do discriminate areas of satisfaction. On Factor II (Trust of Staff), only 64% expressed satisfaction and on Factor III (Environment) only 48.7% expressed satisfaction with 34% expressing dissatisfaction. More patients expressed dissatisfaction with the levels of noise on the inpatient unit (44.4%) than were satisfied (35.9%). Approximately the same number of patients expressed satisfaction and dissatisfaction with the amount of privacy on the ward. 5. Internal reliability of the three factors replicated in

194

W.R.

HOLCOMB,

N.A.

ADAMS,

this study was acceptable (alphas = .92, .80, & .70). The means and percentage satisfied and dissatisfied for the items are presented by each factor for 756 patients in state inpatient psychiatric facilities in both rural and urban areas. These norms can provide a basis for comparing facilities and programs. 6. The present study has focused on scale construction, internal reliability and construct validity of the factor structure of the ICSS. Prior research has not been consistent in relating patient satisfaction measures to

H.M.

PONDER,

and R. REITZ

other clinical, program, and patient characteristics (Tanner, 1981; Kalman, 1983; Attkisson & Zwick, 1982). Before the ICSS can be fully accepted as a program evaluation tool and as an outcome measure, more research needs to go beyond these preliminary results to establish its concurrent validity. However, the present instrument does hold promise as a reliable and valid measure of consumer satisfaction with inpatient psychiatric care.

REFERENCES ATTKISSON, C. C., & ZWICK, R. (1982). The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program

LEHMAN, A. F., & ZASTOWNY, T. R. (1983). Patient satisfaction with mental health services: A meta-analysis to establish norms. Eval-

uation and Program Planning, 6, 265-274.

Planning, 5, 233-237. in

M., NGUYEN, T. D., & ATTKISSON, C. C. (1981). Artifact in client satisfaction assessment: Experience in community mental health settings. Evaluation und Program Planning, 4, 139-150.

J., LECKIE, A., & HARPER, D. satisfaction: methodological issues.

LOVE, R. E., CAID, C. D., & DAVIS, A. (1979). The user satisfaction survey: consumer evaluation of an inner city community mental health center. Evuluution and the Health Professions, 5, 42-54.

LEVOIS,

CATTELL, R. B. (1978). The scientific use of factor behavioral and life sciences. New York: Plenum Press. EL-GUEBALY, N., TOEWS, (1983). On evaluating patient

analysis

Canadian Journal of Psychiatry, 28, 24-29. ESSEX, opment,

D. W., FOX, J. A., & GROOM, J. M. (1981). The develfactor analysis, and revision of a client satisfaction form.

Community Mental Health Journal, 17, 226-235. GORSUCH, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, Lawrence Erlbaum.

KALMAN, psychiatric

NJ:

T. P. (1983). An overview of patient satisfaction with treatment. Hospital and Community Psychiatry, 34, 48-

54. LARSEN, D. L., ATTKISSON, C. C., HARGREAVES, W. A., & NGUYEN, T. D. (1979). Assessment of client/patient satisfaction: development of a general scale. Evaluation and Progrum Planning,

MAYER, J. E., & ROSENBLATT, between mental patients and staff.

A. (1974). Clash in perspective

American Journal of Orthopsy-

chiatry, 44, 432-441. NUNNALLY, J. C. (1978). York: McGraw-Hill.

Psychometric

theory

(2nd ed.).

New

PASCOE, G. C. (1983). Patient satisfaction in primary health care: a literature review. Evaluation and Program Planning, 6, 185-210. RAY, A. A., SALL, J. P., SAFFER, M., JOYNER, WHATLEY, J. K. (Eds.). (1982). SAS user’s guide. Cary, Institute.

S. P., & NC: SAS

SLATER, V., LINN, M. W., & HARRIS, R. (1982). A satisfaction with mental health care. Comprehensive Psychiatry, 23, 68-74.

2, 197-207. LEBOW, J. (1982). Consumer satisfaction with mental ment. Psychological Bulletin, 91, 244-259.

health treat-

LEBOW, J. L. (1983). Client satisfaction ment. Evaluation Review, 7, 729-752.

health

with mental

TANNER, B. A. (1981). Factors influencing client satisfaction with mental health services: a review of quantitative research. Evaluation

and Program Planning, 4, 279-286. treat-

TANNER, B. A. (1982). A multi-dimensional client satisfaction tionnaire. Evaluation and Program Planning, 5, 161-167.

ques-

LEBOW, J. L. (1984). Research assessing consumer satisfaction with mental health treatment: A review of findings. Evaluation and Pro-

WEINSTEIN, R. M. (1979). Patient attitudes toward mental hospitalization: a review of quantitative research. Journal of Health and

gram Planning, 6, 237-246.

Social Behavior, 20, 231-258.