Patient satisfaction with outpatient psychiatric services

Evaluation and Program Planning, Vol. 16, pp. 109-I 18, 1993 Printed in the USA. All rights reserved.

Copyright

PATIENT SATISFACTION PSYCHIATRIC

0

0149-7189/93 $6.00 + .OO 1993 Pergamon Press Ltd.

WITH OUTPATIENT SERVICES

Qualitative and Quantitative Assessments

MICHEL PERREAULT and PIERRE Community

Psychiatric

Center,

Douglas

Hospital,

ST~PHANE SABOURIN and PIERRE Universitk

de Montreal,

LEICHNER MontrPal

GENDREAU

Montrkal

ABSTRACT The main objectives of this study are, in the coniext of outpatieni services in ps~vchia~ry: (a) to compare quantitative and qualitative sat~sfa~t~o~ measures to evaluate if high satisfart~on scores generally observed are related to specific characteristics of fhe quantitative methodologies that are most often used, and (6) to identify service dimensions that are associated with satisfaction from thepatients’ viewpoint. Personal interviews were conducted with 263 psychiatric outpatients. Open-ended questions werepresented and were followed by the assisted administration of two standardized satisfaction scales: the Client Satisfaction Questionnaire (CSQ-8) and the SHARP-K Comparison of scores derived from the qualitative and quantitative measures showed that expre.~sions of dissatisfaction appeared mainly in open-ended questions. Content analysis also showed that service dimensions most frequentIy evoked by patients were: (a) physica! environnlent and atmosphere, (b) personnel quaIities, and (c) intervention characteristics. Theorefieaf and ~nethodolog~~a~ implications of these results are discussed.

services in particular (Lebow, 1983a) and health care in general (Lebow, 1983b), as well as other domains like job satisfaction (e.g., Gilmer’s study cited in Berger, 1983) or quality of life satisfaction (Baker & Intagliata, 1952). AIthough this does not pose problems for the administrators who seek reactive methods favoring a positive view of intervention, it represents a major obstacle, however, to evaluators who are looking for unbiased indices for decision making information or for scientific purposes (Lebow, 1982). According to LeVois and his colleagues, the elevated rates that are generally obtained represent “one of the most perplexing aspects of the client satisfaction literature” (LeVois, Nguyen, & Attkisson, 1981, p. 140). They

According to Ciarlo (1982), accountability for public mental health programs has become more differentiated: Funders, program managers, and legislators are not only concerned with the numbers and targets of services delivered, but are also looking for data regarding treatment efficacy to justify program implementation and maintenance. In a context in which the Community Mental Health Centres have the obligation to evaluate their programs, much of the work on patient satisfaction has been generated around this requirement (Kalman, 1983). Although patient satisfaction may be a relatively easy way to assess outcome, it presents a number of difficulties. Important questions about the validity of measures of satisfaction have been raised in light of the high level of satisfaction reported in most studies of mental health

This work was supported in part by a grant from the Fends de Rechercheen Sante du QuCbec to the first two authors (FRSQ, grant No. 881315). Requests for reprints should be sent to Michel Perreault, Research Associate, Community Psychiatric Center, Dougla\ Hospital, 6875 boul. I..aSalie, Verdun, QuCbec, CANADA H4H I R3.

109

MICHEL

110

PERREAULT

suggests that this factor alone may be the cause of the high positive responding obtained with satisfaction measures. According to this hypothesis, the specificity of an instrument could have an effect on the distribution of scores. For example, focussing on patients’ experiences with a specific service program could minimize contamination of results that may be due to more generalized attitudes about health or health services (Attkisson, Roberts, & Pascoe, 1983). Consequently, multidimensional scales that cover specific aspects of services should present a less negatively skewed distribution than a unidimensional or general measure. The global scale may not adequately tap aspects of care responsible for relative dissatisfaction (Pascoe, 1983).

suggest different socio-psychological artifacts that could influence the expression of satisfaction/dissatisfaction, including social desirability, response bias, and cognitive consistency. Other authors have referred to methodological factors, especially the sampling procedures that could be biased in favour of heavy users (Blais, 1990), and also by the effect of response anonymity (Soelling & Newell, 1983). The content specificity of the questionnaire may also contribute to the ceiling effects associated with satisfaction measures. Berger (1983) suggests that global scales may elicit responses from the client that do not require discrimination because of the inability of the instrument to make fine gradations of satisfaction levels. He also MULTJDIMENSIONAL Many research projects were conducted in the 1980s on the development of multidimensional satisfaction scales for mental health services (Tanner, 1982; Distefano, Pryer, & Baker, 1983; Holcomb, Adams, Ponder, & Reitz, 1989; Perreault, Leichner, Sabourin, 61 Gendreau, 1992). To different degrees, results obtained with these scales seem to present the same problem already identified with global scales (e.g., high scores, a strong negatively skewed distribution, and difficulties in identifying dissatisfied users). A search of the evaluation literature revealed that only the Evaluation Ranking Scale (ERS) shows a more norma1 distribution than the Client Satisfaction Questionnaire scores (Attkisson, Roberts, & Pascoe, 1983). The ERS consists of a two-stage process: (I) patients rank 6 dimensions of health services delivery (i.e., services ofQUALITATIVE

APPROACH

Another way to improve the content validity of satisfaction scales is to use a qualitative approach to document users’ concerns. Standardized scales focussed on aspects of service delivery judged important by patients may represent a significant improvement over scales constructed from literature searches or under the guidance of a panel (e.g., Larsen, Attkisson, Hargreaves, & Nguyen, 1979, for the CSQ; Tanner, 1982, for the SHARP-V; Attkisson, Roberts, & Pascoe, 1983, for the ERS, etc.). Two recent studies describe such a qualitative approach in developing scales. First, Elbeck and Fecteau (1990) used focus groups in which patients were invited to describe the characteristics of an ideal acute care unit. Unfortunately, the only information provided regarding their methodology concerns their use of three focus groups, one all male (n = 6), one all female (n = 6), and one tnixed (n = 10). QUALITATIVE Results of qualitative evaluation sess patient satisfaction directly.

may also be used to asIn fact, the open-ended

et al.

SCALES fered, service results, locations and appointments, nurses and doctors, etc.) in terms of importance; and (2) patients rate each dimension on a O-100 scale. The ranking task was found to affect the ratings obtained. When the results from a group who had completed both tasks in the usual sequence were compared to those of another group that had to answer the ERS in the reverse order (i.e., rating before ranking), Attkisson and his colleagues found that satisfaction scores obtained in the reverse order were significantly higher. They interpret this finding by attributing an organizing effect to the ranking task whereby patients become familiarized with the dimensions to be rated, yielding resuits that are more normally distributed and adding potentially greater discriminative capacity to the measuring instrument.

IN SCALE

DEVELOPMENT

They do not report qualitative results or their method of using the information gathered to construct their standardized satisfaction scale. Another recent example comes from Holcomb, Adams, Ponder, and Reitz (1989), who used brainstorming sessions with groups of patients to guide the construction of their scale. The resulting list of patient “1ikes”and “dislikes” was presented to 20 direct care professionals for editing, after which it was converted into 5-point Likert scales. Here again, little information is provided about the qualitative methodology and the data analyses involved with the information collected. While the degree to which their scores were negatively skewed is comparable to that of global scales, it is interesting to note that the score of the items on “Hospital Environment” made it possible to identify 34O;‘o“dissatisfied” patients, a percentage that is higher than usual with a global scale. EVALUATION questions included in many questionnaires may serve such purpose, although they have played a minor role

111

Patient Satisfaction up to now. When utilized, open-ended questions are presented frequently at the end of self-administered standardized questionnaires in order to provide fuller information (e.g., Larsen et al., 1979; Love, Caid, & Davis, 1979; Essex, Fox, & Groom, 1981). This “secondary” role is demonstrated in the literature by a lack of information describing procedures, analyses, and results

COMPARISON

OF QUALITATIVE

Weinstein (1979) conducted a critical review of qualitative and quantitative research about patient attitudes towards hospitalization in mental health facilities, and concluded that qualitative research has described patients’ attitudes towards mental hospitals in unfavorable terms. He attributes these findings to problems of reliability and validity (e.g., that the inter-judge reliability of the data gathered was not established, that patients interviewed did not constitute representative samples, that patients were not systematically used as sources of information, and that the types of data collected are almost impossible to quantify). Essex et al. (1980) strongly disagreed with Weinstein’s main conclusions. Their most important concerns are related to the following points: (a) how Weinstein interpreted “favorableness” from quantitative reports, assuming that a score above the scale midpoint means that a patient is satisfied despite the lack of norms to interpret the true meaning of that midpoint; (b) the comparison of two approaches that were assessing two different research questions: qualitative studies that were measuring the experience within a mental hospital versus quantitative studies that were based on patients’ attitudes towards hospitalization; and (c) the “uncritical” acceptance of quantitative studies examined as orientations and assumptions are not questioned as they are for qualitative studies. One of the few studies in which both qualitative and quantitative approaches were systematically compared was conducted by Greenley, Schulz, Nam, & Peterson (1985). The study involved the analysis of results from a group, self-administered, paper-and-pencil questionnaire, which included closed-ended questions followed by open-ended questions. The questionnaire was completed by a total of 177 inpatients. Results from the closed-ended questions revealed a high proportion of “extremely satisfied” and are consistent with the results obtained through open-ended questions showing a majority of satisfied respondents. A group comparison of level of satisfaction on closed- and open-ended questions suggests that patients expressed similar satisfaction levels on both types of questions. Some characteristics of that study may limit, however, the interpretation of the results obtained. Questions may be raised about the acceptability of the procedure used, given that 25% of the patients selected were judged too ill to participate. In addition, only 76 subjects were used for the qualitative analyses out of the 177 who completed

that are based on qualitative assessment of satisfaction. Lebow (1983a), in his review of the literature on patient satisfaction, does not cite any study comparing results from open- and closed-ended questions. The main source of qualitative information mentioned concerns the open-ended feedback obtained from complaints.

AND QUANTITATIVE

EVALUATIONS

“usable” questionnaires, suggesting a large proportion of incomplete open-ended questions. Greenley and his colleagues described three major factors to account for this situation: (a) the extra effort required to formulate a response; (b) mechanical difficulties in writing a response; (c) fatigue associated with the open-ended questions being placed at the end of the questionnaire. While the first factor identified is inherent to open-ended questions, the effects of the last two factors may be reduced by a different methodological procedure. Closed-ended questions may also be related with problematic evaluation procedures: the paper-and-pencil, self-administered form generally used with psychiatric patients may not be particularly appropriate for all psychiatric patients-especially in psychiatric settings-as shown by the high proportion of patients considered too ill to answer the questionnaire (e.g., Gravel, Boyer, & Lamontagne, 1984). As a result, patients who are among the “heavy users” of psychiatric services and whose opinions are consequently important to consider might tend to be underrepresented. Even among the patients who can participate, chronic mentally ill patients seem to have more difficulties as demonstrated by Nguyen, Attkisson, and Stegner (1983), who found a large amount of missing data using the CSQ with this group. Some solutions to improve the suitability of the assessment may be to provide individual assistance to the patient when completing the questionnaire or to have an interviewer read the questions and write the respondent’s answers. While the latter method may simplify the task for the respondent, both methods involve some practical difficulties: (a) individual oral or assisted administration is longer and requires more interviewing skills; (b) the participation of independent interviewers is needed to maintain the anonymity of respondents. Even if the literature on qualitative and quantitative assessments has presented clear findings concerning high satisfaction scores obtained most often with standardized scales, results appear somewhat conflicting in regard to the qualitative assessments. Weinstein’s conclusions that qualitative assessments are biased towards dissatisfaction have been severely criticized by Essex and his colleagues (Essex et al., 1980). Greenley et al. (1985) came to a different conclusion from their study and claim that both qualitative and quantitative approaches produce similar results, but a number of difficulties have been identified with their methodology. The present study constitutes an effort to correct some

112

MICHEL

PERREAULT

of the methodological problems and major weaknesses described by Weinstein (1979) regarding qualitative studies of patients’ opinions. The assessment procedure involved oral administration of questionnaires during personal interviews, with a special effort to draw a representative sample of the clientele from an outpatient psychiatric service. An attempt was also made to collect qualitative data that was quantifiable and that allowed inter-judge reliability to be verified. Our main goal was

et al.

to compare results obtained from standardized satisfaction scales to those obtained through a qualitative assessment. More specifically, we sought to evaluate whether the high satisfaction scores generally observed are related to specific characteristics of quantitative methodologies. The second objective was to identify service dimensions that users of outpatient services spontaneously associated with their satisfaction.

METHOD Setting Patients from two outpatient clinics of the Douglas Hospital community program participated in this study. The Douglas Hospital is a 700-bed psychiatric hospital. The two clinics comprise six multidisciplinary teams serving the adult population of a catchment area of 122,000 in the south-western sector of Montreal. The clientele and the services offered in these clinics may be considered similar to those found in outpatient clinics of general psychiatry departments.

been hospitalized for a psychiatric problem, 25% had one hospitalization, and 34% had two or more.

Subjects The participants in this study were francophone patients attending these clinics. The sample was selected randomly from the “active” files of the clinics- that is, patients who had at least one contact at the clinic in the last 6 months-with no restrictions based on diagnosis or on clinical characteristics of patients. A stratified random sampling technique was used with the stratification variable being the six multidisciplinary teams. Three hundred fifty-three patients were contacted, and 263 (74.5%) agreed to participate. The 90 refusals occurred as follows: (a) 71 patients refused to participate after an interviewer explained the project and tried to schedule an appointment over the telephone; (b) 18 patients refused at the second contact when, at the beginning of the personal interview, the interviewer gave more details about the project and asked the patient to sign a consent form; and (c) one patient never showed up for the appointments he had accepted when contacted by phone. Three percent of the original sample (11 patients) were impossible to reach due to changes in their addresses, phone numbers, or clinics. No statistical differences in terms of age, sex, or distribution of psychotic disorders were found between patients who participated and patients who refused. Subjects who completed the interview ranged in age from 18 to 65, with a mean age of 44.4 years (SD = 13.0). About one-third of the participants were male (36.1070), and two-thirds were female (63.9%). Based on the DSMIII-R classification, half of the participants (50.6%) had a diagnosis of psychotic disorder or mood disorder (“major affective disorder”). “Neurotic” disorders, including anxiety and dysthymia (25%) and adjustment disorders (15%) were the other most frequent diagnostic categories. Forty-two percent of the respondents had never

Measures The Client Satisfaction Questionnaire (CSQ) of Larsen et al. (1979) has been widely used as a global measure of satisfaction and has good psychometric characteristics (Lebow, 1983a). Larsen et al. (1979) constructed a short version of this scale using items that presented the highest loadings on the first factor and that exhibited good inter-item and item-total correlations. A high degree of internal consistency within the instrument was indicated by an alpha coefficient of .93. Many studies have favored the CSQ-8 over the longer versions because its psychometric properties are generally equivalent or superior to those of the CSQ-18 (Nguyen, Attkisson, & Stegner, 1983; Attkisson & Zwick, 1982). Studies done with the French Canadian version of the CSQ-18 have yielded similar results (Sabourin, Perusse, & Gendreau, 1989). In the present study, a high alpha coefficient of .92 was obtained using the French version of the CSQ-8 administered to 263 patients. The SHARP-V is a standardized satisfaction scale of 25 dichotomous (yes/no) items covering 5 different aspects of satisfaction (SHARP is an anagram for Satisfaction, Helpfulness, Accessibility, Respect, and Partnership). Tanner (1982) has presented results from a principal component analysis supporting the multidimensionality of his scale. Grouping the items of the five factors, he obtained a single higher order factor accounting for 42.9% of the total variance and a reliability of .92 when assessed with the Kuder-Richardson KR-20. This total factor presents a mean close to 22 with a skewness of -2.05 and a kurtosis of 4.89. Since the maximum score is 25, the SHARP scale, like most satisfaction scales, does not present a good normal distribution. However, the specificity of the items and the ease of responding to yes/no answers make it distinctive from the

Procedure Personal interviews were conducted by four trained interviewers using a standardized protocol. Interviewers were all master’s level psychology students hired as research assistants for this study. All interviews were held in a separate room in each clinic. The average duration of the interviews was 38 min (SD = 12 min).

Patient Satisfaction CSQ. The SHARP scale was translated into French (using the back-translation method with bilingual translators), pretested, and used in this study. Unfortunately, the multidimensional structure was not replicated with the French translation administered to the 187 participants. A principal component analysis with a varimax rotation gave a one-factor solution, with only one factor having an Eigenvalue higher than 1. These results on the construct validity of the French translation led us to base the analyses of the SHARP-V on the total satisfaction score only while ignoring the subscales’ scores. Since an important objective for the utilization of open-ended questions was the identification of service and treatment dimensions considered important by the patients themselves, it was decided to present openended questions at the beginning of the interview, prior to the standardized questionnaires. This procedure was adopted to minimize the contamination that could result from answering questions about specific service aspects. Questions were asked by an interviewer who wrote down the comments of the respondents (use of a tape recorder was considered but seemed to interfere with the expression of responses). Four open-ended questions were used. The first two are similar to those used by Greenley et al. (1985) and covered satisfaction (“service aspects for which you are satisfied”) and dissatisfaction (“service aspects for which you are not satisfied”). The other two questions were not directly formulated in terms of satisfaction or dissatisfaction, allowing another opportunity for subjects to express their dissatisfaction without the emotional loading associated with the words “dissatisfied” or “not satisfied.” Question 3 and question 4 were “what would you like to change?” and “what do you think about how the clinic is functioning?” In order to identify the service treatment dimensions most frequently referred to, and to create a score that could be compared with the ones generated by standardized measures, answers to the open-ended questions were coded according to the following procedures. The first step involved establishing units on the basis of the grouping of words or sentences corresponding to a same concept. This process was reproduced twice, by two independent coders, in order to estimate the reliability of this method for the segmentation of the comments. A total of 25 1 units were identified for the first

113

question, 148 for the second, 152 for the third, and 226 for the fourth. The inter-judge agreement was high; judges agreed for 96.5% of the unit segmentations for question 1 and for over 98% of the segmentations for questions 2,3, and 4. For the few cases where agreement was not obtained at first, consensus was easily reached after a brief discussion. The second step was related to unit categorization. Category definitions were created on the basis of service dimensions. The procedure was based on inferred categories rather than on the use of a pre-established category scheme developed by investigators (see Deslauriers, 1988, and Weber, 1986). Units were grouped according to their similarities by two independent judges who worked separately. Judges made a list of categories and definitions that were discussed with two other judges. By consensus, this list of definitions was developed to cover five broad categories: (a) environment, which included comments on the atmosphere, physical environment, and locations; (b) personnel, clinical and non-clinical (e.g., “I like my therapist,” “he is competent, ” “the secretaries are OK”); (c) intervention, which covers both the activities offered and the results of the treatment (e.g., “the group meetings are very important,” “it is really helping me”); (d) clinics’ functioning, including service accessibility and choice of professionals (e.g., “I would prefer to see another doctor,” “it takes too long to reach somebody when there is an emergency”); and (e) general comments (e.g., “I like the clinic, ” “ I am satisfied”). Coding of all units by two new coders on the basis of the five main categories gave good intercoder reliability with agreement for 89% of the codings. Conflicting codings were resolved by consensus with a third party. Since the objective of the open-ended questions was to compare results obtained with standardized scales, a score had to be assigned to every unit. A trichotomized (1 = dissatisfaction, 2 = satisfaction for some aspects and dissatisfaction for others, 3 = satisfaction) scheme was used to code all units. The average score was used when a subject expressed more than one comment for a specific category. This scheme was used for the four open-ended questions, and the coders agreed on 95% or more of the codings for each question (i.e., agreement percentage based on number of units where coders agreed divided by the total number of units).

RESULTS Means and standard deviations for the satisfaction measures are presented in Table 1. Both CSQ and SHARP-V distributions were negatively skewed: Even though the possible range for CSQ scores can vary between 8 (lowest satisfaction score) and 32 (highest score), the mean score obtained was 28.4 (SD = 4.1) with a skewness of - 1.8. The SHARP-V scores showed a similar pattern: Even though possible scores can vary between 25 and 50, a mean score of 46.3 (SD = 3.6) was obtained with

a skewness of -2.0. Scores derived from the trichotomized coding of the open-ended questions varied differently depending on the questions. Question 1 was oriented towards satisfaction with different aspects of services (“what are the service aspects for which you are satisfied?“). It elicited responses expressing a very high degree of satisfaction, with a mean score of 2.97 out of a possible score of 3. Question 4 (“how the clinic is functioning”) was more neutral in that

114

MICHEL

PERREAULT

et al.

TABLE 1 DESCRIPTIVE

STATISTICS

OF SATISFACTION

MEASURES Scores Standard

Measure

n

Minimum

Maximum

Mean

Deviation

Skewness

Kurtosis

CSQ-8

253

8

32

28.4

41

-1.8

3.6

SHARP-V

187

28

50

46.3

3.6

-2.0

5.1

1 “Aspects for which you are satisfied” Cl.2 “Aspects for which you are not satisfied” Q.3 “What you would like to change”

252

1

3

2.97

.I 7

-8.6

86.0

149

1

3

1 .l 1

.36

3.9

152

1

3

1.33

.72

1.8

Q.4

226

1

3

2.78

.57

Q.

“How

the clinic

is functioning”

its focus was not directed explicitly towards aspects of satisfaction or dissatisfaction, unlike the other openended questions. A negatively skewed distribution was obtained, however, and like the standardized scales it has generated an elevated satisfaction score (mean of 2.78). Questions 2 and 3 both covered dissatisfaction with the services. The results obtained were different in that scale distributions were positively skewed and the satisfaction averages were low. The mean satisfaction score was 1.11 for question 2 (“the service aspects for which you are dissatisfied”); question 3 with a somewhat less reactive wording (“the things you would like to change”) yielded 2 responses out of 10 related to satisfaction with the majority of responses expressing dissatisfaction (see Table 2). A low average of 1.33 was obtained for this question. It is important to add, however, that response rates varied for each question. As indicated by Table 2, the number of subjects who expressed comments to question 1 (“satisfaction”) was about two times higher than for questions 2 and 3 (“dissatisfaction” questions). These results suggest that more subjects had “positive” comments about the services. However, even if the “dissatisfaction” questions did not elicit as many answers as the satisfaction questions, they seem to tap dissatisfaction successfully as compared to the other open-ended questions and the standardized scales. In fact, a large proportion of subjects expressed dissatisfaction only in response to these questions. Also, more than 45% of the 252 subjects who gave a satisfaction answer to the first open-ended question (“aspects for which you are satisfied”), also gave a dissatisfaction answer for the second open-ended question (“aspects for which you are not satisfied”). A comparison of satisfaction scores from open- and closed-ended questions also suggests a tendency for a high proportion of “satisfied” subjects to express dissatisfaction when given the opportunity. Table 2 presents crosstabulations of the coding of open-ended questions and CSQ scores. Due to the negative skewness of the satisfaction scores, these were divided into two catego-

16.0 1.5

-2.5

4.6

ries. A cut-off point of 30 was chosen to distinguish the high scores (“perfect” satisfaction, when eight ratings of satisfaction were given for every CSQ-8 item). In the remaining category, subjects gave a “somewhat dissatisfied” score, defined by one or more “not satisfied” or “not very satisfied” answers. There were 200 subjects

TABLE 2 CODING

OF OPEN-ENDED

QUESTIONS

TO CSQ-8

IN RELATION

ANSWERS CSQ-8

Open-Ended

1. “Service

aspects

“Perfect”

“Somewhat

Satisfaction

Dissatisfied”

Score

Score

n

Questions for which

%

n

%

97.0

49

94.2

3.0

2

3.8

you

are satisfied” Satisfied

194

Somewhat

satisfied

6

Dissatisfied

2. “Service

0 _~___ 200

aspects

for which

0 100

1 52

1.9 100

you

are not satisfied” Satisfied Somewhat

satisfied

Dissatisfied

2

1.9

1

2.2

10

9.7

3

6.6

_*s_ 91 103

3. “What

you would

20

17.4

2

5

4.4

1

90

78.3 ___-__ 100

37

89.3

35

satisfied

Dissatisfied

115 4. “How

the clinic

159 satisfied

Dissatisfied

“Perfect” fied”

satisfaction

answer

“Somewhat faction

100

34

5.4 2.7 91.9 100

is functioning”

Satisfied Somewhat

46

like to change”

Satisfied Somewhat

91.3 100

score:

72.9

9

5.1

6

12.5

10 ~~-178

5.6

7

14.6

gave either

100

a “very

48

satisfied”

100 or “satis-

to the 8 items of the CSQ-8. dissatisfied”

score:

gave at least

to the 8 items of the CSQ-8.

1 answer

of dissatis-

Patient Satisfaction

CORRELATIONS

TABLE 3 BETWEEN THE DIFFERENT SATISFACTION CSQ-8

Standardized CSQ-8 SHARP-V

SHARP-V

Q.l

.I 2 (n=183) -.08 (n = 108) .20* (n = 109) .27” (n = 169)

-.37$ (n=145) -.31$ (n = 150) -.06 (n = 222)

MEASURES Q.2

0.3

Q.4

-

-

-

-

-

.I0 (n= 135)

-

.71 T (n = 186)

Q.2 “Aspects for which you are not satisfied” 0.3 “What things you would like to change” Q.4 “How the clinic is functioning”

‘p <

AND DISSATISFACTION

satisfaction measures

Scores derived from open-ended questions Q. 1 “Aspects for which you are satisfied”

**p

115

.12* (n = 248) .02 (n = 148) .25T (n = 150) .30t (n = 224)

.39$ (n=104) .ll (n = 132)

-

.05

< .Ol

tp < ,001

$p < .OOOl

who obtained these subjects, ions from the ond and third

Correlations between the different satisfaction measures are presented in Table 3. The highest correlation coefficient was between the standardized scales, that is the CSQ and the SHARP-V (r = .71). Question 4, on

“perfect” CSQ scores. Close to half of however, expressed some negative opintwo “dissatisfaction” questions (i.e, secopen-ended questions).

CLASSIFICATION

TABLE 4 OF SCORES DERIVED FROM OPEN-ENDED QUESTIONS ON SERVICE SATISFACTIONa Categories

Questionsb I. “Service aspects for which you are satisfied” Satisfied Somewhat satisfied Dissatisfied 2. “Service aspects for which you are not satisfied” Satisfied Somewhat satisfied Dissatisfied 3. “What you would like to change” Satisfied Somewhat satisfied Dissatisfied 4. “How the clinic is functioning” Satisfied Somewhat satisfied Dissatisfied

Cinics’ Functioning

Intervention

Personnel

Environment n

%

n

%

n

%

n

%

114 1

98.3

90

97.8

2.2 0 100

7 1

116

2 0 92

31 4 0 35

88.6

1 .o 1 .o 100

0 8

87.5 12.5 0 100

3 7 88 98

3.1 7.1 89.8 100

0 5 38 43

0 11 .6 88.4 100

0

0

22 3 110 135

16.3 2.2 81.5 100

11 1 0 i?

91.6 8.3 0 100

167 7 14 188

88.8 3.7 7.4 100

26 7 2 35

74.3 20.0 5.7 100

1

0 1

5 -6 1 1 3 T 2 0 1 3

11.4 0 100 0 16.7 83.3 100 20.0 20.0 60.0 100 66.7 0 33.3 100

0

0

1 i

100 100

0 0

-

0

-

0

-

0

-

0

-

0

-

0

-

aBase of n = 258 %omments emitted were classified in three categories: (1) satisfaction, (2) satisfaction on some aspects but dissatisfaction on some others, (3) dissatisfaction. Comments from the three categories were emitted for every question.

116

MICHEL

PERREAULT

how clinics are functioning, covers general satisfaction and appeared to give results more similar to standardized scales than the other 3 open-ended questions, which were oriented either towards satisfaction or dissatisfaction opinions. That question was significantly correlated with the standardized scales (.30 with CSQ and .27 with the SHARP-V) but was not significantly correlated with the other open-ended questions (Table 3). As expected, dissatisfaction scores (i.e., second and third open-ended questions) were significantly correlated among themselves (.35) and negatively correlated with the first openended question on satisfaction (--.37 and -.31). Results from the open-ended questions also contributed to another major objective of this research project, that is, to identify service dimensions that were considered to be important by users. Table 4 presents results of the content analysis based on answers generated by

et al.

the open-ended questions. Answers were grouped according to the service dimensions referred to by the patients. Dimensions that were most often spontaneously mentioned with regard to satisfaction and dissatisfaction were: (a) physical environment and atmosphere (69% of the total answers); (b) personnel (23.4%); (c) intervention characteristics (63@i’o); and (d) clinics’ functioning (1.2%) Distribution of the responses according to the trichotomized categorization (satisfaction, mixed satisfaction, and dissatisfaction) presented on Table 4 suggests that the satisfaction level may vary from one service aspect to the other. For example, at the question on “how the clinic is functioning,” 11.1% of the comments on the environment are related to dissatisfaction or mitigated satisfaction, while, for the same question, 25.6% of the comments on personnel fall within the same categories.

DISCUSSION Based on interviews with 263 patients, this study is, to our knowledge, the first to compare results systematically from a quantitative and a qualitative approach to the assessment of patient satisfaction with psychiatric outpatient services. This study is characterized also by the use of oral administration of questionnaires during personal interviews, a procedure that appears to be better adapted to the psychiatric clientele than the selfadministered paper-and-pencil form most often used. Although open-ended questions may have allowed patients to evaluate more precise aspects of services than general satisfaction scales, scores were not more likely to be normally distributed. All open-ended and closedended questions produced negatively skewed distributions, with the exception of open-ended questions oriented towards dissatisfaction. Our findings suggest that the fact that questions are open- or closed-ended is not the only determinant of the type of distribution obtained. The focus of the questions and the acceptability of the data gathering procedure appears to influence the responses obtained. The focus of the questionsthat is, whether they are oriented towards satisfaction or dissatisfaction measurement - seems to act as an important factor that influences the type of answers that are provided. When subjects are asked about their satisfaction with services, or even when they have to respond to more “neutral” questions, such as “how is the clinic functioning?,” they tend to express satisfaction answers. For these questions, responses were distributed similarly, or were even more skewed than were the standardized scales, with very high satisfaction scores. A majority of patients expressed, however, “negative” comments about services when they answered open-ended questions oriented towards the aspects of services for which they were not satisfied. Such results were not obtained by Greenley et al. (1985) with

their hospitalized patients; they concluded that results would be the same whether qualitative or quantitative methods were used. The differences in the results of their study and our findings justifies a closer look at the two procedures used, in order to evaluate how the characteristics of each methodology might interact with the expression of dissatisfaction. The procedure used in our study was based on personal interviews initiated with oral administration of the open-ended questions; Greenley et al. (1985) used a group self-administered questionnaire in which openended questions had been at the end. While 23% of their subjects made no response to the open-ended questions, the overall nonresponse rate was less than 4% for the corresponding questions in our study. As predicted, our oral questionnaire administration appears to make formulation of answers to open-ended questions easier than the self-administered questionnaire. Another explanation may be related to the “reactivity” of the test situation as it seems particularly sensitive to demand characteristics. It is possible that our subjects perceived that the procedure provided for a better protection of their anonymity as compared to the subjects who underwent assessment in groups. In any case, the expression of opinions appears to be facilitated by our procedure, and the increased response rate may itself have had an impact on the type of responses obtained-that is, if nonresponders tend to be less satisfied with services, a procedure that increases the response rate should provide an overall satisfaction profile that is fess “positive.” It is apparent that two important characteristics of the qualitative methodology used- the focus of the questions and how adapted is the evaluation protocol for the population surveyed-may have an impact on satisfaction reports. In this light, such a qualitative methodol-

Patient

ogy may be used advantageously in different evaluation contexts. One advantage of the open-ended questions lies in the potential to identify patients’ preoccupations in relation to the services provided to them. Another advantage is the opportunity to measure aspects of the patients’ opinions that do not seem to be tapped by standardized instruments, as shown by responses to dissatisfaction questions of the present research. When comparing our results and our methodology to Greenley et al. (198.5), it is possible to conclude that closed-ended questions could adequately elicit dissatisfaction without the need for a qualitative approach in that results depend more on the concept measured (i.e., satisfaction or dissatisfaction) than on the approach used to measure it (i.e., qualitative or quantitative). While our study was not designed specifically to test this hypothesis, our results do not tend to support this “wording/phrasing” hypothesis. Tanner’s SHARP scale is constituted of 25 dichotomic items, 10 of which address possible dissatisfaction (e.g., “Were you disappointed by what happened here?” “Was there a lot of ‘red tape’ involved in being seen here,” “Did your therapist seem to dislike you,” etc.). The total score of the scale, however, is highly correlated with the CSQ, and the “dissatisfaction” items are distributed in a manner similar to the other scales. Again, it seems that dissatisfaction comments are expressed most readily in a qualitative context, and only when the data-gathering procedure has high “acceptability” to the subjects. It is possible that satisfaction and dissatisfaction are perceived as two different concepts, and that two different measurement instruments should be developed and used to produce a more valid assessment of services from the patients’ viewpoint. Research conducting a systematic comparison of satisfaction and dissatisfaction assessments using both qualitative and quantitative methods would be useful in testing this hypothesis. Some questions could be raised about the applicabil-

Satisfaction

117

ity of the method used in this study to other situations where program evaluation must be carried out with limited resources. Our use of the qualitative approach involved a large amount of work to quantify the data and evaluate the inter-judge reliability at different steps of the process. One of the main objectives for such a procedure was to compare the results to the ones obtained from quantitative satisfaction measures. For program evaluation purposes, this costly procedure is not necessarily appropriate. In a formative evaluation context, in which the main objective is to generate useful information for better decision making about the services given, a more appropriate methodology might consist simply of complementing the information provided by an easy-to-use standardized and validated scale with focus groups or in-depth interviews. This form of triangulation, where qualitative and quantitative approaches are used in tandem, would not be as costly and time consuming as critics have suggested (i.e., Reichardt & Cook, 1979). It might meet many of the objectives of patient satisfaction surveys if standardized scales are used for monitoring purposes, while qualitative information provides indications of what created an improvement or a deterioration of the situation. As indicated by this study, qualitative information also can generate interesting insights into improving the content validity of the evaluation instruments and protocols. Our results suggest that satisfaction with the physical environment and atmosphere and with qualities of the treating and clerical personnel are aspects of services that need to be covered in order to insure good content validity of the satisfaction measures for outpatient services. A large proportion of the patients interviewed have linked these aspects of services spontaneously to their satisfaction with services received. Development of multidimensional scales along these broad service dimensions could help create more specific and valid program evaluations that would be more sensitive to patients’ concerns.

REFERENCES ATTKISSON, C.C., & ZWICK, R. (1982). The Client Satisfaction Questionnaire: Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program Planning, 5, 233-231.

DeJulio. The assessment York: Wiley.

ofpsychotherapy

outcome

(pp. 56-80). New

BLAIS, R. (1990). Assessing patient satisfaction with health care: Did you drop somebody? Canadian Journal of Program Evaluation, 5, I-13.

ATTKISSON, C., ROBERTS, R.E., & PASCOE, G.C. (1983). The evaluation ranking scale: Clarification of methodological and procedural issues. Evaluation and Program Planning, 6, 349-358.

CIARLO, J.A. (1982). Accountability revisited: The arrival of client outcome evaluation. Evaluation and Program Planning, 5, 31-36.

BAKER, F., & INTAGLIATA, J. (1982). Quality of life in the evaluation of community support systems. Evaluation and Program Planning, 5, 69-79.

DESLAURIERS, five [Qualitative du Quebec.

BERGER, M. (1983). Toward maximizing the utility of consumer satisfaction as an outcome. In M.J. Lambert, E.R. Christensen, & S.S.

DISTEFANO, M.K., PRYER, M.W., & BAKER, F.B. (1983). Factor structure of a clients’ satisfaction scale with psychiatric inpatients. Psychological Reports, 53, 1155-l 159.

J.P. (1988). Les mkthodes de la recherche qualitaresearch methods]. Quebec: Presses de l’universite

MICHEL

118

PERREAULT

et al

ELBECK, M., & FECTEAU, Cl. (1990). Improving the validity of measures of patient satisfaction with psychiatric care and treatment. Hospital and Community Psychiatry, 41, 998-1001.

LEVOIS, M., NGUYEN, T.D., & ATTKISSON, C.C. (1981). Artifact in client satisfaction assessment: Experience in community mental health settings. Evaluation and Program Planning, 4, 139-150.

ESSEX, M., ESTROFF, S., KANE, S., McLANAHAN, S., ROBBIN, J., DRESSER, R., & DIAMOND, R. (1980). On Weinstein’s “Patient attitudes toward mental hospitalization: A review of quantitative research.” Journal of Healih and Social Behavior, 21, 393-396.

LOVE, R.E., CAID, C.D., & DAVIS, A. (1979). The user satisfaction survey: Consumer evaluation of an inner city community mental health center. Evaluation and the Health Professions, 1, 42-54.

ESSEX, D.W., FOX, J.A., & GROOM, J.M. (1981). The development, factor analysis, and revision of a client satisfaction form. Communit_v Mental Health Journal, 17, 226-235. GRAVEL, G.B., tude des malades patients’ attitude 10, 39-44.

BOYER, R., & LAMONTAGNE, Y. (1984). Attichroniques envers l’hopital psychiatrique [Chronic concerning the psychiatric hospital]. L’Encephale,

GREENLEY, J.R., SCHULZ, R., NAM, S.H., & PETERSON, R.W. (1985). Patient satisfaction with psychiatric inpatient care: Issues in measurement and application. Research in Community and Mental Health, 5, 303-319. HOLCOMB, W.R., ADAMS, N.A., PONDER, H.M., & REITZ, R. (1989). The development and construct validation of a consumer satisfaction questionnaire for psychiatric inpatients. Evaluation and Program Planning, 12, 189-194. KALMAN, T.P. (1983). An overview of patient satisfaction with psychiatric treatment. Hospital and Community Psychiatry, 34, 48-54. LARSEN, NGUYEN, velopment 197-207.

D.L., ATTKISSON, C.C., HARGREAVES, W.A., & T.D. (1979). Assessment of client/patient satisfaction: Deof general scale. Evaluation and Program Planning, 2,

LEBOW, J.L. (1982). Pragmatic decisions in the evaluation of consumer satisfaction with mental health treatment. Evaluation and Program Planning, 5, 349-356. LEBOW, J.L. (1983a). Research assessing consumer satisfaction with mental health treatment: A review of findings. Evaluation and Program Planning, 6, 21 l-236. LEBOW, J.L. (1983b). Similarity and differences between mental health and health care evaluation studies assessing consumer satisfaction. Evaluation and Program Planning, 6, 231-245.

NGUYEN, T.D., ATTKISSON, C.C., & STEGNER, B.L. (1983). Assessment of patient satisfaction: Development and refinement of a service evaluation questionnaire. Evaluation and Program Planning, 6, 299-314. PASCOE, G.C. (1983). Patient A literature review and analysis. 6, 185-210.

satisfaction Evaluation

in primary health care: and Program Planning,

PERREAULT, M., LEICHNER, P., SABOURIN, S., & GENDREAU, P. (1992). Caracteristiques psychometriques de la version Canadienne-Francaise d’une tchelle de satisfaction pour patients hospitalises en psychiatric [Psychometric properties of the FrenchCanadian translation of a satisfaction scale for hospitalized psychiatric patients]. Canadian Journal of Psychiatry, 37, 221-227. REICHARDT, C.S., &COOK, T.D. (1979). Beyond qualitative versus quantitative methods. In T.D. Cook & C.S. Reichardt, Qualitative and quantitative methods in evaluation research (pp. 7-49). Beverly Hills: Sage. SABOURIN, S., PERUSSE, D., & GENDREAU, P. (1989). Les qualites psychometriques de la version Canadienne-Francaise du Questionnaire de Satisfaction du Consommateur de services psychotherapeutiques. Revue Canadienne des Sciences du Comportement, 21, 149-159. SOELLING, M.E., &NEWELL, T.G. (1983). Effects of anonymity and experimenter demand on client satisfaction with mental health services. Evaluation and Program Planning, 6, 329-333. TANNER, B.A. (1982). A multi-dimensional ment. Evaluation and Program Planning, WEBER,

R.W. (1986). Basic content

client satisfaction 5, 161-167.

analysis.

Beverly

WEINSTEIN, R.M. (1979). Patient attitudes toward talization: A review of quantitative research. Journal Social Behavior, 20, 237-258.

instru-

Hills: Sage.

mental hospiof Health and