- Email: [email protected]
Patient satisfaction with mental health services: A meta-analysis to establish norms
and Program Plannmg, Vol. Printed in the USA. All rights reserved.
Evaluation
PATIENT
6,
pp.
265-214,
SATISFACTION
0149-7189/83 $3.00 + .OO CopyrIght 0 1984 Pergamon Press Ltd
1983
WITH MENTAL
A Meta-Analysis
ANTHONY F. LEHMAN
HEALTH
SERVICES:
to Establish Norms
and THOMAS R. ZASTOWNY
University of Rochester School of Medicine
ABSTRACT Patients typically express high rates of satisfaction with their mental health care. This finding and the lack of well controlled studies on patient satisfaction in the literature underscore the need for meaningful guidelines for clinicians and program evaluators in interpreting patient satisfaction data. To address this problem a meta-analysis was undertaken to establish norms on patient sattifaction for various types of mental health programs. Programs were categorized according to three dimensions: inpatient vs. outpatient vs. residential care; chronic vs. nonchronic; and conventional vs. innovative. Meta-analysis procedures were modified to accommodate the single-group study designs that dominate the literature. The analysis revealed that chronic patients express less satisfaction with their treatment compared to non-chronic patients. Innovative programs are viewed more positively than conventional ones. No differences were found in rates of patient satisfaction between inpatient and outpatient programs. Acceptably reliable norms and confidence intervals of patient satisfaction were established for conventional inpatient programs serving either chronic or non-chronic patients; conventional outpatient programs for non-chronic patients; and for all programs combined according to chronic vs. non-chronic, inpatient vs. outpatient, and conventional vs. innovative. However, data were insufficient to compute norms for other program types. The norms thus established can be used for comparative purposes by program evaluators. A cumulative, national data base on patient satisfaction is recommended to further refine these norms.
Mental health professionals have shown a sustained and growing interest in patient satisfaction with mental health services, despite persistent conceptual ambiguity and problems in the measurement of patient satisfaction. Several articles in this current volume as well as recent review articles provide extensive discussions of the measurement and meaning of patient satisfaction with mental health services (Kalman, 1983; Lebow, 1982; Weinstein, 1981). It is well established that the majority of patients express positive attitudes about their mental health care. Although this finding may be encouraging news for care providers, it underscores the need for meaningful guidelines to assist clinicians and program evaluators in interpreting the typically high levels of satisfaction expressed by their patients. In this paper we present a meta-analysis of the existing quantitative data in the literature on patient satisfaction with mental health services in order
to establish preliminary norms against which to compare specific programs. This analysis provides a further refinement of the literature reviewed elsewhere (Lebow, 1982; Weinstein, 1981), and identifies areas in which existing data are insufficient to establish norms. Unfortunately, well-controlled studies of patient satisfaction are rare in the literature, and experimental designs are usually not practical when program evaluations are requested by clinicians or administrators. Nonetheless there must be some reference points from which to interpret levels of patient satisfaction if such evaluations are to be of any value other than to cast a positive glow upon most programs. One option for developing such reference points is to pool data from several studies, none of which alone provides an adequate comparison, but which together can provide normative information for comparative purposes. Such comparisons complement well-designed studies,
Requests for reprints should be sent to Anthony F. Lehman, Department of Psychiatry, University of Rochester Medical Center, 300 Crittenden Blvd.,
Rochester,
NY 14642.
265
266
ANTHONY
F. LEHMAN
and are preferable to the usual situation in which a single program is evaluated cross-sectionally and the results interpreted in a relative vacuum. To approach this program, we developed a classification scheme for mental health programs along three program dimensions: (a) inpatient vs. outpatient vs. residential care; (b) chronic vs. non-chronic patients; and (c) conventional vs. innovative treatment. Although a variety of other program characteristics may be related to patient satisfaction, we chose these three dimensions because they are basic and because most existing programs can be classified readily into 1 of the 12 subtypes defined by this 3 x 2 x 2 typology. This classification system formed the basis to conduct a meta-analysis of patient satisfaction according to program type. Meta-analysis refers to various analytic procedures for reviewing the literature (Rosenthal, 1980) pioneered by Smith and Glass (1977). It is particularly useful for combining the results of independent studies to assess questions that no single study can address. Early metaanalyses were targeted on psychotherapy outcome studies, but in recent years the technique has been applied to a variety of topics including various reward systems (Cotton 8z Cook, 1982), sex differences (Cooper, 1979), and surgical recovery (Devine & Cook,
and THOMAS
R. ZASTOWNY
in press). The interested reader is referred to an excellent article by Rosenthal (1983) for an overview and some illustrations of meta-analysis procedures. The two most commonly employed statistics in metaanalysis are effect size (ES) and percent difference (PO>. Effect size is calculated as the difference between the means of two groups divided by the standard deviation of the control group. The PD is computed as the difference between the group means divided by the control group mean, multiplied by 100. Various procedures to weight studies for sample size and other relevant conditions are available and the interested reader is referred to the Journal of Consulting and Clinical Psychology (51 [I], 1983) for a series of reviews on meta-analysis. Applying meta-analytic techniques to the available literature, we sought answers to the following questions: 1. What are the typical levels of patient satisfaction with the various types of mental health programs? 2. Do the levels of patient satisfaction vary according to type of program (inpatient vs. outpatient vs. residential, conventional vs. innovative) or type of patient (chronic vs. non-chronic)? 3. Can norms of patient satisfaction be established for the various program types?
METHODS The studies analyzed were identified from three recent, comprehensive reviews of patient satisfaction with mental health services (Kalman, 1983; Lebow, 1982; Weinstein, 1981), supplemented by a computerized library search of the literature on patient satisfaction and psychiatric care since 1981. Of 98 empirical studies thus identified, 35 studies (37 programs) had sufficient quantitative data for inclusion in the analysis on rates of patient satisfaction; 27 studies (33 programs) had sufficient data for inclusion in a meta-analysis of mean satisfaction levels. A total of 52 separate studies (59 programs) were thus reviewed. These are included in the Appendix. Each study was abstracted for the following information: (a) year of study, (b) type(s) of programs, (c) sample size, (d) procedure for assessing satisfaction, (e) rate of satisfaction (r70), (f) rate of dissatisfaction (o/o), (g) satisfaction scale range and midpoint, (h) mean satisfaction score, and (i) standard deviation of satisfaction score. The studies were classified according to the matrix in Table 1. The differentiation of inpatient, outpatient, and residential care programs is self-explanatory. “Chronic” patients were identified as persons with a life history of psychiatric hospitalizations totaling at least 6 months. For studies that did not specify the amount of hospitalization, we accepted patients as chronic if they were described as “chronic”
or if the site of treatment was a state or Veterans Administration hospital. Finally, programs were classified as “innovative” only if they were specifically identified as a new program or were presented as improvements over the standard type of treatment available. All other programs were considered conventional. The rarity of studies that provide between-group comparisons of patient satisfaction presented an initial methodologic problem in this review because metaanalytic techniques rely on such comparative data. To deal with this, we employed a modified standard normal deviate, z-score, computed as follows: 7 = (Observed mean) - (Scalemidpoint) . Observed standard deviation
(1)
For those studies in which an observed mean was available, but a standard deviation was not, the standard deviation was estimated from the ratio of mean satisfaction/standard deviation in the studies that provided both statistics. This z-score reflects the deviation of the observed mean satisfaction from the numerical midpoint of the satisfaction scale, the hypothetical 50% satisfaction reference point. This reference value was used because it is the level that would be observed by chance alone and because it is the null hypothesis implicit in many
A Meta-Analysis to Establish Norms
267
dent’s t-test and for calculation of confidence intervals around the mean satisfaction levels in different types of programs. In all analyses, studies were weighted according to sample size (Cooper, 1979). As such, the procedures represent a modification of available metaanalytic techniques and provide a method for conducting meta-analysis on literature with predominantly single-group designs.
satisfaction studies. We then used these modified standard normal deviates from individual studies as the data base for computing meta-analytic statistics, including x2 tests for heterogeneity of study results and the Stouffer z-statistic and “fail-safe N” for assessing the level of confidence in the results (Cooper, 1979; Rosenthal, 1983). Finally, these z-scores were used for between-group comparisons by means of Stusingle group
RESULTS rates of satisfaction and dissatisfaction were 70.6% and 18.970, respectively. The lowest level of satisfaction was recorded in conventional, residential care programs for chronic patients (49.5%); the highest rates of dissatisfaction were recorded in conventional chronic inpatient programs (30.5%), innovative nonchronic outpatient programs (27.4%), and conventional residential care programs (22.3%).
Studies by Year and Program Type A total of 59 programs in 52 studies was analyzed. Table 1 summarizes the numbers and types of programs studied during each 5-year interval between 1955 and February 1983. The number of quantitative studies increased steadily during that time, particularly since 1970. The vast majority of programs were conventional (51/59 = 86~0), 47~0(28/59) were inpatient, and 58%(34/59) served patients with non-chronic disorders. The types of programs studied shifted from predominantly chronic inpatient programs prior to 1970 to predominantly non-chronic outpatient and inpatient programs after 19’70. Since 1975 there has been increasing interest in long-term residential care programs.
Meta-Analysis of Standardized Satisfaction Scores The high degree of skewness in patient satisfaction among the studies can be appreciated by comparison to a normal standard z-distribution. In a normal standard ~-distribution, 50% of scores are less than 0.0. In contrast, no study found a standardized satisfaction score less than 0.0, i.e., none found a mean satisfaction level below the numerical midpoint of the scale used for measuring satisfaction. The median standardized satisfaction score was 1.70, and the mean satisfaction score weighted for sample size was 1.47. If the
Rates of Patient Satisfaction and Dissatisfaction by Program Type The rates of satisfaction were high, exceeding 50% for all but one program type (see Table 2). The overall
TABLET
MENTAL HEALTH PATIENT SATISFACTION STUDIES REVIEWED BETWEEN 1955 AND 1983 BY YEAR AND PROGRAM TYPE
Years of Study Type of Program I. Inpatient A. Chronic 1. Conventional 2. innovative B. Non-Chronic 1. Conventional 2. Innovative Il. Outpatient A. Chronic 1. Conventional 2. Innovative B. Non-Chronic 1. Conventional 2. Innovative Ill. Residentiala A. Chronic 1. Conventional 2. Innovative Totals
1955-59
1960-64
1965-69
1970-74
1975-79
1980-83
Total Years
2 0
2 1
3 0
4 0
3 0
0 0
14 1
0 0
3 0
0 0
1 0
6 0
2 1
12 1
0 0
0 0
0 0
0 0
0 0
4 0
4 0
0 0
0 0
0 1
5 0
9 1
4 1
18 3
0 0 2
0 0 6
0 0 4
0 1 11
2 2 23
1 0 13
3 3 59b
a No non-chronic residential programs were reported. bPrograms were evaluated in 52 separate studies.
268
ANTHONY F. LEHMAN and THOMAS R. ZASTOWNY
RATES
OF PATIENT
Program
SATISFACTION
Type
Inpatient Non-Chronic, conventional Non-Chronic, innovative Long-Term, conventional Long-Term, innovative Outpatient Non-Chronic, conventional Non-Chronic, innovative Long-Term, conventional Long-Term, innovative Long-Term residential Conventional Innovative Overall
TABLE 2 AND DISSATISFACTION
Percent Satisfaction
78.2 85.0 69.4 -
(WV
(6;545) (1;33) (8;1,136)
BY PROGRAM Percent Dissatisfaction
TYPE
(KN)a
15.5 -
(3;380) -
30.5 -
(4;601) -
13.3 27.4 -
(11;1,500) (2; 124) -
78.0 64.0 73.3 -
(x72) (2;62) -
49.5 77.1
(3,129)
22.3 8.7
(3;878) (3;129)
70.6
(37;4,819)
18.9
(26;3,612)
(13;1,964)
(3;878)
Note. Rates reflect weighted adjustments for sample aK = number of programs, N = number of patients
distribution of scores was normal, the median and mean scores would be 0.0. In fact, the observed median and mean satisfaction scores were between one and two standard deviations above this hypothetical mean. The first issue addressed in the meta-analysis was whether or not there was heterogeneity among the satisfaction scores; i.e., was there evidence for significant variability among them? Using the x2 statistic of Rosenthal and Rubin (Rosenthal, 1983), we found significant heterogeneity (x2 = 49.6, df = 30, p < .025). Given this finding, the next question addressed was whether this heterogeneity could be accounted for by variability among studies within program types or by variability across program types. The latter issue would be of particular relevance in the development of program-specific norms. Table 3 displays the weighted mean satisfaction scores, standard deviations, numbers of studies, ranges of scores, and tests for heterogeneity for the various program types. The distribution of these studies by program type is shown in Figure 1. Significant heterogeneity was found only among the studies of conventional, chronic inpatient programs, and could be accounted for entirely by one study with a standardized mean satisfaction of 4.93 (Rice, Klett, Berger, Sewall, & Lemkau, 1963). The next highest score among that group of studies was 2.59. Removal of this outlying study produced a weighted mean satisfaction score of 0.88 for these chronic inpatient programs. To avoid bias, this outlying study was excluded in all subsequent analyses. However, even after removal of this study the trend remained for significant heterogeneity among all the studies (x’ = 40.74, df = 29, 0.10 > p > 0.05).
sizes of individual
studies.
Next we considered whether or not the levels of patient satisfaction varied across program types. Based upon the data presented in Table 3, we found that chronic patients tended to express less satisfaction with their treatment than did non-chronic patients for all program types combined [z(chronic) = 1.41, ,-(nonchronic) = 1.90; t = 1.29, p = .10 (one-tailed)]. This trend held for both inpatient and outpatient services, although it was statistically significant only for outpatient programs [:(chronic, inpatient) = 0.88, inpatient) = 1.62, t = 1.23, n.s.; z(non-chronic, z(chronic, outpatient) = 1.Ol, z(non-chronic, outpatient) = 2.20, t = 2.46, p = .025 (one-tailed)]. No significant difference in patient satisfaction was found between inpatient and outpatient programs (;(inpatient) = 1.61, z(outpatient) = 1.89; t = 0.71, n.s.1. Finally, innovative programs as a group were associated with greater patient satisfaction than conventional programs [,-(innovative) = 3.12, z(conventiona1) = 1.71; t = 2.33, p < .025]. Preliminary Norms and Confidence Intervals for Satisfaction with Program Types Using the data from Table 3 it was possible to compute confidence intervals for the mean satisfaction levels for various program types. These 90% and 95% confidence intervals are given in Table 4 and provide norms against which additional program evaluations can be compared. It should be noted that for some categories of programs the small number of studies yields confidence intervals that are too broad to permit meaningful comparison. Based upon the confidence intervals given in Table 4, we can conclude that inadequate data exist to establish preliminary norms for conventional outpatient programs for chronic patients
A Meta-Analysis
DESCRIPTIVE
STATISTICS
ON MEAN
to Establish Norms
TABLE 3 SATISFACTION
LEVELS
269
FOR VARIOUS
PROGRAM
Type
Mean za (Standard Deviation)
N of Subjects; NC of Studies
Range of Unweighted z-Scores
I. Inpatient A. Chronic 1. Conventional
1.13(1.51)
2138;8
0.15-4.93
0.88(1.11)
1996;7
0.15-2.59
2. Innovative B. Non-Chronic 1. Conventional
5.23(NAb)
76;l
NA”
1.62(1.14)
968;7
0.0-3.25
2. Innovative II. Outpatient A. Chronic 1. Conventional
2.20(NAb)
33;l
NAb
l.Ol(O.54)
411;3
Program
[Outlying
study
removed]
2. Innovative B. Non-Chronic 1. Conventional
2. Innovative
III.
Residential (Chronic) A. Conventional B. Innovative
TYPES
Heterogeneity Test
x’ = 16.03 df = 7 p = .025 x1 = 7.36 df = 6 p =NS NAb x1 = 7.85 df = 6 iib
-
0.80-l
-
57
-
2.20(0.72)
929;8
0.84-2.95
2.35( 1 .OO)
156;3
0.91-4.07
0.85(NA”) 0.81 (NAb)
278; 1 54;l
az-Scores from individual studies are weighted for sample size in calculations bNA = not applicable because only one study is available. CN includes only those studies that reported mean satisfaction levels.
and for the four program types in which only one study was available for analysis. For other classes of programs, the confidence intervals are sufficiently narrow to permit reasonable comparisons with new studies. Also given in Table 4 are z-scores, computed by the Stouffer method (Rosenthal, 1983), which indicate the probability that the results of multiple studies combined are due to chance, i.e., the likelihood of a Type I error in the meta-analysis. The “fail safe” numbers of studies are also given in Table 4. These numbers indicate the smallest number of additional or unreported studies showing no difference between the observed mean satisfaction and the scale midpoint that would be needed to reverse the conclusion of a significant difference between the observed mean and midpoint. The larger this number, the greater the confidence that a true difference exists. The Stouffer z and “fail-safe N” statistics indicate that for most of the various types of programs servicing chronic patients, our confidence is low that the mean satisfaction level is statistically significantly different from the scale midpoint. For in-
NAb NA’J
NS
x= = 0.30 df = 2 p = NS x1 = 3.11 df =7 p = NS x1 = 2.02 df = 2 p = NS NA” NA”
of the mean z.
stance, only 7 “null” studies would be needed to eliminate the trend toward positive satisfaction with conventional, chronic inpatient programs. For most other types of programs for chronic patients, as well as for the various innovative programs for which only one study was available, the levels of confidence are also low. However, Table 4 does demonstrate that the levels of confidence in results for several types of programs for non-chronic patients are high, requiring large numbers of null studies to reverse the conclusions of the meta-analysis. An Example. The following examples illustrate the use of Table 4. Suppose an administrator in a local community mental health center (CMHC) measures an outpatient population of chronic patients in an ongoing traditional treatment program. Assume for illustration purposes his/her raw satisfaction score (3 = 59.7; sd = 0.8; scale midpoint = 50), using Equation 1, translates into a modified standard normal deviate z of 0.99. Using Table 4 s/he could then compare the study with the norms using traditional calculations
ANTHONY
270
F. LEHMAN
and THOMAS
R. ZASTOWNY
. ,Else”thal
.,Kdl”
. ,KOlf””
1969,
. (Kalman
.,Fles,er . (Mange”
1982,
.(Lo”e
. ,Gy”lher.
1963)
. ,Slater
-------------_--------------------------_-----_-------
i
1981,
. (k”0,5
. ,tdayer 1974,
1979,
1981,
. (JustIce
. lDlslela”a
OJ
r
I INPATIENT CHRONIC INNOVATIVE
INPATIENT CHRONIC CONVENTIONAL
. (Ahned I INPATIENT NON-CHRONIC CONVENTIONAL
1 INPATIENT NON-CHRONIC INNOVATIVE
1 OUTPATIENT CHRONIC CONVENTIONAL
of Satisfaction
TABLE
Confidence
I. Inpatient A. Chronic 1. Conventional 2. Innovative B. Non-Chronic 1. Conventional 2. Innovative II. Outpatient A. Chronic 1. Conventional 2. Innovative
V.
VI.
=NA
B. Non-Chronic 1. Conventional 2 Innovative Residential (Chronic) A. Conventional B. Innovative All Programs A. Chronic B. Non-Chronic All Programs A. Inpatient B. Outpatient All Programs A. Conventional B. Innovative =
Not applicable
because
1966)
1982,
I OUTPATIENT NON-CHRONIC CONVENTIONAL
Scores by Program
1 OUTPATIENT NON-CHRONIC INNOVATIVE
. ,Mlman . ,B”dscm
1982, 1978)
1 RESIDENTIAL
Type.
4
PRELIMINARY NORMS AND LEVELS OF CONFIDENCE PATIENT SATISFACTION WITH MENTAL HEALTH CARE PROGRAM
Type of Program
WEIGHTED FOR
1976,
Figure 1. Distribution
FOR
MEAN
SAMPLE SIZE
. iPark
1981, . (ARklSSO”
. (Souelen 1955) . (Ellsworth 1972)
GRAND
1978,
. (W&w 1981, . ,Edvm& 19781
. (Em”
,982, . (Grab 19781
IV.
1978,
1979, 19821
. (Gymher 1953, . ,Kalma” 1982, _~‘~p’el.~56)____________________________~LKolluv1978)____U_NW_EIGH_TE_D_GRAN_D_MAN . ,IrnE 1962)
2
III.
1983,
Intervals
TYPES
Stouffer z (One-Tailed Probability)
Fail-Safe
95%
0.88 + 0.81 5.23 f NAa
0.88 f 1.03 NAa
1.46(0.07) NAa
7.0 NAa
1.62 zt 0.84 2.02 + NAa
1.62 + 1.06 NAa
3.29( < .OOl) NAa
40.5 NAa
1.01 + 2.40 -
1.01 +z 4.84 -
2.09( < .02) -
2.20 +z 0.52 2.35 + 1.68
2.20 f 0.66 2.35 f 2.48
4.35( < .OOOl) 3.84( < .OOOl)
106.5 15.4
0.85 + NAa 0.81 * NAa
NAa NAa
NAa NAa
NAa NAa
1.41 f 0.51 1.90 f 0.43
1.41 +z 0.63 1.90 f 0.53
1.88( < .04) 5.99( < .OOOl)
77.9 462.6
1.61 zt 0.46 1.89 zt 0.53
1.61 zt 0.56 1.89 zt 0.65
2.63( < ,005) 5.45( < 0001)
200.5 210.1
1.71 f 0.37 3.12 zt 1.52
1.71 l 0.45 3.12 + 1.99
4.23( < .OOOl) 3.68( < .OOOl)
820.2 84.9
only one study
is available.
N
(p = .05)
90 %
0 -
A Meta-Analysis to Establish Norms around the confidence intervals (line 6, Table 4; x1 = 1.01 f 2.40 or x, = 1.01 f 4.84). In this case s/he would conclude that the study compared favorably with the existing literature, but was not significantly different from the expected value (i.e., xZ for chronic outpatient conventional programs). Another researcher who found a normal deviate z of 5.98 (x = 76.9; sd = 4.5; scale midpoint = 50) for the same comparison group would conclude s/he had achieved a significant departure from the existing norms even using the .95 confidence interval. The preceding examples present the simplest use of Table 4. More elaborate meta-analytic comparisons could be made by calculating new Stouffer’s z’s for overall confidence in observed effects, recalculating the x2 statistic of Rosen-
271
thal and Rubin for heterogeneity, or determining the “fail-safe N” estimates. Each of these comparisons would require the pooling of data from the existing studies with the data from the new study under comparison (See Rosenthal [1983] and Cooper [1979] for information on these procedures). When conducting such comparisons researchers need to keep in mind a number of potentially biasing effects in meta-analysis. Wilson and Rachman (1983) and Fisk (1983) suggest areas for important consideration that will affect meta-analysis results including: (a) selection of the data base for meta-analysis, (b) qualitative vs. quantitative weighting systems, and (c) conceptual classification schemes.
DISCUSSION The literature on patient satisfaction with mental health services remains limited in many ways. Nonetheless, this paper demonstrates that the accumulated data permit a preliminary meta-analysis to establish norms of patient satisfaction with different types of mental health programs. The many shortcomings of this literature will not be discussed at length here because they have been well reviewed elsewhere (Kalman, 1983; Lebow, 1982; Weinstein, 1981). These limitations include the lack of controlled studies that compare different groups of patients or programs, the lack of consistency in the instruments used to measure satisfaction, and the absence of a clear theoretical framework of patient satisfaction. In order to cope with these problems in this review we modified the usual meta-analytic approaches to accommodate the lack of data on effect sizes and percent differences between treatment groups often available in most studies used in a standard meta-analysis. We also limited the analyses to global measures of patient satisfaction because the literature is simply inadequate to permit a more refined analysis according to specific dimensions of patient satisfaction. Despite these methodologic limitations, the analyses undertaken here at the very least help to ascertain the degree to which norms can be established and demonstrate a method for accumulating data from future studies. As has been well documented elsewhere (Lebow, 1982; Weinstein, 1981), the levels of satisfaction found in most all studies are heavily skewed toward the positive end of satisfaction scales. However, we found that the use of a quantitative and continuous variable, defined here as a standard normal deviate z-score, revealed greater variability in results than is readily evident from the dichotomization of findings into rates of satisfaction and dissatisfaction. Still, the findings of such overall positive levels of patient satisfaction further underscore the need for program-specific norms against which to compare future program eval-
uations. This also raises questions about the psychometric properties of the various scales used to measure patient satisfaction. It is apparent that work is needed in the development of scales that might provide more normal distributions of responses and a larger variability of responses, although it is certainly possible that “satisfaction” may not conform to traditional psychometric expectations. Despite the skewness of satisfaction scores we did find significant heterogeneity in the levels of patient satisfaction reported among the studies reviewed. More specifically, we found that chronic patients expressed less satisfaction with their treatment programs than did non-chronic patients and that greater patient satisfaction was expressed with innovative treatment programs as compared to more conventional programs. The differences indicate the importance of developing program type-specific norms rather than overall norms. Finally, we were able to develop preliminary mean satisfaction levels for certain types of programs and to place confidence intervals around these means. These statistics, displayed in Table 4, can be used by program evaluators and other investigators for comparative purposes with their own data. All that is required for comparison is knowledge of the observed mean level of satisfaction, the observed standard deviation, and the absolute midpoint of the specific scale used to measure satisfaction. Unfortunately, we were only able to develop such norms for conventional inpatient programs for chronic and non-chronic patients, and conventional outpatient programs for non-chronic patients. For all other types of programs insufficient data are available to establish meaningful norms. By pooling across the various program dimensions employed in this study, that is, chronic vs. non-chronic, inpatient vs. outpatient, and conventional vs. innovative, we were also able to develop means and confidence intervals for these types of programs. The levels of confi-
272
ANTHONY
F. LEHMAN
and THOMAS
dence in these estimates are reasonably good considering the small numbers of studies available, and provide some point of reference for investigators and program evaluators who lack any other comparative data. Recommendations that would allow firmer conclusions to be reached and future meta-analyses to be conducted include: 1. Researchers should include as full and complete a set of results as possible. Statistics should include, at a minimum, means, standard deviations, and sample size. Psychometric properties of the scale should also be reported. If available, standard error of the mean should be recorded. 2. Power of the full experiment should be stated, which would allow Type I/Type I1 errors to be estimated. 3. More truly experimental studies are required that examine explicit hypotheses or compare rates of satisfaction between and among clusters of studies. 4. Continued meta-analyses need to be conducted that would update “norms” for satisfaction and help
R. ZASTOWNY
establish expected “cutoffs” ing to program types.
for satisfaction
accord-
In summary, we were able to develop acceptable estimates of the levels of patient satisfaction with various types of mental health programs. These estimates will be useful for comparative purposes as long as the many limitations of the data are kept in mind. Most importantly, we have demonstrated the feasibility of developing such norms and propose that a cumulative data base be established and updated on a regular basis. Systematic refinement and modification of the analytic approach presented here will be needed. Metaanalysis is a relatively new concept and its methods have not yet been perfected (Fisk, 1983). Furthermore, its application in this study was of necessity novel because of the state of the literature on patient satisfaction. As always, although such techniques are more objective and impartial than some of earlier less empirically-based reviews (Cotton & Cook, 1982), the ultimate value of the findings must be determined by the larger scientific community.
REFERENCES COOPER, H. M. Statistically meta-analysis of sex differences
combining independent in conformity research. Personaltty and Social Psychology, 1919, 37, 131-146.
studies: Journu/
A oj
RICE, C. E., KLETT, S. L., BERGER, LEMKAU, P. V. The ward evaluatton
D. G., SEWALL, L. G., & scale. Journal of Clinical
Psychology, 1963, 19, 251-258. ROSENTHAL, R. Assessing the effects of psychotherapy.
the statistical
and social importance
of
COTTON, J. L., & COOK, M. S. Meta-analyses and the effects of various reward systems: Some different conclusions from Johnson et al. Psychological Bulletin, 1982, 92, 176183.
Psychology, 1983, SI, 4-13.
DEVINE, E. C., & COOK, T. D. Effects of psychoeducational interventtons on length of post-surgical stay: A meta-analytical review of 49 studies. Nursing Research, in press.
ROSENTHAL, R. (Ed.) New directions for methodolog_v of socral and behavioral science: Quantitative assessment of research domains. San Francisco: Jossey-Bass, 1980.
FISK.
SMITH, outcome
D. W’. The meta-analytic
revolution
in outcome
research.
Journal of Consulting and Clinical Psychology, 1983, 51, 65-70. KALMAN, psychtatric
T. P. treatment.
An
overview
of
patient
satisfaction
with
Hospital and Communrty Psychiutry, 1983,
34, 48-54. LEBOW,
J. Consumer
satisfaction
with mental
health
treatment.
Psychological Bulletin, 1982, 9/, 244-259.
Journal of Consultrng and Cltnical
M. L., & GLASS, G. V. Meta-analysis of psychotherapy studies. American Psvchologist, 1977, 32, 752-760.
WEINSTEIN, R. M. Attitudes toward psychtatrtc hospitalized patients: A review of quantitative Science and Medicme, 1981, 158. 301-314. WILSON, evaluation
treatment research.
among
Socral
G. T., & RACHMAN, S. .I. Meta-analysis and the of psychology outcome: Limitattons and liabtlnies. Jour-
nal of Consulting and Clinical Psychology, 1983, 51, 54-69.
APPENDIX
Studies Included in Computerized Library Search of Literature on Patient Satisfaction and Psychiatric Care Since 1981
AHMED, hl. B., & KOLTUV, M. Evaluation of an inpatient service by consumer feedback. Australian and New Zeukund Journal of
P.sychratry, 1976, 10, 263-268.
ATTKISSON, C. C., & ZWICK, R. The Client Satisfaction Questionnaire: Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluatron and Program
Planning, 1982, 5, 233-237. ALLEN, J. C., & BARTON, G. M. Patient comments about hospitalization: Implications for change. Comprehensive Psychi-
BALCH,
atrv, 1976, 17, 631-640.
S. B. Client evaluation
P., IRELAND,
J. F., McWILLIAMS, S. A., & LEWIS, of a community mental health service: Rela-
A Meta-Analysis
to Establish
Norms
273
tion to demographic
and treatment variables. Amerfean Journaf of Community Psychology, 1977. 5, 243-247.
a& patient: A voice to be heard. British Journaf of Psychiatry, 1979, 235, 115-121.
BENE-KOCIEMBA, A., COTTON, P. G., & FORTGANG, R. C. Assessing patient satisfaction with state hospital and aftercare services. American Journal of Psychiatry, 1982, 139, 660662.
WYNE, J. B., & LADGUX, P. Patients’ opinions of outpatient clinic services. Hospitaf and Community Psychiatry, 1973, 24, 627-628.
BUDSON, R. D. The psychiatric haffway house. Pittsburgh: University of Pittsburgh Press, 1978.
GROB, M. C., EISEN, S. V., & BERMAN, J. S. Three years of follow-up monitoring: Perspectives of formerly hospitalized patients and their families. Comprehensive Psychiatry, 1978, 19, 491-499.
CHIEN, C. P., & COLE, J. 0. Landlord-supervised cooperative apartments: A new modality for community-based treatment. American Journal of Psychiatry, 1973, 130, 156-159. COHEN, C. I., SICHEL, W. R., & BERGER, D. The use of amidManhattan hotel as a support system. Community Mentaf Heafth Journal, 1977, 13, 76-83.
GYNTHER, M. D., REZNIKOFF, M., & FISHMAN, M. Attitudes of psychiatric patients toward treatment, psychiatrists and mental hospitals. Journai of Nervous and Mental Disease, 1963, (36, 68-71.
DENNER, B., & HALPRIN, F. Clients and therapists evaluate clmical services. American Journal of Community Psychofogy, 1974, 2, 373-386.
HARGREAVES, W. A., SHOWSTACK, J., FLOHR, R., BRADY, C., & HARRIS, S. Treatment acceptance following intake assignment to individual therapy, group therapy, or contact group. Archives of Generaf Psychfatty, 1974, 31, 343-349.
DENNER, B., & HALPRIN, F. Measuring consumer satisfaction in a community outpost. American Journal of Community Psychofogy, 1974, 2, 213-222.
HART, W. T., 8c BASSETT, L. Measuring consumer satisfaction in a mental health center. Hospital and Community Psychiatry, 1975, 26, 512-515.
DISTEFANO, M. K., PRYER, M. W., & GARRISON, J. L. Clients’ satisfaction and inter-personal trust among hospitalized psychiatric patients. Psychologicaf Reports, 1981, 49, 420-422.
IMRE, P., &WOLF, S. Attitudes of patients and personnel toward mental patients. Journal of Clmical Psychoiogy, 1%2,1& 232-234.
EDER, D., & KUKULSKI, S. A scale for measuring patient satisfaction with hospital services. Hospital and Community Psychratry, 1975, 26, 15-16. EDWARDS, D. W., YARVIS, R. M., MUELLER, D. P., & LANGSLEY, D. G. Does patient satisfaction correlate with success? Hosprtal and Community Psychiatry, 1978, 29, 188-190. EISEN, S. V., & GROB, M. C. Assessing consumer satisfaction from letters to the hospital. Hospital and Community Psychiatry, 1979,30, 344-347. EISEN, S. V., & GROB, M. C. Measuring discharged patients’ satisfaction with care at a private psychiatric hospital. Hospitaf and Community Psychiatry, 1982, 33, 227-228. EISENTHAL, S., KOOPMAN, C., & LAZARE, A. Process analysis of two dimensions of the negotiated approach in relation to satisfaction in the initial interview. Journaf of Nervous and Mentaf Dtsease, 1983, 171, 49-54. ELLSWORTH, R., & MARONEY, R. Characteristics of psychiatric programs and their effects on patients’ adjustment. Journaf of Consuiting and Clinical Psychology, 1972, 39, 436-447. FIESTER, A. R. Goal attainment and satisfaction scores for CMHC clients. American Journal of Communfty Psychofogv, 1979, 7, 181-188. FRYLING, V. B., & FRYLING, A. G. Patients’ attitudes toward sociotherapy. Psychiatric Quarterfy (Supplement), 196434, 97-115. GLENN, R. N. Measuring patients’ opinions about hospitali~tion using the Client Satisfaction Scale. Hospitaf and Community Psychiatry, 1978, 29, 627-628. GORDON, D., ALEXANDER, D. A., & DIETZAN, J. The psychi-
JANSEN, D. G., & ALDRICH, M. W. State hospital psychiatric patients evaluate their treatment teams. Hospital and Community Psychiatry, 1973, 24. 768-770. JUSTICE, B., & McBEE, G. A client satisfaction survey as one element in evaluation. Community Mental Health Journal, 1978, 14, 248-252. KAHN, M. W., & JONES, N. F. A comparison of attitudes of mental patients from various mental hospital settings. Journal of Cfinical Psychology. 1972,25, 312-316. KALMAN, T. P., TALON, N. S., FRANCES, A., & KOCSIS, J. H. A controlled study of satisfaction among psychobiology research patients. American Journal of Psychiatry, 1982, 139, 344-347. KLOPFER, W. G., WYLIE, A. A., & HILLSON, J. S. Attitudes toward mental hospitals. Journal of Clinical Psychoiogy, 1956, 12, 361-365. KOLTUV, M., AHMED, M. B., & MEYER, M. A study of outpatient satisfaction with mental health center services. Hospttal and Community Psychiatry, 1978, 29, 179-181, KOTIN, J., & SCHUR, J. M. Attitudes of discharged mental patients toward their hospital experiences. Journal of Nervous and Mentaf Dketze, 1969, 149, 408-414. LAMB, H. R. The new asylums in the community. Archives of General Psychiatry, 1979, 36. 129- 134. LEHMAN, A. F., WARD, N. C., & LINN, L. S. Chronic mental patients: The quahty of life issue. American Journal o~Psychfafry, 1982, 139, 1271-1276. LEVOIS, M., NGUYEN, T. D., & ATTKISSON, C. C. Artifact in client satisfaction assessment: Experience in community mental
274 health
rettings.
Evrrlwfion
and
ANTHONY
F. LEHMAN
Program
Plannmng, 19811 4,
139-150.
and
THOMAS
R.
ZASTOWNY
RICE, C. E., KLETT, S. L., BERGER, LEMKAU, P. V. The ward evaluatton
D. G., SEWALL., L. G., & scale. Journul of Clmrcal
Ps.veho~og_v,1963, f9. 251-258. LINN,
L. S. The
mental hospital Psychiairy, 1968, 31, 213-223.
from
the pattent
perspectwe.
LOVE, R. E., CAID, C. D., & DAVIS, A. The uber satisfactlon survey: Consumer evaluation of an inner city community mental health center. ~vui~~t~on und the Healrh Professions, 1979, i,
42-S. MANGEN, S P., & GRIFFITH, J. H. Patient satisfaction wth commumty psychiatric nursing: A prospective controlled study. Journul of Advanced Nursing, 1982, 7, 477-482.
ROBACK, H., & SNYDER, W. U. A comparison of hocpitahzed mental patients’ adjustment wth their attitudes toward psychiatric hospitals. Journul of Clinrcul Psychology, 1965, 21, 228-230. S. P., & AVIRAM, U. The rnen~ul~y iI/ in ~o~~unit~lbused sheltered care. Ne% York: John Wiley and Son\, 1978.
SEGAL,
SLATER, V.. LINN, M. W., &HARRIS, R. Outpatient of mental health care. Sourhern Me&al Journal,
evaluation
1981, 74,
1217-1219. MAYER, J. E., & ROSENBLATT, A. Clash rn perspective between mental patients and staff. Amerrcan Journal of Orthopsychratry,
1974. 44. 432-441.
SNELLING, E., & WALKER, F. Patient batlsfactlon ulrh facilities at an acute admission p\ychlatrlc unit. New Zealand Medicul Jour-
nal, 1978, 87, 94-97. PARK, L. C., SLAUGHTER, R., COVI, L., & KNIFFIN, H. G. The subjective experience of the research patient: An investlgatlon of psychiatric outpatIents to the research treatment snuation. Journal of Nervous und Men&f Lkease, 1966, 143, 199-206.
SOUELEM,
0. bfental
patients’
attitudes
toward
mental
hospitals.
Journal of Clrntcal Psychology, 1955, II, 181-185.
181-190.
TESSLER, R. C. Clients’ reactions to imtlal mterwe&s: Determinants of relatlonshlp-centered and problem-centered satisfaction. Journal of Counseling Psychology, 1975, 22, 187- 191.
REINHART, R., & CAZAVELAN, J. Mental health consumer survey on a shoestring budget -it 15 possible? .4merrcan Journal of
WEBER. tal heaith
Communrt_v Psychology, 1975, 3, 273-216.
29, 193-194.
POWELL, B. J., SHAW, D., & O’NEAL, C. Chent evaluation of a clintc’, serwes. Hosp~tui und Communrty Psychiatry, 1971, 22,
D. J., & TILLEY, D. H. Patient~‘evaluations of the menservice at a health sciences campus. College Health, 1981,
Evaluation
PATIENT
6,
pp.
265-214,
SATISFACTION
0149-7189/83 $3.00 + .OO CopyrIght 0 1984 Pergamon Press Ltd
1983
WITH MENTAL
A Meta-Analysis
ANTHONY F. LEHMAN
HEALTH
SERVICES:
to Establish Norms
and THOMAS R. ZASTOWNY
University of Rochester School of Medicine
ABSTRACT Patients typically express high rates of satisfaction with their mental health care. This finding and the lack of well controlled studies on patient satisfaction in the literature underscore the need for meaningful guidelines for clinicians and program evaluators in interpreting patient satisfaction data. To address this problem a meta-analysis was undertaken to establish norms on patient sattifaction for various types of mental health programs. Programs were categorized according to three dimensions: inpatient vs. outpatient vs. residential care; chronic vs. nonchronic; and conventional vs. innovative. Meta-analysis procedures were modified to accommodate the single-group study designs that dominate the literature. The analysis revealed that chronic patients express less satisfaction with their treatment compared to non-chronic patients. Innovative programs are viewed more positively than conventional ones. No differences were found in rates of patient satisfaction between inpatient and outpatient programs. Acceptably reliable norms and confidence intervals of patient satisfaction were established for conventional inpatient programs serving either chronic or non-chronic patients; conventional outpatient programs for non-chronic patients; and for all programs combined according to chronic vs. non-chronic, inpatient vs. outpatient, and conventional vs. innovative. However, data were insufficient to compute norms for other program types. The norms thus established can be used for comparative purposes by program evaluators. A cumulative, national data base on patient satisfaction is recommended to further refine these norms.
Mental health professionals have shown a sustained and growing interest in patient satisfaction with mental health services, despite persistent conceptual ambiguity and problems in the measurement of patient satisfaction. Several articles in this current volume as well as recent review articles provide extensive discussions of the measurement and meaning of patient satisfaction with mental health services (Kalman, 1983; Lebow, 1982; Weinstein, 1981). It is well established that the majority of patients express positive attitudes about their mental health care. Although this finding may be encouraging news for care providers, it underscores the need for meaningful guidelines to assist clinicians and program evaluators in interpreting the typically high levels of satisfaction expressed by their patients. In this paper we present a meta-analysis of the existing quantitative data in the literature on patient satisfaction with mental health services in order
to establish preliminary norms against which to compare specific programs. This analysis provides a further refinement of the literature reviewed elsewhere (Lebow, 1982; Weinstein, 1981), and identifies areas in which existing data are insufficient to establish norms. Unfortunately, well-controlled studies of patient satisfaction are rare in the literature, and experimental designs are usually not practical when program evaluations are requested by clinicians or administrators. Nonetheless there must be some reference points from which to interpret levels of patient satisfaction if such evaluations are to be of any value other than to cast a positive glow upon most programs. One option for developing such reference points is to pool data from several studies, none of which alone provides an adequate comparison, but which together can provide normative information for comparative purposes. Such comparisons complement well-designed studies,
Requests for reprints should be sent to Anthony F. Lehman, Department of Psychiatry, University of Rochester Medical Center, 300 Crittenden Blvd.,
Rochester,
NY 14642.
265
266
ANTHONY
F. LEHMAN
and are preferable to the usual situation in which a single program is evaluated cross-sectionally and the results interpreted in a relative vacuum. To approach this program, we developed a classification scheme for mental health programs along three program dimensions: (a) inpatient vs. outpatient vs. residential care; (b) chronic vs. non-chronic patients; and (c) conventional vs. innovative treatment. Although a variety of other program characteristics may be related to patient satisfaction, we chose these three dimensions because they are basic and because most existing programs can be classified readily into 1 of the 12 subtypes defined by this 3 x 2 x 2 typology. This classification system formed the basis to conduct a meta-analysis of patient satisfaction according to program type. Meta-analysis refers to various analytic procedures for reviewing the literature (Rosenthal, 1980) pioneered by Smith and Glass (1977). It is particularly useful for combining the results of independent studies to assess questions that no single study can address. Early metaanalyses were targeted on psychotherapy outcome studies, but in recent years the technique has been applied to a variety of topics including various reward systems (Cotton 8z Cook, 1982), sex differences (Cooper, 1979), and surgical recovery (Devine & Cook,
and THOMAS
R. ZASTOWNY
in press). The interested reader is referred to an excellent article by Rosenthal (1983) for an overview and some illustrations of meta-analysis procedures. The two most commonly employed statistics in metaanalysis are effect size (ES) and percent difference (PO>. Effect size is calculated as the difference between the means of two groups divided by the standard deviation of the control group. The PD is computed as the difference between the group means divided by the control group mean, multiplied by 100. Various procedures to weight studies for sample size and other relevant conditions are available and the interested reader is referred to the Journal of Consulting and Clinical Psychology (51 [I], 1983) for a series of reviews on meta-analysis. Applying meta-analytic techniques to the available literature, we sought answers to the following questions: 1. What are the typical levels of patient satisfaction with the various types of mental health programs? 2. Do the levels of patient satisfaction vary according to type of program (inpatient vs. outpatient vs. residential, conventional vs. innovative) or type of patient (chronic vs. non-chronic)? 3. Can norms of patient satisfaction be established for the various program types?
METHODS The studies analyzed were identified from three recent, comprehensive reviews of patient satisfaction with mental health services (Kalman, 1983; Lebow, 1982; Weinstein, 1981), supplemented by a computerized library search of the literature on patient satisfaction and psychiatric care since 1981. Of 98 empirical studies thus identified, 35 studies (37 programs) had sufficient quantitative data for inclusion in the analysis on rates of patient satisfaction; 27 studies (33 programs) had sufficient data for inclusion in a meta-analysis of mean satisfaction levels. A total of 52 separate studies (59 programs) were thus reviewed. These are included in the Appendix. Each study was abstracted for the following information: (a) year of study, (b) type(s) of programs, (c) sample size, (d) procedure for assessing satisfaction, (e) rate of satisfaction (r70), (f) rate of dissatisfaction (o/o), (g) satisfaction scale range and midpoint, (h) mean satisfaction score, and (i) standard deviation of satisfaction score. The studies were classified according to the matrix in Table 1. The differentiation of inpatient, outpatient, and residential care programs is self-explanatory. “Chronic” patients were identified as persons with a life history of psychiatric hospitalizations totaling at least 6 months. For studies that did not specify the amount of hospitalization, we accepted patients as chronic if they were described as “chronic”
or if the site of treatment was a state or Veterans Administration hospital. Finally, programs were classified as “innovative” only if they were specifically identified as a new program or were presented as improvements over the standard type of treatment available. All other programs were considered conventional. The rarity of studies that provide between-group comparisons of patient satisfaction presented an initial methodologic problem in this review because metaanalytic techniques rely on such comparative data. To deal with this, we employed a modified standard normal deviate, z-score, computed as follows: 7 = (Observed mean) - (Scalemidpoint) . Observed standard deviation
(1)
For those studies in which an observed mean was available, but a standard deviation was not, the standard deviation was estimated from the ratio of mean satisfaction/standard deviation in the studies that provided both statistics. This z-score reflects the deviation of the observed mean satisfaction from the numerical midpoint of the satisfaction scale, the hypothetical 50% satisfaction reference point. This reference value was used because it is the level that would be observed by chance alone and because it is the null hypothesis implicit in many
A Meta-Analysis to Establish Norms
267
dent’s t-test and for calculation of confidence intervals around the mean satisfaction levels in different types of programs. In all analyses, studies were weighted according to sample size (Cooper, 1979). As such, the procedures represent a modification of available metaanalytic techniques and provide a method for conducting meta-analysis on literature with predominantly single-group designs.
satisfaction studies. We then used these modified standard normal deviates from individual studies as the data base for computing meta-analytic statistics, including x2 tests for heterogeneity of study results and the Stouffer z-statistic and “fail-safe N” for assessing the level of confidence in the results (Cooper, 1979; Rosenthal, 1983). Finally, these z-scores were used for between-group comparisons by means of Stusingle group
RESULTS rates of satisfaction and dissatisfaction were 70.6% and 18.970, respectively. The lowest level of satisfaction was recorded in conventional, residential care programs for chronic patients (49.5%); the highest rates of dissatisfaction were recorded in conventional chronic inpatient programs (30.5%), innovative nonchronic outpatient programs (27.4%), and conventional residential care programs (22.3%).
Studies by Year and Program Type A total of 59 programs in 52 studies was analyzed. Table 1 summarizes the numbers and types of programs studied during each 5-year interval between 1955 and February 1983. The number of quantitative studies increased steadily during that time, particularly since 1970. The vast majority of programs were conventional (51/59 = 86~0), 47~0(28/59) were inpatient, and 58%(34/59) served patients with non-chronic disorders. The types of programs studied shifted from predominantly chronic inpatient programs prior to 1970 to predominantly non-chronic outpatient and inpatient programs after 19’70. Since 1975 there has been increasing interest in long-term residential care programs.
Meta-Analysis of Standardized Satisfaction Scores The high degree of skewness in patient satisfaction among the studies can be appreciated by comparison to a normal standard z-distribution. In a normal standard ~-distribution, 50% of scores are less than 0.0. In contrast, no study found a standardized satisfaction score less than 0.0, i.e., none found a mean satisfaction level below the numerical midpoint of the scale used for measuring satisfaction. The median standardized satisfaction score was 1.70, and the mean satisfaction score weighted for sample size was 1.47. If the
Rates of Patient Satisfaction and Dissatisfaction by Program Type The rates of satisfaction were high, exceeding 50% for all but one program type (see Table 2). The overall
TABLET
MENTAL HEALTH PATIENT SATISFACTION STUDIES REVIEWED BETWEEN 1955 AND 1983 BY YEAR AND PROGRAM TYPE
Years of Study Type of Program I. Inpatient A. Chronic 1. Conventional 2. innovative B. Non-Chronic 1. Conventional 2. Innovative Il. Outpatient A. Chronic 1. Conventional 2. Innovative B. Non-Chronic 1. Conventional 2. Innovative Ill. Residentiala A. Chronic 1. Conventional 2. Innovative Totals
1955-59
1960-64
1965-69
1970-74
1975-79
1980-83
Total Years
2 0
2 1
3 0
4 0
3 0
0 0
14 1
0 0
3 0
0 0
1 0
6 0
2 1
12 1
0 0
0 0
0 0
0 0
0 0
4 0
4 0
0 0
0 0
0 1
5 0
9 1
4 1
18 3
0 0 2
0 0 6
0 0 4
0 1 11
2 2 23
1 0 13
3 3 59b
a No non-chronic residential programs were reported. bPrograms were evaluated in 52 separate studies.
268
ANTHONY F. LEHMAN and THOMAS R. ZASTOWNY
RATES
OF PATIENT
Program
SATISFACTION
Type
Inpatient Non-Chronic, conventional Non-Chronic, innovative Long-Term, conventional Long-Term, innovative Outpatient Non-Chronic, conventional Non-Chronic, innovative Long-Term, conventional Long-Term, innovative Long-Term residential Conventional Innovative Overall
TABLE 2 AND DISSATISFACTION
Percent Satisfaction
78.2 85.0 69.4 -
(WV
(6;545) (1;33) (8;1,136)
BY PROGRAM Percent Dissatisfaction
TYPE
(KN)a
15.5 -
(3;380) -
30.5 -
(4;601) -
13.3 27.4 -
(11;1,500) (2; 124) -
78.0 64.0 73.3 -
(x72) (2;62) -
49.5 77.1
(3,129)
22.3 8.7
(3;878) (3;129)
70.6
(37;4,819)
18.9
(26;3,612)
(13;1,964)
(3;878)
Note. Rates reflect weighted adjustments for sample aK = number of programs, N = number of patients
distribution of scores was normal, the median and mean scores would be 0.0. In fact, the observed median and mean satisfaction scores were between one and two standard deviations above this hypothetical mean. The first issue addressed in the meta-analysis was whether or not there was heterogeneity among the satisfaction scores; i.e., was there evidence for significant variability among them? Using the x2 statistic of Rosenthal and Rubin (Rosenthal, 1983), we found significant heterogeneity (x2 = 49.6, df = 30, p < .025). Given this finding, the next question addressed was whether this heterogeneity could be accounted for by variability among studies within program types or by variability across program types. The latter issue would be of particular relevance in the development of program-specific norms. Table 3 displays the weighted mean satisfaction scores, standard deviations, numbers of studies, ranges of scores, and tests for heterogeneity for the various program types. The distribution of these studies by program type is shown in Figure 1. Significant heterogeneity was found only among the studies of conventional, chronic inpatient programs, and could be accounted for entirely by one study with a standardized mean satisfaction of 4.93 (Rice, Klett, Berger, Sewall, & Lemkau, 1963). The next highest score among that group of studies was 2.59. Removal of this outlying study produced a weighted mean satisfaction score of 0.88 for these chronic inpatient programs. To avoid bias, this outlying study was excluded in all subsequent analyses. However, even after removal of this study the trend remained for significant heterogeneity among all the studies (x’ = 40.74, df = 29, 0.10 > p > 0.05).
sizes of individual
studies.
Next we considered whether or not the levels of patient satisfaction varied across program types. Based upon the data presented in Table 3, we found that chronic patients tended to express less satisfaction with their treatment than did non-chronic patients for all program types combined [z(chronic) = 1.41, ,-(nonchronic) = 1.90; t = 1.29, p = .10 (one-tailed)]. This trend held for both inpatient and outpatient services, although it was statistically significant only for outpatient programs [:(chronic, inpatient) = 0.88, inpatient) = 1.62, t = 1.23, n.s.; z(non-chronic, z(chronic, outpatient) = 1.Ol, z(non-chronic, outpatient) = 2.20, t = 2.46, p = .025 (one-tailed)]. No significant difference in patient satisfaction was found between inpatient and outpatient programs (;(inpatient) = 1.61, z(outpatient) = 1.89; t = 0.71, n.s.1. Finally, innovative programs as a group were associated with greater patient satisfaction than conventional programs [,-(innovative) = 3.12, z(conventiona1) = 1.71; t = 2.33, p < .025]. Preliminary Norms and Confidence Intervals for Satisfaction with Program Types Using the data from Table 3 it was possible to compute confidence intervals for the mean satisfaction levels for various program types. These 90% and 95% confidence intervals are given in Table 4 and provide norms against which additional program evaluations can be compared. It should be noted that for some categories of programs the small number of studies yields confidence intervals that are too broad to permit meaningful comparison. Based upon the confidence intervals given in Table 4, we can conclude that inadequate data exist to establish preliminary norms for conventional outpatient programs for chronic patients
A Meta-Analysis
DESCRIPTIVE
STATISTICS
ON MEAN
to Establish Norms
TABLE 3 SATISFACTION
LEVELS
269
FOR VARIOUS
PROGRAM
Type
Mean za (Standard Deviation)
N of Subjects; NC of Studies
Range of Unweighted z-Scores
I. Inpatient A. Chronic 1. Conventional
1.13(1.51)
2138;8
0.15-4.93
0.88(1.11)
1996;7
0.15-2.59
2. Innovative B. Non-Chronic 1. Conventional
5.23(NAb)
76;l
NA”
1.62(1.14)
968;7
0.0-3.25
2. Innovative II. Outpatient A. Chronic 1. Conventional
2.20(NAb)
33;l
NAb
l.Ol(O.54)
411;3
Program
[Outlying
study
removed]
2. Innovative B. Non-Chronic 1. Conventional
2. Innovative
III.
Residential (Chronic) A. Conventional B. Innovative
TYPES
Heterogeneity Test
x’ = 16.03 df = 7 p = .025 x1 = 7.36 df = 6 p =NS NAb x1 = 7.85 df = 6 iib
-
0.80-l
-
57
-
2.20(0.72)
929;8
0.84-2.95
2.35( 1 .OO)
156;3
0.91-4.07
0.85(NA”) 0.81 (NAb)
278; 1 54;l
az-Scores from individual studies are weighted for sample size in calculations bNA = not applicable because only one study is available. CN includes only those studies that reported mean satisfaction levels.
and for the four program types in which only one study was available for analysis. For other classes of programs, the confidence intervals are sufficiently narrow to permit reasonable comparisons with new studies. Also given in Table 4 are z-scores, computed by the Stouffer method (Rosenthal, 1983), which indicate the probability that the results of multiple studies combined are due to chance, i.e., the likelihood of a Type I error in the meta-analysis. The “fail safe” numbers of studies are also given in Table 4. These numbers indicate the smallest number of additional or unreported studies showing no difference between the observed mean satisfaction and the scale midpoint that would be needed to reverse the conclusion of a significant difference between the observed mean and midpoint. The larger this number, the greater the confidence that a true difference exists. The Stouffer z and “fail-safe N” statistics indicate that for most of the various types of programs servicing chronic patients, our confidence is low that the mean satisfaction level is statistically significantly different from the scale midpoint. For in-
NAb NA’J
NS
x= = 0.30 df = 2 p = NS x1 = 3.11 df =7 p = NS x1 = 2.02 df = 2 p = NS NA” NA”
of the mean z.
stance, only 7 “null” studies would be needed to eliminate the trend toward positive satisfaction with conventional, chronic inpatient programs. For most other types of programs for chronic patients, as well as for the various innovative programs for which only one study was available, the levels of confidence are also low. However, Table 4 does demonstrate that the levels of confidence in results for several types of programs for non-chronic patients are high, requiring large numbers of null studies to reverse the conclusions of the meta-analysis. An Example. The following examples illustrate the use of Table 4. Suppose an administrator in a local community mental health center (CMHC) measures an outpatient population of chronic patients in an ongoing traditional treatment program. Assume for illustration purposes his/her raw satisfaction score (3 = 59.7; sd = 0.8; scale midpoint = 50), using Equation 1, translates into a modified standard normal deviate z of 0.99. Using Table 4 s/he could then compare the study with the norms using traditional calculations
ANTHONY
270
F. LEHMAN
and THOMAS
R. ZASTOWNY
. ,Else”thal
.,Kdl”
. ,KOlf””
1969,
. (Kalman
.,Fles,er . (Mange”
1982,
.(Lo”e
. ,Gy”lher.
1963)
. ,Slater
-------------_--------------------------_-----_-------
i
1981,
. (k”0,5
. ,tdayer 1974,
1979,
1981,
. (JustIce
. lDlslela”a
OJ
r
I INPATIENT CHRONIC INNOVATIVE
INPATIENT CHRONIC CONVENTIONAL
. (Ahned I INPATIENT NON-CHRONIC CONVENTIONAL
1 INPATIENT NON-CHRONIC INNOVATIVE
1 OUTPATIENT CHRONIC CONVENTIONAL
of Satisfaction
TABLE
Confidence
I. Inpatient A. Chronic 1. Conventional 2. Innovative B. Non-Chronic 1. Conventional 2. Innovative II. Outpatient A. Chronic 1. Conventional 2. Innovative
V.
VI.
=NA
B. Non-Chronic 1. Conventional 2 Innovative Residential (Chronic) A. Conventional B. Innovative All Programs A. Chronic B. Non-Chronic All Programs A. Inpatient B. Outpatient All Programs A. Conventional B. Innovative =
Not applicable
because
1966)
1982,
I OUTPATIENT NON-CHRONIC CONVENTIONAL
Scores by Program
1 OUTPATIENT NON-CHRONIC INNOVATIVE
. ,Mlman . ,B”dscm
1982, 1978)
1 RESIDENTIAL
Type.
4
PRELIMINARY NORMS AND LEVELS OF CONFIDENCE PATIENT SATISFACTION WITH MENTAL HEALTH CARE PROGRAM
Type of Program
WEIGHTED FOR
1976,
Figure 1. Distribution
FOR
MEAN
SAMPLE SIZE
. iPark
1981, . (ARklSSO”
. (Souelen 1955) . (Ellsworth 1972)
GRAND
1978,
. (W&w 1981, . ,Edvm& 19781
. (Em”
,982, . (Grab 19781
IV.
1978,
1979, 19821
. (Gymher 1953, . ,Kalma” 1982, _~‘~p’el.~56)____________________________~LKolluv1978)____U_NW_EIGH_TE_D_GRAN_D_MAN . ,IrnE 1962)
2
III.
1983,
Intervals
TYPES
Stouffer z (One-Tailed Probability)
Fail-Safe
95%
0.88 + 0.81 5.23 f NAa
0.88 f 1.03 NAa
1.46(0.07) NAa
7.0 NAa
1.62 zt 0.84 2.02 + NAa
1.62 + 1.06 NAa
3.29( < .OOl) NAa
40.5 NAa
1.01 + 2.40 -
1.01 +z 4.84 -
2.09( < .02) -
2.20 +z 0.52 2.35 + 1.68
2.20 f 0.66 2.35 f 2.48
4.35( < .OOOl) 3.84( < .OOOl)
106.5 15.4
0.85 + NAa 0.81 * NAa
NAa NAa
NAa NAa
NAa NAa
1.41 f 0.51 1.90 f 0.43
1.41 +z 0.63 1.90 f 0.53
1.88( < .04) 5.99( < .OOOl)
77.9 462.6
1.61 zt 0.46 1.89 zt 0.53
1.61 zt 0.56 1.89 zt 0.65
2.63( < ,005) 5.45( < 0001)
200.5 210.1
1.71 f 0.37 3.12 zt 1.52
1.71 l 0.45 3.12 + 1.99
4.23( < .OOOl) 3.68( < .OOOl)
820.2 84.9
only one study
is available.
N
(p = .05)
90 %
0 -
A Meta-Analysis to Establish Norms around the confidence intervals (line 6, Table 4; x1 = 1.01 f 2.40 or x, = 1.01 f 4.84). In this case s/he would conclude that the study compared favorably with the existing literature, but was not significantly different from the expected value (i.e., xZ for chronic outpatient conventional programs). Another researcher who found a normal deviate z of 5.98 (x = 76.9; sd = 4.5; scale midpoint = 50) for the same comparison group would conclude s/he had achieved a significant departure from the existing norms even using the .95 confidence interval. The preceding examples present the simplest use of Table 4. More elaborate meta-analytic comparisons could be made by calculating new Stouffer’s z’s for overall confidence in observed effects, recalculating the x2 statistic of Rosen-
271
thal and Rubin for heterogeneity, or determining the “fail-safe N” estimates. Each of these comparisons would require the pooling of data from the existing studies with the data from the new study under comparison (See Rosenthal [1983] and Cooper [1979] for information on these procedures). When conducting such comparisons researchers need to keep in mind a number of potentially biasing effects in meta-analysis. Wilson and Rachman (1983) and Fisk (1983) suggest areas for important consideration that will affect meta-analysis results including: (a) selection of the data base for meta-analysis, (b) qualitative vs. quantitative weighting systems, and (c) conceptual classification schemes.
DISCUSSION The literature on patient satisfaction with mental health services remains limited in many ways. Nonetheless, this paper demonstrates that the accumulated data permit a preliminary meta-analysis to establish norms of patient satisfaction with different types of mental health programs. The many shortcomings of this literature will not be discussed at length here because they have been well reviewed elsewhere (Kalman, 1983; Lebow, 1982; Weinstein, 1981). These limitations include the lack of controlled studies that compare different groups of patients or programs, the lack of consistency in the instruments used to measure satisfaction, and the absence of a clear theoretical framework of patient satisfaction. In order to cope with these problems in this review we modified the usual meta-analytic approaches to accommodate the lack of data on effect sizes and percent differences between treatment groups often available in most studies used in a standard meta-analysis. We also limited the analyses to global measures of patient satisfaction because the literature is simply inadequate to permit a more refined analysis according to specific dimensions of patient satisfaction. Despite these methodologic limitations, the analyses undertaken here at the very least help to ascertain the degree to which norms can be established and demonstrate a method for accumulating data from future studies. As has been well documented elsewhere (Lebow, 1982; Weinstein, 1981), the levels of satisfaction found in most all studies are heavily skewed toward the positive end of satisfaction scales. However, we found that the use of a quantitative and continuous variable, defined here as a standard normal deviate z-score, revealed greater variability in results than is readily evident from the dichotomization of findings into rates of satisfaction and dissatisfaction. Still, the findings of such overall positive levels of patient satisfaction further underscore the need for program-specific norms against which to compare future program eval-
uations. This also raises questions about the psychometric properties of the various scales used to measure patient satisfaction. It is apparent that work is needed in the development of scales that might provide more normal distributions of responses and a larger variability of responses, although it is certainly possible that “satisfaction” may not conform to traditional psychometric expectations. Despite the skewness of satisfaction scores we did find significant heterogeneity in the levels of patient satisfaction reported among the studies reviewed. More specifically, we found that chronic patients expressed less satisfaction with their treatment programs than did non-chronic patients and that greater patient satisfaction was expressed with innovative treatment programs as compared to more conventional programs. The differences indicate the importance of developing program type-specific norms rather than overall norms. Finally, we were able to develop preliminary mean satisfaction levels for certain types of programs and to place confidence intervals around these means. These statistics, displayed in Table 4, can be used by program evaluators and other investigators for comparative purposes with their own data. All that is required for comparison is knowledge of the observed mean level of satisfaction, the observed standard deviation, and the absolute midpoint of the specific scale used to measure satisfaction. Unfortunately, we were only able to develop such norms for conventional inpatient programs for chronic and non-chronic patients, and conventional outpatient programs for non-chronic patients. For all other types of programs insufficient data are available to establish meaningful norms. By pooling across the various program dimensions employed in this study, that is, chronic vs. non-chronic, inpatient vs. outpatient, and conventional vs. innovative, we were also able to develop means and confidence intervals for these types of programs. The levels of confi-
272
ANTHONY
F. LEHMAN
and THOMAS
dence in these estimates are reasonably good considering the small numbers of studies available, and provide some point of reference for investigators and program evaluators who lack any other comparative data. Recommendations that would allow firmer conclusions to be reached and future meta-analyses to be conducted include: 1. Researchers should include as full and complete a set of results as possible. Statistics should include, at a minimum, means, standard deviations, and sample size. Psychometric properties of the scale should also be reported. If available, standard error of the mean should be recorded. 2. Power of the full experiment should be stated, which would allow Type I/Type I1 errors to be estimated. 3. More truly experimental studies are required that examine explicit hypotheses or compare rates of satisfaction between and among clusters of studies. 4. Continued meta-analyses need to be conducted that would update “norms” for satisfaction and help
R. ZASTOWNY
establish expected “cutoffs” ing to program types.
for satisfaction
accord-
In summary, we were able to develop acceptable estimates of the levels of patient satisfaction with various types of mental health programs. These estimates will be useful for comparative purposes as long as the many limitations of the data are kept in mind. Most importantly, we have demonstrated the feasibility of developing such norms and propose that a cumulative data base be established and updated on a regular basis. Systematic refinement and modification of the analytic approach presented here will be needed. Metaanalysis is a relatively new concept and its methods have not yet been perfected (Fisk, 1983). Furthermore, its application in this study was of necessity novel because of the state of the literature on patient satisfaction. As always, although such techniques are more objective and impartial than some of earlier less empirically-based reviews (Cotton & Cook, 1982), the ultimate value of the findings must be determined by the larger scientific community.
REFERENCES COOPER, H. M. Statistically meta-analysis of sex differences
combining independent in conformity research. Personaltty and Social Psychology, 1919, 37, 131-146.
studies: Journu/
A oj
RICE, C. E., KLETT, S. L., BERGER, LEMKAU, P. V. The ward evaluatton
D. G., SEWALL, L. G., & scale. Journal of Clinical
Psychology, 1963, 19, 251-258. ROSENTHAL, R. Assessing the effects of psychotherapy.
the statistical
and social importance
of
COTTON, J. L., & COOK, M. S. Meta-analyses and the effects of various reward systems: Some different conclusions from Johnson et al. Psychological Bulletin, 1982, 92, 176183.
Psychology, 1983, SI, 4-13.
DEVINE, E. C., & COOK, T. D. Effects of psychoeducational interventtons on length of post-surgical stay: A meta-analytical review of 49 studies. Nursing Research, in press.
ROSENTHAL, R. (Ed.) New directions for methodolog_v of socral and behavioral science: Quantitative assessment of research domains. San Francisco: Jossey-Bass, 1980.
FISK.
SMITH, outcome
D. W’. The meta-analytic
revolution
in outcome
research.
Journal of Consulting and Clinical Psychology, 1983, 51, 65-70. KALMAN, psychtatric
T. P. treatment.
An
overview
of
patient
satisfaction
with
Hospital and Communrty Psychiutry, 1983,
34, 48-54. LEBOW,
J. Consumer
satisfaction
with mental
health
treatment.
Psychological Bulletin, 1982, 9/, 244-259.
Journal of Consultrng and Cltnical
M. L., & GLASS, G. V. Meta-analysis of psychotherapy studies. American Psvchologist, 1977, 32, 752-760.
WEINSTEIN, R. M. Attitudes toward psychtatrtc hospitalized patients: A review of quantitative Science and Medicme, 1981, 158. 301-314. WILSON, evaluation
treatment research.
among
Socral
G. T., & RACHMAN, S. .I. Meta-analysis and the of psychology outcome: Limitattons and liabtlnies. Jour-
nal of Consulting and Clinical Psychology, 1983, 51, 54-69.
APPENDIX
Studies Included in Computerized Library Search of Literature on Patient Satisfaction and Psychiatric Care Since 1981
AHMED, hl. B., & KOLTUV, M. Evaluation of an inpatient service by consumer feedback. Australian and New Zeukund Journal of
P.sychratry, 1976, 10, 263-268.
ATTKISSON, C. C., & ZWICK, R. The Client Satisfaction Questionnaire: Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluatron and Program
Planning, 1982, 5, 233-237. ALLEN, J. C., & BARTON, G. M. Patient comments about hospitalization: Implications for change. Comprehensive Psychi-
BALCH,
atrv, 1976, 17, 631-640.
S. B. Client evaluation
P., IRELAND,
J. F., McWILLIAMS, S. A., & LEWIS, of a community mental health service: Rela-
A Meta-Analysis
to Establish
Norms
273
tion to demographic
and treatment variables. Amerfean Journaf of Community Psychology, 1977. 5, 243-247.
a& patient: A voice to be heard. British Journaf of Psychiatry, 1979, 235, 115-121.
BENE-KOCIEMBA, A., COTTON, P. G., & FORTGANG, R. C. Assessing patient satisfaction with state hospital and aftercare services. American Journal of Psychiatry, 1982, 139, 660662.
WYNE, J. B., & LADGUX, P. Patients’ opinions of outpatient clinic services. Hospitaf and Community Psychiatry, 1973, 24, 627-628.
BUDSON, R. D. The psychiatric haffway house. Pittsburgh: University of Pittsburgh Press, 1978.
GROB, M. C., EISEN, S. V., & BERMAN, J. S. Three years of follow-up monitoring: Perspectives of formerly hospitalized patients and their families. Comprehensive Psychiatry, 1978, 19, 491-499.
CHIEN, C. P., & COLE, J. 0. Landlord-supervised cooperative apartments: A new modality for community-based treatment. American Journal of Psychiatry, 1973, 130, 156-159. COHEN, C. I., SICHEL, W. R., & BERGER, D. The use of amidManhattan hotel as a support system. Community Mentaf Heafth Journal, 1977, 13, 76-83.
GYNTHER, M. D., REZNIKOFF, M., & FISHMAN, M. Attitudes of psychiatric patients toward treatment, psychiatrists and mental hospitals. Journai of Nervous and Mental Disease, 1963, (36, 68-71.
DENNER, B., & HALPRIN, F. Clients and therapists evaluate clmical services. American Journal of Community Psychofogy, 1974, 2, 373-386.
HARGREAVES, W. A., SHOWSTACK, J., FLOHR, R., BRADY, C., & HARRIS, S. Treatment acceptance following intake assignment to individual therapy, group therapy, or contact group. Archives of Generaf Psychfatty, 1974, 31, 343-349.
DENNER, B., & HALPRIN, F. Measuring consumer satisfaction in a community outpost. American Journal of Community Psychofogy, 1974, 2, 213-222.
HART, W. T., 8c BASSETT, L. Measuring consumer satisfaction in a mental health center. Hospital and Community Psychiatry, 1975, 26, 512-515.
DISTEFANO, M. K., PRYER, M. W., & GARRISON, J. L. Clients’ satisfaction and inter-personal trust among hospitalized psychiatric patients. Psychologicaf Reports, 1981, 49, 420-422.
IMRE, P., &WOLF, S. Attitudes of patients and personnel toward mental patients. Journal of Clmical Psychoiogy, 1%2,1& 232-234.
EDER, D., & KUKULSKI, S. A scale for measuring patient satisfaction with hospital services. Hospital and Community Psychratry, 1975, 26, 15-16. EDWARDS, D. W., YARVIS, R. M., MUELLER, D. P., & LANGSLEY, D. G. Does patient satisfaction correlate with success? Hosprtal and Community Psychiatry, 1978, 29, 188-190. EISEN, S. V., & GROB, M. C. Assessing consumer satisfaction from letters to the hospital. Hospital and Community Psychiatry, 1979,30, 344-347. EISEN, S. V., & GROB, M. C. Measuring discharged patients’ satisfaction with care at a private psychiatric hospital. Hospitaf and Community Psychiatry, 1982, 33, 227-228. EISENTHAL, S., KOOPMAN, C., & LAZARE, A. Process analysis of two dimensions of the negotiated approach in relation to satisfaction in the initial interview. Journaf of Nervous and Mentaf Dtsease, 1983, 171, 49-54. ELLSWORTH, R., & MARONEY, R. Characteristics of psychiatric programs and their effects on patients’ adjustment. Journaf of Consuiting and Clinical Psychology, 1972, 39, 436-447. FIESTER, A. R. Goal attainment and satisfaction scores for CMHC clients. American Journal of Communfty Psychofogv, 1979, 7, 181-188. FRYLING, V. B., & FRYLING, A. G. Patients’ attitudes toward sociotherapy. Psychiatric Quarterfy (Supplement), 196434, 97-115. GLENN, R. N. Measuring patients’ opinions about hospitali~tion using the Client Satisfaction Scale. Hospitaf and Community Psychiatry, 1978, 29, 627-628. GORDON, D., ALEXANDER, D. A., & DIETZAN, J. The psychi-
JANSEN, D. G., & ALDRICH, M. W. State hospital psychiatric patients evaluate their treatment teams. Hospital and Community Psychiatry, 1973, 24. 768-770. JUSTICE, B., & McBEE, G. A client satisfaction survey as one element in evaluation. Community Mental Health Journal, 1978, 14, 248-252. KAHN, M. W., & JONES, N. F. A comparison of attitudes of mental patients from various mental hospital settings. Journal of Cfinical Psychology. 1972,25, 312-316. KALMAN, T. P., TALON, N. S., FRANCES, A., & KOCSIS, J. H. A controlled study of satisfaction among psychobiology research patients. American Journal of Psychiatry, 1982, 139, 344-347. KLOPFER, W. G., WYLIE, A. A., & HILLSON, J. S. Attitudes toward mental hospitals. Journal of Clinical Psychoiogy, 1956, 12, 361-365. KOLTUV, M., AHMED, M. B., & MEYER, M. A study of outpatient satisfaction with mental health center services. Hospttal and Community Psychiatry, 1978, 29, 179-181, KOTIN, J., & SCHUR, J. M. Attitudes of discharged mental patients toward their hospital experiences. Journal of Nervous and Mentaf Dketze, 1969, 149, 408-414. LAMB, H. R. The new asylums in the community. Archives of General Psychiatry, 1979, 36. 129- 134. LEHMAN, A. F., WARD, N. C., & LINN, L. S. Chronic mental patients: The quahty of life issue. American Journal o~Psychfafry, 1982, 139, 1271-1276. LEVOIS, M., NGUYEN, T. D., & ATTKISSON, C. C. Artifact in client satisfaction assessment: Experience in community mental
274 health
rettings.
Evrrlwfion
and
ANTHONY
F. LEHMAN
Program
Plannmng, 19811 4,
139-150.
and
THOMAS
R.
ZASTOWNY
RICE, C. E., KLETT, S. L., BERGER, LEMKAU, P. V. The ward evaluatton
D. G., SEWALL., L. G., & scale. Journul of Clmrcal
Ps.veho~og_v,1963, f9. 251-258. LINN,
L. S. The
mental hospital Psychiairy, 1968, 31, 213-223.
from
the pattent
perspectwe.
LOVE, R. E., CAID, C. D., & DAVIS, A. The uber satisfactlon survey: Consumer evaluation of an inner city community mental health center. ~vui~~t~on und the Healrh Professions, 1979, i,
42-S. MANGEN, S P., & GRIFFITH, J. H. Patient satisfaction wth commumty psychiatric nursing: A prospective controlled study. Journul of Advanced Nursing, 1982, 7, 477-482.
ROBACK, H., & SNYDER, W. U. A comparison of hocpitahzed mental patients’ adjustment wth their attitudes toward psychiatric hospitals. Journul of Clinrcul Psychology, 1965, 21, 228-230. S. P., & AVIRAM, U. The rnen~ul~y iI/ in ~o~~unit~lbused sheltered care. Ne% York: John Wiley and Son\, 1978.
SEGAL,
SLATER, V.. LINN, M. W., &HARRIS, R. Outpatient of mental health care. Sourhern Me&al Journal,
evaluation
1981, 74,
1217-1219. MAYER, J. E., & ROSENBLATT, A. Clash rn perspective between mental patients and staff. Amerrcan Journal of Orthopsychratry,
1974. 44. 432-441.
SNELLING, E., & WALKER, F. Patient batlsfactlon ulrh facilities at an acute admission p\ychlatrlc unit. New Zealand Medicul Jour-
nal, 1978, 87, 94-97. PARK, L. C., SLAUGHTER, R., COVI, L., & KNIFFIN, H. G. The subjective experience of the research patient: An investlgatlon of psychiatric outpatIents to the research treatment snuation. Journal of Nervous und Men&f Lkease, 1966, 143, 199-206.
SOUELEM,
0. bfental
patients’
attitudes
toward
mental
hospitals.
Journal of Clrntcal Psychology, 1955, II, 181-185.
181-190.
TESSLER, R. C. Clients’ reactions to imtlal mterwe&s: Determinants of relatlonshlp-centered and problem-centered satisfaction. Journal of Counseling Psychology, 1975, 22, 187- 191.
REINHART, R., & CAZAVELAN, J. Mental health consumer survey on a shoestring budget -it 15 possible? .4merrcan Journal of
WEBER. tal heaith
Communrt_v Psychology, 1975, 3, 273-216.
29, 193-194.
POWELL, B. J., SHAW, D., & O’NEAL, C. Chent evaluation of a clintc’, serwes. Hosp~tui und Communrty Psychiatry, 1971, 22,
D. J., & TILLEY, D. H. Patient~‘evaluations of the menservice at a health sciences campus. College Health, 1981,