Testing the service satisfaction scale in Puerto Rico

EVALUATION and PROGRAM PLANNING PERGAMON

Evaluation and Program Planning 21 (1998) 81-92

Testing the service satisfaction scale in Puerto Rico’ Gisela Negrh-Vel8zquez”,*,

Margarita

Alegria”, Mildred Vera”, Daniel H. Freemanb

aCenterfor Evaluation and Sociomedical Research, Graduate School of Public Health, University of Puerto Rico. PO Box 365067, San Juan 00936-5067, Puerto Rico (CIES) b Department of Prevention Medicine and Community Health, University of Texas Medical Branch, Galveston, TX, U.S.A.

Abstract The present study examines the psychometric properties of the short version of the service satisfaction scale (SSS) and evaluates the consistency, reliability and interpretability of the findings for low income consumers of outpatient mental health services in Puerto Rico. The SSS is shown to be a valid instrument for assessing satisfaction among the target group. Respondents who received mental health services in both the public and the private system and users of the physical health sector identified satisfaction with the practitioner and with the outcome of care as the two most important dimensions of their responses to satisfaction. Consumers who used the mental health specially sector responded differently to the SSS in comparison to those who used the physical health sector. In this case, one factor integrated items of satisfaction with the practitioner and items of outcome with care, and another factor contained items related to satisfaction with the type of treatment. 0 1998 Elsevier Science Ltd. All rights reserved.

In the past two decades, policy makers and administrators have increasingly considered consumers’ opinions about mental health services as legitimate indicators of quality and outcome of care (Fiester and Fort, 1978; Pascoe, 1983). Governmental requirements to include the consumers’ perspectives in the evaluation of mental health services have facilitated this shift. Among the scientific community, studies of the psychometric properties of satisfaction scales have persuaded many researchers and evaluators to accept this construct as one type of indicator of quality of care and as an outcome measure. Major reviewers of this topic agree that, although few in number, these studies have provided empirical evidence of the consumers’ ability to differentiate specific dimensions involved in their responses to satisfaction with mental health services, and have demonstrated the validity and reliability of their responses (Pascoe and Attkisson, 1983; Lebow, 1983a; Ruggeri, 1994). Despite the greater acceptance and interest in satisfaction studies, there is still little agreement on what are the most important dimensions involved in consumers’ responses to satisfaction. The most common barriers in achieving consensus are methodological issues such as

* Corresponding author. ’ This study was funded by Grant 5ROl MH42655 from the Division of Biometric Health.

and Clinical

Applications,

National

Institute

of Mental

SO149-7189/98 $19.00 Q 1998 Eisevier Science Ltd. All rights reserved. PII:SO149-7189(97)00047-S

diversity in the type of satisfaction scale employed, the large number of scales containing dissimilar items, the variety of treatment environments where the scales are used and the lack of examination of the psychometric properties of the scales (Hall and Dornan, 1980; Pascoe and Attkisson, 1983; Lebow, 1982, 1983a, 1983b; Pascoe, 1983; Ruggeri, 1994). Still, studies that have included the domains of satisfaction with the provider, satisfaction with the outcome of services and general satisfaction have frequently identified these factors as relevant dimensions immersed in consumers’ responses to satisfaction with mental health and physical health services (Murphy, 1980; Kirchner, 1981; Slater et al., 1981; Essex et al., 1981; Jones and Zuppell, 1982; Bene-Kociemba et al., 1982; Distefano et al., 1981; Sishta et al., 1986; Greenfield and Attkisson, 1989; Pascoe and Attkisson, 1983; Lebow, 1982, 1983a, 1983b; Pascoe, 1983; Carscaddon and Wells, 1990; Huxley and Warner, 1992; Ruggeri and Dall’ Agnola, 1993; Ruggeri and Greenfield, 1995). Satisfaction with access to services, center accountability, office procedures, medicines, convenience, cost and physical environment, among many others, have also been reported in the mental health literature but with less frequency (see Ruggeri, 1994, for a list of factors reported in the past 13 years). Mental health satisfaction studies conducted in the U.S.A. have rarely included Puerto Ricans and Latinos in general. When minorities are included, the sample size is frequently too small to conduct data analyses for ethnic

82

G. Negrdn-Vekizquez et a/./Evaluation and Program Planning 21 (1998) 81-92

minority groups. With the exception of one study with a large sample of Mexican-Americans (Robert and Attkisson, 1983) satisfaction data concerning Latin0 groups is basically non-existent. Not surprisingly, the appropriateness of most satisfaction measures to assess responses among Latinos is still vague. Furthermore, it is also unknown whether the factor structure that emerges from Latin0 clients’ responses to satisfaction differ from those already reported in the health and mental health literature. The present study has two main objectives. First, to examine the psychometric properties of the short version of the service satisfaction scale (Greenfield and Attkisson, 1989) when used in a low income community sample of users of outpatient mental health services in Puerto Rico. This step will clarify the appropriateness of the SSS scale for this population. Second, the study seeks to examine the consistency, reliability and interpretability of the findings for Puerto Rican consumers when controlling for the type of health care sector used for mental health reasons (physical vs specially mental health) and the type of industry (public vs private services). This step will help evaluate whether the dimensions involved in the responses of Puerto Rican consumers are consistent across sectors and systems of care.

1. The service satisfaction

scale

The service satisfaction scale (SSS) was created by Greenfield and Attkisson (1989, 1995). The authors wanted to develop a multifactorial scale that could be used across sectors of care and with diverse ethnic population groups. Ware’s taxonomy of satisfaction with medical care (Ware et al., 1975; Ware et al., 1978) and Larsen and colleagues’ client satisfaction questionnaire (Larsen et al., 1979) provided the foundation work for the SSS. The end result of this effort was a long and a short version of the SSS. Both versions of the SSS were, tested in three different sites in California. Two of them were physical and mental health facilities at two different universities, while the third included four private nonprofit mental health organizations in Southern California. Factor analysis procedures showed that both the long and the short version of the SSS were indeed multidimensional measures of satisfaction (Greenfield and Attkisson, 1989; Attkisson and Greenfield, 1995). Still, the short version yielded more reliable factors. Table 1 presents the factors identified in the short SSS version. Because of the limited number of ethnic minority individuals included in the study (only 22% of the sample), these findings could not be generalized to diverse ethnic minority groups. Recently, the SSS was modified and used in Italy to examine satisfaction with community-based mental

health services (Ruggeri and Dall’Agnola, 1993; Ruggeri and Greenfield, 1995). Nevertheless, the SSS has not been validated among Latin0 groups. Further testing of the SSS among individuals from diverse ethnic backgrounds is still needed.

2. Methods and measures This study is based on an island-wide community survey conducted in 1989 among non-institutionalized residents of low income areas of Puerto Rico, aged 18-64 years. The population was sampled using a two-stage stratified cluster sample of the island. The strata were urban and rural areas. According to an economic index developed by the U.S. Department of Labor, these segments were classified as economically disadvantaged. The variables included in the index were media house rent, family income and house value. Eligible housing units were identified using the sampling frame prepared by the Department of Labor and Human Resources. A probability sample of clusters yielded a sample of 2682 housing units classified as economically depressed. Of these, 1642 units were eligible to participate in the study. The Kish method was used to randomly select one respondent from each eligible household (Kish, 1965). Lay interviewers successfully conducted a total of 1777 face-toface interviews during February to October 1989. The sample was weighted to make inferences to the Puerto Rican population living in low income areas. All analyses included the underlying survey design. Sampling weights were assigned depending on whether a participant was a resident of an urban or rural strata. Approximately 91.9% of the selected individuals were interviewed, so the inverse of this response rate was applied to further inflate the sample. The response rate inflation weight was computed for each cluster of housing units and applied to all individuals in the cluster. In addition, all individuals were assigned a post stratification weight that matched the age-sex population data reported by the U.S. Bureau of the Census of persons classified as poor in the Island for 1990. More details on the design and methodology of the present study can be found elsewhere (Alegria et al., 1991; Negron, 1994). The present study is based on a sub-sample of 243 respondents (13.67%) who received mental health services from the specialty sector and/or the physical care sector during the previous year. The short version of the service satisfaction scale was used to measure consumers’ satisfaction with mental health care received on the Island. The SSS was translated into Spanish using a simultaneous multiple translation procedure and panel deliberations (Negron, 1994). When the initial version of the SSS was facilitated to the Center for Evaluation and Sociomedical Research, the scale had a 7-point Likert type response range, where I was ‘terrible’ and seven was

G. Negh-Velcizquez Table Factors

I indentified

in the short version

of the service satisfaction

ITEMS:

How did you feel..

,’

Greenfield Attkisson

With the outcome

2.

With the outcome of services in terms of your general well-being of illness? About the medication?/about non-prescription of medication? About the relief of your symptoms? About referrals With services in general?

I. 8. 9. 10. Il. 12. 13

With With With With With With With

the the the the the the the

of services in terms of helping

scale in California

1.

3. 4. 5. 6.

83

et al./Eualuation and Program Planning 21 (1998) 81-92

you resolve your problems? and prevention

PO

Psychometric Properties Cronbach x PO’ = 0.80

PO PO PO PO PMS

opportunity to select the professional? knowledge of the professional? capacity of the professional to listen and understand your problems? personal treatment offered by the professional? confidentiality and respect that the professional showed to you? explanations of the professional regarding the treatment? thoroughness of the professional?

Cronbach G( PMS’ = 0.87

PMS PMS PMS PMS PMS PMS PMS

Satisfaction with: PO = Perceived Outcome; PMS = Practitioner’s Manners and Skills. ’ Two items related to emergency care were not included in this final version of the SSS scale due Attikisson, 1989). ’ To calculate the Cronbach a for the ‘PO’ factor, Greenfield and Attkisson included three additional in this factor in the long version of the SSS scale. These were amount of help received, information on contribution to achieving life goals (see page 276, Greenfield and Attkisson, 1989). ’ To calculate the Cronbach G(for the ‘PMS’ factor, Greenfield and Attkisson included an item related 276, Greenfield and Attkisson, 1989. ’ For verbatim wording and original order of items, see page 273, Greenfield and Attkisson, 1989.

‘delighted’. This initial version was later changed by the authors to a 5-point Likert scale. By the time this adaptation was made, the community survey on which this article is based was already in progress. Because of this limitation, comparisons with the original SSS scale will be made with caution. For the present study, the length of the original short version of the SSS was modified to include an item related to satisfaction with ‘frequency of services’ increasing the total number of items from 14 to 15. This addition was made to briefly include an important aspect related to the ability of the system to provide care to the population in need. This item was congruent with two dimensions from Ware’s taxonomy (availability and accessibility/ convenience), which was one of the conceptual dimensions of the service satisfaction scale (Ware et al., 1975, 1978). Differences in a set of characteristics of consumers who used specific sectors (physical/specialty mental health) and systems of care (public/private) were tested with Z scores. This analysis was conducted previous to factor analyses procedures on the SSS to evaluate if the sociodemographic characteristics and health insurance status of the groups of interests differed. Missing values were handled through likewise methods. Changes in the total number of consumers reported across sectors and systems of care are a product of missing values.

&

(I 989)4

to high non-response

rate (Greenfield

and

items which have met criteria for inclusion how to get the most from the services, and to ‘type kind of services offered’ (see page

3. Results Table 2 displays the characteristics of the groups of interest. As seen in Table 2, the majority of consumers from both the public and private system were females. No significant differences were found in the marital status and age of consumers who used either system of care. However, consumers who used the private system of care had significantly higher levels of education (3 12 years) and were significantly more likely to be employed than consumers who used the public system. Because the majority of consumers who used the public system of care had Medicaid, this group emerged as significantly more likely to have insurance when compared to consumers who used the private system of care (P < 0.05). The characteristics of consumers who used the physical health sector to receive mental health care were very similar to those consumers who used the mental health specialty sector. However, significant differences were found in their employment status, with higher employment rates reported by consumers who used the physical health sector (P d 0.05). Consumers from the mental health specialty sector were significantly more likely to be retired or incapacitated and significantly less likely to be students (Table 2). Before examining the factor structure of the service

84

G. Negrbn-Vekizquez

Table 2 Characteristics

of consumers

Sectors:

of mental

et al./Evaluation

and Program Planning 21 (1998) 81-92

health services by health sector and health system utilized for mental health reasons

Total Sample n = 215 N = 110,970

Public System n= 91 N = 44,820

%

%

(SE)

%

(SE)

%

(SE)

%

(SE)

Sex Male Female

39.1 60.3

36.8 63.2

(5.7) (5.7)

41.6 58.4

(4.9) (4.9)

37.9 62.1

(4.3) (4.3)

45.1 54.9

(7.6) (7.6)

Age 18-24 25-34 35-44 45-54 55-64

14.4 15.3 20.4 29.1 20.2

16.3 18.1 15.0 30.1 20.5

(3.8) (4.3) (4.2) (6.0) (5.1)

13.1 13.5 24.1 29.4 20.0

(4.0) (2.8) (3.9) (4.7) (3.4)

16.8 15.9 19.4 28.8 19.1

(3.4) (2.7) (3.4) (4.1) (2.9)

6.9 13.7 23.5 32.4 23.5

(3.4) (4.2) (6.1) (7.8) (6.4)

Education <12yrs > 12 yrs

63.0 37.0

71.1 28.9

(5.1) (5.1)

57.6 42.4

(5.4) (5.4)*

61.5 38.5

(4.7) (4.7)

67.7 32.3

(6.9) (6.9)

Marital Status Married Non-married

60.3 39.7

63.3 36.7

(5.0) (4.2)

58.4 41.6

(5.0) (4.2)

62.1 37.9

(3.7) (3.7)

54.9 45.1

(7.1) (7.1)

25.6 32.7 32.2

15.2 49.1 33.3

(3.9) (5.9) (6.1)

32.7 21.6 31.4

(4.8)*

28.9 32.8 26.6

(4.0)*

(4.2) (4.0)

(4.0) (3.4)

15.7 32.4 49.0

(4.8) (7.1) (7.8)*

9.5

2.4

(1.6)

14.3

(4.0)

11.7

(3.3)*

2.9

(2.1)

27.0 73.0

92.8 7.2

(2.8)*’

59.6 40.4

(5.0) (5.0)

70.2 29.8

(3.8) (3.8)

81.4 18.6

(6.0) (6.0)

Sample n Population

Consumers

N

Private System n= 124 N= 66,150

Physical Sector n = 163 N = 83,430

Mental Sector n = 52 N = 27,540

characteristics

Employment Employed Unemployed Retired or incapacitated Student Insurance Yes No *‘Insurance

included

(2.8)

Medicaid.

satisfaction scale across the groups of interest, the response distribution of the SSS was examined (data not shown). It was found that approximately 90% of the participants did not respond to one of two items related to medication. Item number 4, which examined satisfaction when medication was not prescribed, was eliminated from further analysis (Table 1). Findings also revealed that about 14% of the participants did not respond to an item related to satisfaction with referrals. This item was also excluded from further analyses. Principal component analysis using orthogonal rotation with the varimax procedure (Catell, 1978) was conducted on the final 13 item scale. The rules of thumb utilized for the selection of items under a factor were: (1) Kaiser criterion-eigenvalue > 1 (Catell, 1978); (2) an item could be included exclusively under one factor, usually the factor that reflects the highest item-loading (loadings > 0.50); and, (3) factors should have more than two items.

The varimax procedure yielded two relatively orthogonal factors in the total sample of consumers. These patterns accounted for approximately 72% of the total variance (Table 3). On the first factor, seven items reached statistical significance using the criteria of highest loading >0.50. Four of these items had been already classified by Greenfield and Attkisson as measuring satisfaction with ‘Perceived Outcome’ (PO) of care. These items were, satisfaction with: (1) the help received to resolve the problem; (2) the help received to maintain well-being and prevent illness; (3) the medication; and (4) the relief of symptoms (Table 3). In the present study, three more items related to satisfaction with the process of care loaded heavily on this factor. These are, satisfaction: (5) with the frequency of services; (6) with the opportunity to select the practitioner; and (7) general satisfaction. This factor accounted for approximately 36% of the explained variance.

G. Negrdn-Vekizquez et al./Eoaluation and Program Planning 21 (1998) 81-92 Table 3 Principal

factor

ITEMS: 1. 2. 3. 5. I. 8. 9. 10. 11. 12. 13. 14. 15.

analysis

with varimax

rotation

orthogonal

transformation

matrix

the service satisfaction Factor

How did you feel..

With the outcome of services in terms of helping you resolve your problems? With the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed to you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional? Total h’ Total Eigenvalues Total Variance Common Variation

85

scale Sl

Factor

S2

h’

0.75*

0.39

0.71

0.78*

0.22

0.67

0.69* oJ33* 0.72* 0.68* 0.69* 0.47 0.44 0.21 0.19 0.46 0.49

0.21 0.25 0.38 0.48 0.35 0.72* 0.79* 0.88* 0.89* 0.76* 0.71*

4.17 36.39 51.00%

4.60 35.38 49.00%

0.53 0.75 0.67 0.69 0.59 0.75 0.83 0.82 0.82 0.73 0.74 9.31 9.37 71.77 100%

Population N = 110,970; Sample n = 215 * Refers to significant variables.

In the second factor, six items related to consumer satisfaction with the provider were found to be significant with loadings above 0.60 (Table 3). These items are: (1) satisfaction with the practitioner’s knowledge and competence; (2) their capacity to listen and to understand the problem; (3) personal treatment; (4) confidentiality; (5) explanations; and (6) thoroughness (Table 3). These items and two additional items had been labelled by Greenfield and Attkisson (1989) as satisfaction with the ‘Perceived Manner and Skills’ of the provider. The contribution of the second factor identified in the present study on the explained variance was similar to the first factor, accounting for approximately 36% of the variance. Factor analyses with oblique rotations were conducted on the SSS used in Puerto Rico (results not shown, but available upon request to the first author). This procedure, which assumes correlation among the rotated factors, resulted in the same factor-structure. Item loadings and eigenvalues were very similar to those reported with orthogonal rotation. Therefore, the interpretation of the factor structure did not change. In general, the content of the two factors identified in the present study reflect the same dimensions reported by Greenfield and Attkisson (1989). However, the itemcomposition of the factors was relatively different. For this reason, the authors selected new labels to classify the factors found among the total sample of Puerto Rican consumers of mental health services (Table 4). The first factor identified was labelled satisfaction with perceived outcome and process of care (POP); the second factor was labelled satisfaction with the perceived personal and

professional treatment (PPT) offered by the practitioner. When examining the internal reliability of the identified subscales, both the PPT and the POP subscales yielded very high coefficients, with Cronbach CIvalues of 0.91 and 0.94, respectively. This finding demonstrates that, in spite of the differences in the content of the factor-based substales identified in the present study and the factor-based subscales identified by Greenfield and Attkisson, both the PPT and the POP subscales are as reliable as the originals subscales reported by Greenfield and Attkisson (1989). 3.1. Testing the SSS among consumers from and the private system of care

the public

In the next step, the total sample of consumers of mental health services was divided into two groups. One consisted of consumers who used the public system to receive mental health care and the other represented consumers of the private system of care. This analysis would allow us to determine if the same factor structure identified on the SSS remained for consumers of the different systems of care. Principal component analysis followed by orthogonal rotation was conducted again on the SSS for each group of interest. Responses provided by consumers from the public system of care replicated the two factor-structure identified among the total sample of consumers. Both the PPT and the POP subscales contained exactly the same items identified among the total sample of consumers (Table 5). The only difference was found in the order and

86

G. Negr6n-Velrizquez et al.lEvaluation and Program Planning 21 (1998) 81-92

Table 4 Comparison

of factors

ITEMS:

1. 2. 3. 5. 7. 8. 9. 10. 11. 12. 13. 14. 15.

indentified

in the United

States and Puerto

Rico

How did you feel.

Greenfield Attkisson

With the outcome of services in terms of helping you resolve your problems? With the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? N/A With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional?

Satisfaction with: PO = Perceived and Professional Treatment.

Outcome;

PMS = Practitioner’s

Manners

strength of the factors. In the public sector, the POP subscale was the stronger factor, accounting for approximately 43% of the explained variance, while the PPT subscale accounted for approximately 33% of the lained variance. Results from the private system of care replicated the factor structure reported among the total sample of con-

Table 5 Principal

factor

ITEMS: 1. 2. 3. 5. I. 8. 9. 10. 11. 12. 13. 14. 15.

analysis

with varimax

rotation

orthogonal

transformation

Negron-Velazquez (1997)

PO

POP

PO

POP

PO PO POP PMS PMS PMS PMS

POP POP POP POP PPT PPT

PMS PMS

PPT PPT

PMS PMS

PPT PPT

and Skills; POP = Perceived

Outcome

and Process;

et al.,

PPT = Personal

sumers (Table 6). An exception was found on the ‘thoroughness’ item where a very small variation in its loading was found between factors (0.01). According to the rules of thumb selected for this study, the ‘thoroughness’ item should have been included under the POP subscale. However, because it is possible that the small variation in the loading of this item was a result of ran-

matrix

the service satisfaction

How did you feel..

With the outcome of services in terms of helping you resolve your problems? With the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed to you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional? Total h2 Total Eigenvalues Total Variance Common Variance

Population N = 44,820; Sample n = 9 1. * Refers to significant variables.

&

(1989)

Factor

scale public sector Sl

Factor

S2

h*

0.45 0.17

0.65* 0.78*

0.63 0.64

0.15 0.22 0.50 0.59 0.48 og3* 0.88* 0X9* 0.86* 0.86* 0?34*

0.78* 0.86* 0.70* 0.62* 0.60* 0.4 0.35 0.1 0.16 0.40 0.3

5.51 42.84 57.00%

4.24 32.61 43.00%

0.63 0.78 0.75 0.75 0.59 0.85 0.90 6.82 0.76 0.89 6.83 9.81 9.81 75.45 100%

G. Negrbn-Velizquez Table 6 Principal

factor

ITEMS: 1. 2. 3. 5. 7. 8. 9. 10. 11. 12. 13. 14. 15.

analysis

with varimax

rotation

orthogonal

et al./Eualuation and Program Planning 21 (1998) 81-92

transformation

matrix

the service satisfaction

scale private

Factor

How did you feel.

With the outcome of services in terms of helping you resolve your problems? With the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed to you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional? Total h’ Total Eigenvalues Total Variance Common Variation

Sl

87

sector Factor

S2

h2

ago* 0.81*

0.37 0.28

0.77 0.73

0.54* 0.79* 0.72* 0.72* 0.78* 0.51 0.50 0.20 0.21 0.49 0.59*

0.38 0.35 0.24 0.37 0.18 0.63* 0.70* 0.88* 0.89* 0.65* 0.58

5.03 38.69 56.00%

3.92 30.15 44.00%

0.44 0.74 0.58 0.66 0.64 0.65 0.74 0.82 0.83 0.78 0.74 8.95 8.95 68.84 100%

Population N = 66,150; Sample n = 124. * Refers to significant variables.

dom error and because this item was conceptually more adequate under the factor related to the practitioner’s personal and professional treatment, the ‘thoroughness’ item was kept as part of the group of items defining the PPT subscale. Among consumers who used the private system of care, satisfaction with the practitioner (PPT) accounted for 30% of the explained variance, while satisfaction with the process and outcome of care (POP) accounted for about 39% of the explained variance. 3.2. Testing the SSS among consumers from and the mental health specialty sector

the physical

In the final step of this analysis, the total sample of consumers of mental health services was divided into two new groups. One represented consumers who used the physical health sector to deal with their mental health problems, and the other group were users of the mental health specialty sector. The consistency of the satisfaction dimensions identified in the service satisfaction scale among the total sample of consumers was then examined across these two groups. Because of the small sample size, specially for the mental health specialty sector, the results and conclusions of factor analytical procedures will be made with caution. Table 7 presents the results for consumers who used the physical health sector of care for mental health reasons. As seen in Table 7, results from principal component analysis with orthogonal rotation replicated the two factor-structure previously identified among the total sample of consumers. Thus, the two satisfaction-related dimensions found among consumers who sought mental

health services in the physical health sector were satisfaction with the process and outcome of care (POP) and satisfaction with the practitioner (PPT). The explanatory power of both the PPT and the POP subscales was well balanced, accounting for approximately 34% and 36% of the explained variance, respectively. Satisfaction responses provided by consumers from the mental health specialty sector were different than those provided by the other groups of interest. Although orthogonal rotation using the varimax procedure yielded two statistically independent patterns of relationship, the item-composition of the two factors had some differences from those already reported in the present study and in Greenfield & Attkisson’s report. As seen in Table 8, the first factor contained eight items. The first four items are related to satisfaction with the outcome of services. These items are, satisfaction with: (1) the help received to resolve the problem; (2) the help received to maintain well-being and prevent illness; (3) the relief of symptoms; and (4) the general satisfaction marker. The last four items identified under the first factor were related to satisfaction with the practitioner. These items are, satisfaction with the practitioner’s: (5) knowledge and competence; (6) capacity to listen and understand the problem; (7) personal treatment; and (8) confidentiality. As presented in Table 8, the loadings were consistently high, with a minimum value of 0.71. This first factor accounted for approximately 44% of the explained variance. Because of the combination of dimensions involved in this factor, it was labelled satisfaction with the ‘Practitioner and the Outcome’ of care (PO). As seen in Table 8, the second factor identified among

88

G. Negrdn-Veltizyuez et al./Evaluation and Program Planning 21 (1998) 81-92

Table 7 Principal

factor analysis

ITEMS: 1. 2. 3. 5. 7. 8. 9. 10. 11. 12. 13. 14. 15.

with varimax

rotation

orthogonal

transformation

matrix the service satisfaction

How did you feel.

Factor

With the outcome of services in terms of helping you resolve your problems? with the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed to you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional? Total h’ Total Eigenvalues Total Variance Common Variation

scale physical Sl

health sector

Factor

S2

hZ

0.75* 0.81*

0.38 0.22

0.71 0.70

0.68* 0.85* 0.69* 0.67* 0.68* 0.47 0.45 0.14 0.12 0.47 0.51

0.13 0.15 0.39 0.45 0.34 0.70* 0.77* 0.90* 0.90* 0.76* 0.69*

4.74 36.46 51.00%

4.47 34.38 49.00%

0.48 0.75 0.63 0.66 0.58 0.71 0.80 0.83 0.82 0.80 0.74 9.21 9.21 70.84 100%

Population N = 83,430; Sample n = 163 * Refers to significant variables.

Table 8 Principal

factor analysis

ITEMS: 1. 2. 3. 5. 7. 8. 9. 10. 11. 12. 13. 14. 15.

with varimax

rotation

orthogonal

transformation

matrix

the service satisfaction

scale mental

Factor

How did you feel..

With the outcome of services in terms of helping you resolve your problems? With the outcome of services in terms of your general well-being and prevention of illness? About the medication? About the relief of your symptoms? With the frequency of services? With services in general? With the opportunity to select the professional? With the knowledge of the professional? With the capacity of the professional to listen and understand your problems? With the personal treatment offered by the professional? With the confidentiality and respect that the professional showed to you? With the explanations of the professional regarding the treatment? With the thoroughness of the professional? Total h2 Total Eigenvalues Total Variance Common Variance

Sl

health sector Factor

S2

h*

0.82* 0.78*

0.38 0.17

0.82 0.65

0.37 0.53* 0.37 0.80* 0.51 0.71* 0.75* 0.78* 0.77* 0.36 0.31

0.72* 0.44 0.80* 0.45 0.68* 0.56 0.52 0.43 0.48 0.87* 0.88*

5.67 43.61 55.00%

4.72 36.30 45.00%

0.65 0.89 0.78 0.85 0.72 0.82 0.84 0.79 0.83 0.88 0.88 10.39 10.39 79.91 100%

Population N = 27,540; Sample n = 52. * Refers to significant variables.

consumers of the mental health specialty sector contained five items. These items are directly related to the type of treatment provided, who provided the treatment, how often is the treatment provided and how well the treatment method(s) are explained to the consumer. More specifically, the items are, satisfaction with: (1) the medi-

cation; (2) the frequency of services; (3) the opportunity to select the practitioner; (4) the explanations of the professional regarding the treatment; and (5) the thoroughness of the professional. Conceptually this factor stresses the treatment aspect of the service experience, and was therefore labelled satisfaction with the ‘Type of Treat-

G. Negrbn-Vekizquez et al./Ecaluation and Program Planning 21 (1998) 81-92 Table 9 Cronbach

LXfor identified

factors

on the service satisfaction

Sector System of Care

Subscales POP

PPT

All Sample Public System Private System Physical Health Sector Mental Health Sector

0.91 0.91 0.91 0.93 0.95

0.94; 0.95 0.91. 0.93. 0.93.

PO

scale TT

POP = Perceived Outcome and Process; PPT = Personal and Professional Treatment; PO = Practitioner and Outcome; TT = Type of Treatment.

ment’ (TT). This factor accounted 36% of the explained variance. 3.3. Internal reliability

for approximately

of the SSS-subscales

The internal reliability of the subscales identified through factor analysis procedures was examined for the total sample of consumers and for each group of interest. As seen in Table 9, the coefficients of internal reliability were very similar for the POP subscale across sectors and systems of care, with Cronbach a values ranging from 0.91 to 0.93. The PPT subscale also yielded high coefficients of internal reliability across sectors and systems of care, ranging from 0.91 to 0.95 (Table 9). The internal reliability of the two different factor structures identified in the mental health specialty sector were also examined. In this case, the subscale related to satisfaction with the ‘Practitioner and the Outcome’ of care (PO) yielded a Cronbach CIvalue of 0.95, while the ‘Type of Treatment’ (TT subscale yielded a coefficient of internal reliability of 0.93. 4. Discussion Findings from factor analytic procedures conducted with the short version of the service satisfaction scale validated the notion of multidimensionality in consumer responses to satisfaction. In general, the findings showed that, when assessing their mental health service experience, Puerto Rican consumers residing in low income areas were able to differentiate two specific satisfactionrelated dimensions. One of these dimensions referred to how well the practitioners treated the consumers. This aspect involved basic empathetic, professional and humane facets of the clinical encounter such as showing respect for the consumer, paying attention to what he/she had to say, maintaining confidentiality, providing clear information, etc. The relevance of these aspects on satisfaction have been previously reported in the medical care and mental health literature (Murphy, 1980; Essex et al., 1981; Kirchner, 1981; Slater et al., 1981; Jones and

89

Zupell, 1982; Bene-Kociemba et al., 1982; Distefano et al., 1981; Dyck and Azim, 1983; Azim and Joyce, 1986; Klein-Buller and Buller, 1987; Greenfield and Attkisson, 1989; Ruggeri and Dall’Agnola, 1993; Ruggeri and Greenfield, 1995). As in other studies, consumers also paid much attention to how effective the services were to alleviate and/or to resolve the problem that brought them to care (Essex et al., 1981; Greenfield and Attkisson, 1989; Ruggeri and Dall’Agnola, 1993; Attkisson and Greenfield, 1995). In general, the emergence of the two satisfaction substales identified in the present study was conceptually congruent with Greenfield and Attkisson’s report on the SSS (1989). Still, the variation identified in the itemcomposition of the factors deserves some attention. First of all, it is important to highlight that two of the original items included under the ‘Perceived Outcome’ subscale reported by Greenfield and Attkisson (1989) were not suitable for analysis in the present study. These items were, satisfaction when medication is not prescribed and satisfaction with referrals. The high non-response rate on these items by Puerto Rican consumers may reflect general characteristics of the services received. More specifically, this situation shows that most of the consumers who participated in the present study received medication as part of the treatment, and that many consumers did not receive referrals to other sources of care. From the items that were suitable for factor analysis procedures, two items consistently emerged under a different factor structure when compared to the report of Greenfield and Attkisson (1989). These were satisfaction with services in genera1 and satisfaction with the opportunity to select the professional. In the present study, these two items were grouped under the factor related to satisfaction with the practitioner, while in Greenfield and Attkisson’s report these two items were part of the factor related to satisfaction with the perceived outcome. A possible explanation for these differences may be that, when assessing satisfaction with the practitioner, Puerto Rican consumers consider not only the practitioner as an individual, but the process through which they initiate and finish the client-practitioner experience. The initial stage of services may be represented by the item related to satisfaction with the opportunity to select the practitioner. The final stage of treatment within this dyad may be represented by the genera1 satisfaction marker. Despite these differences, the psychometric properties of the factor-based subscales identified in the present study demonstrated the high internal reliability of the SSS for a poor Hispanic population, supporting the conclusion of Greenfield and Attkissons about the scale. Interestingly, the responses of consumers from the specialty sector yielded two factors, but the item composition and the interpretability of the factors was different to those identified among the other groups of interest. More specifically, the first factor identified in this group (sat-

90

G. Negrdn-Vehizquez et al./Evaluation and Program Planning 21 (1998) 81-92

isfaction with the ‘Practitioner and the Outcome’ of care) contained a combination of items which could be seen as an integrated version of the two factors identified among the other groups of interest: satisfaction with the way in which the practitioner offered the services and satisfaction with the outcome of care. The content of this factor is conceptually congruent with a general satisfaction factor frequently cited in the physical and mental health literature (Fiester and Fort, 1978; Larsen et al., 1979; Love et al., 1979; Distefano et al., 1981; Sishta et al., 1986; Carscaddon and Wells, 1990; Huxley and Warner, 1992; Ruggeri and Dall’Agnola, 1993). The second satisfaction factor identified in the present study among consumers from the speciality sector was related to the type of treatment received. The ‘Type of Treatment’ factor involved the ability of selecting the provider, the type of medication prescribed, the explanations of the professional regarding the treatment, the thoroughness, and the frequency of services. A conceptually-congruent factor was reported in a recent study conducted among patients’, relatives, and professionals in community-based psychiatric services in Italy using the Verona Service Satisfaction Scale (Ruggeri and Dall’Agnola, 1993). The emergence of the different factors identified in the specialty group when compared to the physical health sector may have been a product of the small sample included in the specialty sector. However, the clear interpretability of the identified factors and their high internal reliability justify the effort to present other possible explanations. As an alternate interpretation, the authors propose that it is possible that the service expectations of consumers who used the mental health specialty sector are different than the expectations of those consumers who used the physical health sector for mental health reasons. More specifically, consumers from the specialty sector may go to treatment expecting to receive the ‘specialized’ care that this sector is supposed to offer. In the physical health sector, satisfaction with the type of treatment did not emerge as a relevant dimension related to satisfaction probably because ‘specialized’ care such as counseling, group therapy, etc. are not specialties within this sector but optional treatment styles (Meredith et al., 1994). Comparing treatment styles used by family physicians and mental health professionals with treatment of depression, Meredith et al. (1994) reported that mental health specialists showed the strongest counseling preferences, provided the most counseling in actual practice and prescribed more medications than family physicians. Although there is a lack of comparable data in Puerto Rico, the findings of Meredith et al. (1994) shows some support for the alternative explanation proposed to interpret the different factors identified in the specialty group. Although the factors identified among consumers from the mental health specialty sector had not been reported

by Greenfield and Attkisson (1989) their emergence showed the capacity of the SSS to capture different satisfaction-related dimensions which were relevant for a particular group of consumers in this study. Further examination of the SSS with a larger sample of consumers from the specialty sector could help clarify these findings. Furthermore, reported differences between the study of Greenfield and Attkisson (1989) and the present report may be explained by cultural differences between the targeted groups on each study (Anglos vs Puerto Ricans), differences in the mental health delivery system, and/or differences in the consumers’ expectations of their service delivery experience. The use of a community sample vis a vis the clinical sample included in the study by Greenfield and Attkisson (1989) may have also accounted for the variation in factor structure found between the two studies. Because the short version of the SSS is not an exhaustive measure of satisfaction, there are many aspects related to satisfaction with mental health care that could not be examined in this study. Still, the underlying dimensions identified on the SSS for the different groups of interest have strong implications for service provision on the Island. During the past four years, the health system in Puerto Rico has experimented a shift to managed care. Some of the critics to this reform include the over-emphasis that managed care places on the outcome of services and the lack of attention to the process and quality of care. The fact that Puerto Ricans living in low income areas were able to differentiate two specific dimensions related to satisfaction highlights the importance that consumers give not only to the outcome of services, but also to the practitioner-client relationship during the service experience. This dimension, which is more related to quality of care, must not be forgotten when evaluating current mental health services in Puerto Rico under managed care. Local program planners, policy makers and evaluators involved in the health reform should also take into account that consumers who use the specialty sector seem to have a different set of service-expectations than those who used the physical health sector of care for mental health reasons. These expectations appear to include special attention to the way in which services are provided (process of care), the type of treatment received, the quality and the outcome of care. Therefore, the inclusion of a multidimensional satisfaction measure in the evaluation of mental health services under the current health care reform is strongly recommended. The validated SSS can be used as an essential outcome measure within the managed care reform. A next step could be to examine the relationship between satisfaction and other outcome measures such as treatment compliance and continuity of care. The present study confirmed the advantages of validating existing satisfaction scales with minority groups. The literature on utilization of health services indicates

G. Negron-Velazquez et aLlEvaluation and Program Planning 21 (1998) 81-92

that patterns of utilization vary widely across ethnic groups (Warheit et al., 1982; Scheffler and Miller, 1989; Hu et al., 1991). Thus, the subjective evaluation of the treatment experience, whether in the specialty or the general care sector, may also vary widely across different ethnic groups. Therefore, taking the time and the effort to clarify the validity, reliability and interpretability of satisfaction measures when used among ethnic-minority groups, and discussing the implications for service provision are, indeed, valuable aims for researchers in the field. 4.1. Limitations of the study The present study has several limitations. Responses provided on the service satisfaction scale corresponded to services received within a one year period. This interval may have affected the consumers’ ability to evaluate their service experience. However, the high internal reliability of the reported factors suggested that this time frame may have influenced, but not distorted, responses provided by the consumers. The findings are not applicable to all poor individuals in Puerto Rico or poor Puerto Ricans in the U.S.A. Poor individuals living in higher income areas of the Island, individuals who are in institutions (such as prison) and the homeless were not included in the study. Therefore, the scale should be validated with these additional populations. Because of cultural differences within Latin0 groups, the reported findings may not be generalizable to other Latin0 groups. Further examination of the adequacy of the SSS when used among diverse ethnic-minority groups, including other Latinos, is still recommended.

Acknowledgements Negron-Velazquez is grateful to: (1) CIES members for kindly agreeing to share available data; (2) Council on Social Work Education; (3) Dr Paul Lerman, Dr Michael Camasso, Dr Uri Aviram and Dr Peter Guarnaccia and Dr Glorissa Canino for their assistance in completing this project; and (4) Daniel R. Garza for editing management. The authors are also grateful to Jose M. Calderon and Marisol Pefia for their assistance in data management, and to Maria Lebron for assisting in the manuscript preparation.

References Alegria, M., Robles, R., Freeman, D. H., Vera, M., Jimenez, A. L., Rios, C., & Rios, R. (1991). Patterns mental health utilization among island Puerto Rican poor. American Journal of Public Health, 81(7), 875-879.

91

Andrews, F. M., & Witney, S. B. (1976). Social indicators of wellbeing and American perceptions of ltfe quality. New York: Plenum Press. Attkisson, C. C., & Greenfield, T. K. (1994). The Client Satisfaction Questionnaire-8 and the Service Satisfaction Questionnaire-30. In M. Maruish (Ed.), Psychological testing: Treatment planning and outcome assessment (pp. 402420). San Francisco: Lawrence Erlbaum Associates. Attkisson, C. C., & Greenfield, T. K. (1995). The Client Satisfaction Questionnaire (CSQ) scales and the Service Satisfaction Scale-30 (SSS-30). In L. I. Sederer & B. Dickey (Eds.), Outcomes assessment in clinicalpractice (pp. 120-127). Baltimore, MD: Williams & Wilkins. Azim, H. F. A., &Joyce, A. (1986). The impact of data-based program modifications on the satisfaction of outpatients in group psychotherapy. Canadian Journal of Psychiatry, 31, 119-122. Bene-Kociemba, A., Cotton, P. G., & Fortgang, R. C. (1982). Assessing patient satisfaction with state hospital and aftercare services. American Journal of Psychiatry, 139(5), 66G662. Carscaddon, D. M., & Wells, G. M. (1990). Rural community mental health consumer satisfaction and psychiatric symptoms. Journal of Community Mental Health, 26, 309-3 18. Catell, R. B. (1978). The scient$c use offactor analysis in behavior and life sciences. New York: Plenum. Distefano, M., Pryer, M., & Garrison, J. (1981). Client satisfaction and interpersonal trust among hospitalized psychiatric patients. Psychological Reports, 49(2), 42&422. Dyck, R. J., & Azym, H. F. (1993. Patient satisfaction in a psychiatric walking clinic. Canadian Journal of Psychiatry, 28, 3&33. Essex, D., Fox, J., & Groom, J. (1981). The development, factor analysis, and revision of a client satisfaction form. Community Mental Health Journal, 17(3), 226235. Fiester, A., & Fort, D. (1978). A method of evaluating the impact of services at a community mental health center. American Journal of Community Psychology, 6,291-302. Greenfield, T. K., & Attkisson, C. C. (1989). Steps toward a multifactorial satisfaction scale for primary care and mental health. Evaluation and Program Planning, 12(3), 271-278. Hall, J. A., & Dornan, M. C. (1990. Patient sociodemographic characteristics as predictors of satisfaction with medical care: A meta analysis. Social Sciences and Medicine, 30(7), 81 l-818. Hu, T., Snowden L. R., Jewell, J. M, & Nguyen, T. D. (1991). Ethnic populations in public mental health service choice and level of use. American Journal of Public Health, 81, 1429-1434. Huxley, P., &Warner, R. (1992). Case management, quality of life, and satisfaction with services of long-term psychiatric patients. Hospital and Community Psychiatry, 43, 799-803. Jones, E. E., & Zupell, C. L. (1982). Impact of client and therapist gender on psychotherapy, process, and outcomes. Journal of Consulting and Clinical Psychology, 50(7), 2599272. Kirchner, J. H. (1981). Patients feedback on satisfaction with direct services received at a community mental health center: A two year study. Psychotherapy: Theory Research and Practice, 18, 359-364. Klein-Buller, M., & Buller, D. B. (1987). Physicians’ communication style and patient satisfaction. Journal of Health and Social Behavior, 28(4), 3755388. Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979). Assessment of client/patient satisfaction: Development of a general scale. Evaluation and Program Planning, 2(3), 197-207. Lebow, J. (1982). Consumer satisfaction with mental health treatment. Psychological Bulletin, 91(2), 244259. Lebow, J. (1983a). Research assessing consumer satisfaction with mental health treatment: A review of findings. Evaluation and Program Planning, 6(34), 21 I-236. Lebow, J. (1983b). Similarities and differences between mental health and health care evaluation studies assessing consumer satisfaction. Evaluation and Program Planning, 6(3/4), 237-245. Love, R. E., Caid, C. D., & Davis, A. (1979). The user satisfaction

92

G. Negrdn-Vehizquez

et aLlEvaluation

survey: Consumer evaluation of an inner city community mental health center. Evaluation and the Health Profession, 2, 42254. Meredith, L. S., Wells, K. B., & Camp, P. (1994). Clinician specialty and treatment style for depressed outpatients with and without medical comorbidities. Archives of Family Medicine, 3(12), 10655 1072.

Munoz, C. M., Alegria, M., Vera, M., & Robles, R. (1993). Choice of mental health provider among Puerto Rican poor. Unpublished manuscript. Murphy, M. J. (1980). An Investigation of Facets of Client Satisfaction with Out-patient Mental Health Services, Dissertation Abstract International,

40(8),

4749.

G. (1994). Consumers

Negron,

Testing

across

satisfaction

with mental health services.

and health settings. Ann Arbor, MI: University Microfilms INC. Pascoe, G. C. (1983). Patient satisfaction in primary health care: A literature review and analysis. Evaluation and Program Planning, 6(3/4),

satisfaction

health sectors

185-210.

Pascoe G. C., & Attkisson, C. C. (1983). The evaluation ranking scale: A new methodology for assessing satisfaction. Evaluation and Program Planning,

6(3/4),

3355348.

Roberts, R. E., & Attkisson, C. C. (1983). Assessing client satisfaction among Hispanics. Evaluation and Program Planning, 6(3/4), 401413.

Rogler,

L. H., Malgady,

Mental

Health:

R. G., & Rodriguez,

A Framework

for Research.

0. (1989). Hispanics and Malabar, FL: Robert

E. Krieger Publishing Co., INC. Rogler, L. H., Cortes, D. E., & Malgady, R. G. (1991). Acculturation and mental health status among Hispanics: Convergence and new directory for research. American Psychologist, 46(6), 585-597. Rogler, L. H., Malgady, R. G., Costatino, G., & Blumenthal, R. (1987). What do culturally sensitive mental health services mean?: The case of Hispanics. American Psychologist, 42(6), 5655570. Ruggeri, M. (1994). Patients’ and relatives’ satisfaction with psychiatric

and Program

Planning

21 (1998)

81-92

services: The state of the art of its measurements. and Psychiatry

Epidemiology,

29(5),

Social Psychiatry

212-227.

Ruggeri, M., & Dall’Agnola, R. (1993). The development and use of the Verona Expectations for Care Scale (VECS) and the Verona Service Satisfaction Scale (VSSS) for measuring expectations and satisfaction with community-based psychiatric services in patients, relatives and professionals. Psychology and Medicine, 23, 5 1 l-523. Scheffler, R. M., & Miller, A. B. (1989). Demand analysis of mental health service use among ethnic subpopulations. Inquiry, 26, 2022 215.

Sishta, S. K., Rinco, S., &Sullivan, J. C. F. (1986). Clients’ satisfaction survey in a psychiatric inpatient population attached to a general hospital. Canadian Journal of Psychiatry, 31, 1123-I 128. Slater, V., Linn, M. W., & Harris, R. A. (1982). Outpatient evaluation of mental health care. Southern Medical Journal, 74(10), 1217-1219. Sue, D. W. (1981). Counseling the Culturally Different: Theory and Practice. Third Edition. New York: Wiley & Sons, Inc. Tanner, B. A. (1982). A multi-dimensional client satisfaction instrument. Evaluation and Program Planning, 5, 161-167. Vera, M., Robles, R., & Alegria, M. (1988). Sociodemographic, cultural and interpersonal risk factors as indicators of mental health needs. Puerto Rico Health Sciences

Journal,

7, 21-25.

Vera, M., Alegria, M., Freeman, D., Robles, R., Rios, R., & Rios, C. (1991). Depressive symptoms among Puerto Ricans: Island poor compared with residents of New York city areas. American Journal of Epidemiology,

134, 502-510.

Warheit, G. T., Vega, W., Schimizu, D., & Meinhardt, K. (1982). Interpersonal coping networks and mental health among four raceethnic groups. Journal of Community Psychology, 10, 3 12-324. Ware, J. E., Davies-Avery, A., & Stewart, A. L. (1978). The measurement and meaning of patient satisfaction. Health and Medical Care Services Review, 1, l-15. Ware, J. E., Jr., Snyder, M. K., Wright, W. R., & Davies, A. R. (1983). Defining and measuring patient satisfaction with medical care. Evaluation

and Program

Planning,

6(3/4),

2477264.