- Email: [email protected]
Comparison of generic and disease-specific measures of quality of life in first-episode psychosis
Journal of Psychiatric Research 47 (2013) 1403e1408
Contents lists available at SciVerse ScienceDirect
Journal of Psychiatric Research journal homepage: www.elsevier.com/locate/psychires
Comparison of generic and disease-specific measures of quality of life in first-episode psychosis Laoise Renwick a, *, John Lyne a, Elizabeth Owens a, Brian O’ Donoghue a, Kevin Madigan a, Niall Turner a, Jonathan Drennan c, Ann Sheridan c, Anthony Kinsella a, Mary Clarke a, b, Eadbhard O’ Callaghan a, b,1 a b c
DETECT Early Psychosis Service, Block 5, Blackrock Business Park, Blackrock, Dublin, Ireland School of Medicine, University College Dublin, Ireland School of Nursing, Midwifery & Health Systems, University College Dublin, Ireland
a r t i c l e i n f o
a b s t r a c t
Article history: Received 18 May 2012 Received in revised form 28 April 2013 Accepted 3 June 2013
Background: Quality of life (QOL) is now recognised as an important measure of outcome that could potentially influence clinical decision-making for those with a first-episode psychosis (FEP). A number of QOL instruments are available however; many differ in their conceptual orientation which may have serious implications for the outcome of QOL studies, interpretation of findings and clinical utility. We aimed to compare two commonly used tools representing both generic and disease-specific constructs to examine whether both tools appraise the same underlying QOL traits and also whether disease-specific tools retain their psychometric properties when used in FEP groups. Methods: We assessed 159 consecutive individuals presenting with FEP in a defined catchment area with two commonly used QOL tools and examined the findings using the multi-trait multi-method matrix. Results: Similarly named domains of QOL between both tools (Psychological Wellbeing, Physical Health, Social Relations) showed good convergent validity using confirmatory factor analysis. However, discriminant validity was not established given that domains loading onto their indicated latent factors were more strongly correlated with their non-corresponding latent factors. Conclusions: A major consideration in undertaking the present study was to assess the extent to which the outcome of QOL studies in FEP were valid and that systematic error did not provide another plausible explanation for findings. Establishing convergent validity demonstrates that either tool could be used satisfactorily to measure the QOL construct identified however; we did not establish discriminant validity. Doing so would have demonstrated that QOL domains are substantively different in that they contain some unique piece of information determining clinical utility. These findings are important for our understanding of multi-dimensional models of QOL. Ó 2013 Elsevier Ltd. All rights reserved.
Keywords: Quality of life First-episode psychosis Generic measures Disease-specific measures
1. Introduction Quality of life (QOL) is now recognized as an important outcome measure, used in the evaluation of both pharmacological and psychosocial treatments during a first episode of psychosis (FEP) (Thorup et al., 2010; Ho et al., 2000). The QOL concept aspires to take a step beyond traditional indicators of outcome, such as hospital recidivism and symptom remission, by reflecting patient-level concerns about their well-being that are recognized as increasingly
* Corresponding author. Tel.: þ353 12791700; fax: þ353 12791799. E-mail address: [email protected] (L. Renwick). 1 Deceased. 0022-3956/$ e see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.jpsychires.2013.06.001
important in clinical decision-making (Cotton et al., 2010; Malla and Payne, 2005). Indeed, the widespread availability of QOL instruments suggests they are popular in both research and in clinical use. Even still, disagreement among experts persists regarding a universal definition of QOL, the perspective from which it should be viewed and its constituent parts. Consequently, measuring clinically meaningful improvement in the quality of patient’s lives is hindered by the lack of consensus on a gold standard approach. Instruments differ in their orientation towards subjective or objective perspectives of QOL (Malla and Payne, 2005), whether QOL comprises a unitary measure or multiple life domains (Awad and Voruganti, 2000) or whether it should reflect the life domains of generic populations or be specific to the influence of specific illnesses on the quality of people’s lives (Melle et al., 2005).
1404
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
As a result, it seems possible that the use of a particular QOL measure may have serious implications for the outcome and tools oriented towards different perspectives may be non-equivalent or biased (Shadish et al., 2002). Inaccurately estimating the quality of people’s lives makes comparison between studies challenging and importantly, has wider implications for the interpretation and application of findings in clinical practice. Multi-trait multi-method matrices (MMTM) are routinely used to evaluate construct validity and the extent to which reliable variance on a measure is due to the method of assessment (Eid et al., 2009; Podsakoff et al., 2003; Campbell and Fiske, 1959). This method requires an examination of all possible correlations between two or more traits that are measured by two or more measures (Schmitt, 1978). Ideally, tests will show that similar traits that are measured by different methods will be correlated strongly and also that different traits measured by either method will not correlate significantly (Walters, 2009). The aim of the present study was to establish and compare the psychometric properties of two frequently used QOL tools in FEP, namely the WHOQOL-Bref (The Whoqol Group, 1998) and the WQLI-Client Version (Becker et al., 1993). Although both instruments endorse the QOL construct as the subjective appraisal of participant with multiple dimensions of life, the former is a generic instrument and the latter is a disease specific instrument. In theory, there should be substantive differences between each tool due to this conceptual distinction in addition to similarities such as, the ability of multidimensional models of QOL to derive clinically relevant results (Ruggeri et al., 2005; Sainfort et al., 1996). A major problem noted with respect to analysing Campbell & Fiske’s criteria for convergent and discriminant validity is the subjectivity in visually inspecting matrices to arrive at valid conclusions (Schmitt, 1978). Thus, several analytic procedures have been utilized with confirmatory factor analysis believed to be most advantageous (Hadorn and Hays, 1991; Kenny and Kashy, 1992). A key aspect of this study was to employ the criteria of Campbell & Fiske in examining the convergent and discriminant validity of QOL tools using confirmatory factor analytic techniques. 2. Methods 2.1. Subjects Between February 2009 and April 2011, we assessed 159 consecutive individuals presenting with FEP aged between 17 and 65 years within a geographically defined catchment area (population approx. 375,000). Participants were excluded if they had prior treatment with anti-psychotic medication for more than 30 days, had an existing learning disability, or had psychosis deemed to be the result of a general medical condition. Individuals satisfying DSM-IV criteria for psychotic disorder diagnoses, primary psychotic disorder (n ¼ 66) or primary mood disorder (n ¼ 20), who reported on their QOL were included in this study. 2.2. Measures Participants’ diagnoses were established using the Structured Clinical Interview for DSM-IV (Association, 2000) and comprised those with both affective and non-affective psychoses at first treatment for psychotic illness. Having a first episode of psychosis referred to patients who were largely untreated within secondary services, however, there may be unsuccessful attempts at gaining entry or attempting to engage a person in treatment. Given this, if they had been prescribed pharmacological treatment for a psychotic illness by a primary care physician for more than 30 days prior to presentation for treatment, they were deemed not suitable
for the study. Onset was denoted by the first noted psychotic symptoms (reality distortion, disorganised speech) by either retrospective assessment with the participant, family or prior healthcare records. Both pharmacological and psychosocial treatments were offered at inception into the clinical service denoting adequate treatment availability as the offset of psychosis. As such, we defined first episode psychosis as the first presentation with psychotic symptoms to secondary mental health services where an adequate trial of pharmacological treatment has not been administered prior to presentation. Given that there are several difficulties in establishing the boundaries of first-episode psychosis and in attempting to ensure that participants are relatively homogenous and representative of the sample from which they were drawn, these operational definitions were utilised in conjunction with weekly consensus meetings chaired by the Professor of Psychiatry or designated equivalent. QOL was assessed using the WHOQOL-Bref and WQLI-Client which are described in the section below. 2.3. Description of QOL instruments Both QOL instruments have been detailed elsewhere and have reported good reliability and validity (Skevington et al., 2004; Becker et al., 1993). See Table 1 for a descriptive comparison of QOL definitions, constructs and operationalisation in both instruments. 2.4. Procedures Data were collected as part of a larger observational study determining outcomes of an epidemiological cohort of participants with FEP. The study raters (post-membership registrars in psychiatry and clinical nurse specialists in psychosis) received training in the use of instruments and were subject to inter-rater reliability testing. Concordance in diagnosing patients with the SCID interview was minimum 90%. Assessments typically commenced within 48 h of receipt of referral for assessment and treatment of FEP, and were conducted sequentially with interviewer assessments first, then self-reported assessments. Self-report assessments were administered after a level of clinical stability had been achieved. Informed consent and ascent was obtained at entry into the study and ethical approval was granted by the St John of God Hospitaller Services Provincial Ethics Committee. 2.5. Statistical analysis MMTM was used to test hypotheses regarding convergent and discriminant validity (Campbell and Fiske, 1959) by assessing variance due to measurement method. As before, using this approach, the domains of QOL are considered traits and the instruments are considered methods and when combined allow for consideration of validity. Convergent validity, defined as the degree of shared variance between indicators of latent constructs, is established by examining the validity diagonals and the columns and rows of the heterotrait-heteromethod triangles. Correlations in the validity diagonal should be significantly different from zero and should be higher than the correlation coefficients in the columns and rows of the corresponding triangles thus indicating a level of agreement between rating similar concepts using different methods (Campbell and Fiske, 1959). In this study, the validity diagonal refers to psychological well-being, physical health and social relations. Conversely, discriminant validity, defined as the extent to which a construct is truly distinct from other constructs (Hair et al., 2010), is established by ascertaining correlation coefficients within the heterotrait-heteromethod triangles show a pattern of inter-
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
1405
Table 1 Descriptive comparison of QOL tools. WQLI-Client version
WHOQOL-Bref
QOL definition
‘a person’s sense of well-being that stems from satisfaction or dissatisfaction with the areas of life that are important to him/her.’
Conceptualisation
Complex, multi-dimensional construct incorporating objective indices and subjective evaluations of same Disease-specific 8 Domains 2 Overall QOL Domains (1 weighted for importance) 117 items General Satisfaction Occupation Psychological Well-being Physical Health Symptoms/Outlook Social Relations Money Activities of Daily Living Goal Attainment 20e30 min Self-administered Four weeks Items within domains rescaled and aggregated to produce domains scores (3 to þ3, positive scoring indicating better QOL)
‘individuals perceptions of their position in life in the context of their culture and value systems in which they live and in relation to their goals, expectations, standards and concerns’ Subjective satisfaction of several life domains that are cross-culturally sensitive Generic 4 Domains 1 Overall QOL Domain (summation of QOL uniscale and satisfaction with health status) 26 items Physical Health Psychological Social Relationships Environment
Orientation No of domains and items
Domains of QOL
Completion time & type of questionnaire Frame of reference Scoring
relationships that are weaker than those for traits that are considered to be similar. The heterotrait-heteromethod triangles contain all other QOL domains assessed by either instrument that should display a pattern of relationships that discriminate between traits (QOL domains). Univariate and bivariate data analyses were conducted using PASW 20. Confirmatory factor analysis was conducted to test the hypothesised structure of comparative QOL methods using maximum likelihood estimation in AMOS 20. The proposed model for examining MMTM, comprising latent variables representing both methods and traits, resulted in an unidentified model or ‘Heywood Case’ where the model failed to converge (Kenny and Kashy, 1992; Blunch, 2013). A series of models were therefore hypothesised to examine three traits measured by two separate methods (Eid et al., 2003). Model 1 to 3 were single-group MMTM designs with each method observed variable loading onto its respective latent factor constituting an attempt to establish convergent validity e.g. psychological well-being in each tool was an indicator (observed variable) of the latent variable psychological well-being. Discriminant validity was determined by replications of this model with successive iterations loading the noncorresponding QOL domains onto each of the hypothesised traits i.e. latent factors. The final model contained no variables that did not correspond with the three latent QOL factors identified in both tools. A necessity prior to model specification and identification and to ensure an unbiased dataset (Schumacker and Lomax, 2004), missing values were imputed having established these data were missing completely at random (MCAR X2 ¼ 104.48, df ¼ 102, p ¼ .413). The fully observed dataset in this instance accounted for 88% of the data thus few imputations were necessary. There are several goodness of fit indices available for each model but commonly reported indices include Chi-square (X2), Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA) and Akaike Information Criterion (AIC), all of which provide global fit measures (Schreiber et al., 2006). To ascertain the reliability and validity, the output also yields information on Construct Reliability (CR) with a value above .7 indicating scale reliability. Estimates of Average Variance Extracted (AVE) and
10e15 min Self-administered Two weeks Mean score of items within each domain produces a domain score (0e100, higher scores indicate greater satisfaction in domain being evaluated)
Maximum and Average Shared Squared Variance (MVA & ASV) are computed to establish convergent and discriminant validity. Evidence of convergent validity is established when CR is greater than AVE and the AVE is above .5, indicating that observed variables within a latent factor explain an acceptable average variance. Evidence of discriminant validity is established when both MSV and ASV are less than AVE, indicating that the latent factor is explained by hypothesised observed variables better than observed variables hypothesised to relate to another latent factor. 3. Results At study entry, there were completed clinician interviews of clinical status for 159 participants and a subgroup had complete information on QOL; WQLI (n ¼ 86) and WHOQOL-Bref (n ¼ 68). There were 66 participants with ratings of QOL on both generic and disease-specific measures comprising the complete sample analysed here and there were no differences in sample characteristics between respondents and non-respondents (Table 2). 3.1. Construct reliability As a measure of internal consistency, analysis provided two separate sets of reliability coefficients for QOL domains measured by each method and latent constructs; Cronbach’s f in MMTM and CR in CFA. Reliabilities in the former set of estimates for the WHOQOL domain scores ranged from .76 to .89 and were between .73 and .85 for the WQLI subscales. Estimates above .7 indicate an acceptable level of internal consistency (Cronbach, 1951). The CR values for each of the latent constructs in the four CFA models are all within acceptable limits (Hair et al., 2010). 3.2. Convergent and discriminant validity As before, convergent validity is derived from the AVE of the indicators of a specific construct. In Model 1 through 3, the AVE is below the recommended level of .5 for each of the latent constructs when additional QOL domains are loaded onto the respective factor.
1406
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
Table 2 Sample characteristics.
Table 4 Goodness of fit indices of multi-method models of QOL. Mean (SD)
Age at presentation
Models
35.1 (11.9) n, (%)
Marital status Not married Married Employed Yes No Diagnosis Schizophrenia spectrum Depression with psychosis Mania with psychosis Admission status In-patient Out-patient Involuntary admission Voluntary admission Medication prescribed (n [ 63) Yes No Administered > 2/52 (n [ 62) Yes No Global assessment of functioning Duration of untreated psychosis (months) Length of unemployment (months, n [ 57) Positive symptoms (SAPS global score) Negative symptoms (SANS global score) Depressive symptoms (CDSS total score)
PW PH SR ALL
49 (74) 17 (26)
50 (76) 7 (10) 9 (14)
df
p
CFI
RMSEA [90%CI]
AIC
94.54 96.27 95.65 8.60
51 51 51 6
.001 .001 .001 .197
.90 .90 .90 8.60
.11 .12 .11 .08
172.54 174.27 173.65 50.60
[.077e.148] [.079e.150] [.078e.149] [.000e.191]
latent factor. There is one exception in Model 4 where the average squared variance is not greater than the AVE but the maximum shared variance remains above the AVE thus discriminant validity cannot be established.
(62) (38) (39) (61)
51 (77) 12 (18) 9 53 32.7 26.6 57.9 6.9 4.5 5.0
X2
Note 2: SR ¼ Social Relations; PW ¼ Psychological Well-being; PH ¼ Physical Health; Model 1 Non-corresponding QOL domains loaded onto PW; Model 2 noncorresponding QOL domains loaded onto PH; Model 3 non-corresponding QOL domains loaded onto SR.
24 (36) 42 (64)
41 25 16 25
Fit indices
4. Discussion
(14) (80) (11.3) (72.7) (84.6) (2.8) (4.4) (6.0)
Conversely, when the additional domains are excluded the AVE is within acceptable limits thus establishing convergent validity for each of the traits measured by different methods (Tables 3 and 4). As observed in the MMTM, the correlations between corresponding QOL domains measuring the same hypothesized traits are not greater than correlations between non-corresponding traits measured by any method. This indicates that discriminant validity has not been established, a finding reproduced in confirmatory factor analysis. Table 5 details test statistics to establish discriminant validity. As can be seen, in Models 1 through 4, the square root of the AVE is less than one the absolute correlations with other factors indicating that domains loading onto their indicated latent factors are more strongly correlated with non-corresponding latent factors. In all Models the AVE is less than the maximum and average square root of the shared variance indicating variance could be further explained by domains not corresponding to their identified
A major consideration in undertaking the present study was to assess the extent to which the outcome of QOL studies in FEP may be partially dependent upon the instrument that is chosen for the task. As such, the study was conducted primarily to assess whether any systematic measurement error provides a plausible alternative explanation for observed relationships in QOL studies in FEP to date. In particular, we aimed to compare two commonly used tools representing both generic and disease-specific constructs to examine whether both tools appraise the same underlying QOL traits and also whether disease-specific tools retain their psychometric properties when used in FEP groups (Melle et al., 2005). A secondary aim was to establish whether scores on QOL domains in either tool were substantively different indicating that, although related by comprising an element of the QOL construct, each would contain some unique piece of information that would produce clinically useful data upon which to base clinical judgements. On the first point, we examined the convergent validity of three corresponding domains in both tools. As a precursory measure, examining the internal consistency of all QOL domains was necessary. With the exception of the occupation domain, coefficients in all domains were above the advised .7 (Cronbach, 1951) and were comparable with previously reported data (Diaz et al., 1999). Weaker internal consistency in the occupation domain could possibly be explained by summing fewer items than other domains but could also be explained by divergence between
Table 3 Multimethod multitrait correlation matrix and Cronbach’s alpha reliability coefficients for the WHOQOL-Bref and the WQLI-Client. WHOQOL-Bref
WHOQOL-Bref
WQLI-Client
PW PH SR EN GS OC PW PH SR SX M ADL
WQLI-Client
PW
PH
SR
EN
GS
OC
PW
PH
SR
SX
M
ADL
.898 .751 .747 .628 .647 .521 .653 .429 .630 .626 .465 .494
.795 .678 .607 .567 .551 .574 .532 .341 .341 .527 .526
.767 .585 .760 .472 .535 .395 .709 .470 .375 .376
.862 .662 .394 .440 .373 .460 .314 .524 .251
.851 .453 .326 .272 .606 .414 .468 .245
.588 .481 .348 .345 .381 .433 .390
.756 .493 .404 .655 .298 .373
.794 .312 .393 .298 .470
.732 .380 .298 .272
.740 .262 .414
.770 .269
797
Note 1: PW ¼ Psychological Well-being; PH ¼ Physical Health; SR ¼ Social Relations; EN ¼ Environment; OC ¼ Occupation; SX ¼ Symptoms; M ¼ Money; ADL ¼ Activities of Daily Living. Correlation coefficients in bold italics indicate stronger relationships for non-corresponding traits than corresponding traits thus discriminant validity has not been established.
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408 Table 5 Convergent and discriminant validity statistics of multi-method models of QOL. Models Model SR PW PH Model SR PW PH Model SR PW PH Model SR PW PH
CR
AVE
MSV
ASV
.838 .871 .726
.722 .466 .581
.861 .861 .828
.754 .845 .737
.838 .807 .857
.722 .680 .434
.817 .906 .906
.811 .858 .862
.873 .808 .732
.471 .682 .590
.867 .867 .769
.789 .818 .740
.838 .810 .737
.722 .685 .597
.794 .794 .741
.704 .768 .678
1
2
3
4
Note 3: SR ¼ Social Relations; PW ¼ Psychological Well-being; PH ¼ Physical Health; Model 1 Non-corresponding QOL domains loaded onto PW; Model 2 noncorresponding QOL domains loaded onto PH; Model 3 non-corresponding QOL domains loaded onto SR.
objective and subjective QOL (Melle et al., 2005) and patient’s satisfaction being less consistent with their level of participation in activities. Nonetheless, it can be deduced that there is substantive evidence of reasonable convergent validity among each of the three QOL domains with corresponding measures and that either method could be used to assess patients subjective QOL in these areas. On the latter point, this study also aimed to establish discriminant validity between multiple QOL domains in each tool. Indeed, multi-dimensional models of QOL have been accepted as necessary (Testa, 2000) and tested as clinically valid by ascertaining logical clinical relationships between QOL domains and clinical variables given their usefulness in planning interventions and assessing a comprehensive range of patient outcomes. However, demonstrating that QOL domains can, to a degree, discriminate between each other is also crucial for establishing the clinical validity of multi-dimensional models of QOL (Katschnig et al., 2006). This study did not confirm that QOL domains in either tool were satisfactorily able to distinguish from each other. Specifically, some domains displayed stronger associations with non-corresponding domains than with their corresponding domains. For example, the social relations domains of the WHOQOL was strongly correlated with its counterpart in the WQLI but was more strongly correlated with the general satisfaction subscale of the WQLI than its counterpart in the WQLI. To our knowledge, this is the first study to assess method variance in QOL assessments in FEP. Research has primarily focused on programme evaluation (Addington et al., 2003; Cotton et al., 2010; Larsen et al., 2011; Malla et al., 2001) and prospectively assessing the determinants of QOL (Gorna et al., 2008; Browne et al., 2000; Malla et al., 2004; Whitty et al., 2004) and have produced inconsistent findings (Renwick et al., 2012; Malla and Payne, 2005). Specifically, comparison across studies has been difficult due to inconsistent findings between studies, although within studies the finding that general psychopathology has been correlated with QOL domains without distinction has been replicated with more consistency (Malla and Payne, 2005; Renwick et al., 2011). Whilst inconsistencies and consistencies may largely be due to methodological nuances between studies, the prospect of utilizing different conceptualizations of QOL that are ill-equipped to provide clinically meaningful information must be considered a possible explanation for these findings. In terms of advancing research on QOL in FEP, Campbell and Fiske (1959) advise that examination of the MMTM will indicate whether methods should be discarded or replaced or whether
1407
concepts should be sharpened in definition to produce a more clinically relevant method of measure a given concept. This is proposed when convergent validity has not been established, however, no similar recommendation has been made to deal with discriminant validity. Hammond et al. (1986) suggest an approach to assessing criterion-related clinical importance, as an extension of the Campbell & Fiske model, to combine convergent and discriminant validity in MMTM with evaluation of differential prediction of clinical variables thus further establishing clinical validity. This would extend this work and assist in establishing much-needed, clinically valid conceptualizations of QOL in the FEP group to adequately assess the psychosocial impact of FEP on QOL. There are obvious limitations to this study, namely, high attrition rates and the use of cross-sectional data which limits the inferences that can be made. Nonetheless, the results of this study indicate that the usefulness and clinical value of multi-dimensional assessments of QOL for FEP merits more detailed consideration. Whilst not detracting from the importance of assessing QOL in FEP, instruments used during the acute phase of the illness may be ill equipped to provide precise information about multiple domains of QOL for targeted interventions. A further consideration is the benefits of QOL assessment for programme evaluation and clinical decision-making in the absence of concurrence on the domains that constitute QOL and an adequate concept that is fit for purpose. Contributors Laoise Renwick is the first author who initiated the study of quality of life in firat episode psychosis and was responsible writing the manuscript. Brian O’Donoghue, Elizabeth Owens, John Lyne, Niall Turner and Kevin Madigan were responsible for data collection, reviewing the manuscript and contributing to ideas within the manuscript. Anthony Kinsella provided statistical advice on the research. Jonathan Drennan and Ann Sheridan supervised the research on quality of life in first-episode psychosis and Mary Clarke is responsible for the design of the DETECT project research. Eadbhard O’Callaghan was the founder of the DETECT project and supervised the research in DETECT. All authors have contributed to and approved the final manuscript. Statement of funding The early psychosis service is funded by the Health Service Executive and formerly St John of God Hospitaller Services. The first author is funded by a Nursing and Midwifery fellowship from the Health Research Board of Ireland. The authors have no ties with industry. Conflict of interest The authors have not ties with industry. Acknowledgements The authors thank all of the people, both clients and clinicians, who participated in this study for generously giving their time. References Addington J, Young J, Addington D. Social outcome in early psychosis. Psychological Medicine 2003;33:1119e24. Association AP. Diagnostic and statistical manual of mental disorders DSM-IV-TR. Virginia: American Psychiatric Association; 2000. Awad AG, Voruganti LN. Intervention research in psychosis: issues related to the assessment of quality of life. Schizophrenia Bulletin 2000;26:557e64. Becker M, Diamond R, Sainfort F. A new patient focused index for measuring quality of life in persons with severe and persistent mental illness. Quality of Life Research 1993;2:239e51.
1408
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
Blunch NJ. Introduction to structural equation modeling using IBM SPSS statistics and AMOS. London: Sage Publications; 2013. Browne S, Clarke M, Gervin M, Waddington JL, Larkin C, O’Callaghan E. Determinants of quality of life at first presentation with schizophrenia. British Journal of Psychiatry 2000;176:173e6. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin 1959;56:81e105. Cotton SM, Gleeson J, Alvarez-Jimenez M, Mcgorry PD. Quality of life in patients who have remitted from their first episode of psychosis. Schizophrenia Research 2010;117:283. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297e334. Diaz P, Mercier C, Hachey R, Caron J, Boyer G. An evaluation of psychometric properties of the client’s questionnaire of the Wisconsin Quality of Life IndexCanadian version (CaW-QLI). Quality of Life Research 1999;8:509e14. Eid M, Geiser C, Nussbeck FW. Multitrait-multimethod analysis in psychotherapy research: new methodological approaches. Psychotherapy Research 2009;19: 390e6. Eid M, Lischetzke T, Nussbeck FW, Trierweiler LI. Separating trait effects from traitspecific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model. Psychological Methods 2003;8:38e60. Gorna K, Jaracz K, Rybakowski F, Rybakowski J. Determinants of objective and subjective quality of life in first-time-admission schizophrenic patients in Poland: a longitudinal study. Quality of Life Research 2008;17:237e47. Hadorn DC, Hays RD. Multitrait-multimethod analysis of health-related quality-oflife measures. Medical Care 1991;29:829e40. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. New Jersey: Pearson Prentice Hall; 2010. Hammond KR, Hamm RM, Grassia J. Generalizing over conditions by combining the multitrait-multimethod matrix and the representative design of experiments. Psychological Bulletin 1986;100:257. Ho BC, Andreasen NC, Flaum M, Nopoulos P, Miller D. Untreated initial psychosis: its relation to quality of life and symptom remission in first-episode schizophrenia. American Journal of Psychiatry 2000;157:808e15. Katschnig H, Freeman H, Sartorius N. Quality of life in mental disorders. 2nd ed. Chichester: John Wiley & Sons; 2006. Kenny DA, Kashy DA. Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 1992;112:165e72. Larsen TK, Melle I, Auestad B, Haahr U, Joa I, Johannessen JO, et al. Early detection of psychosis: positive effects on 5-year outcome. Psychological Medicine 2011;41: 1461e9. Malla A, Payne J. First-episode psychosis: psychopathology, quality of life, and functional outcome. Schizophrenia Bulletin 2005;31:650e71. Malla AK, Norman RM, Mclean TS, Macdonald C, Mcintosh E, Dean-Lashley F, et al. Determinants of quality of life in first-episode psychosis. Acta Psychiatrica Scandinavica 2004;109:46e54.
Malla AK, Norman RM, Mclean TS, Mcintosh E. Impact of phase-specific treatment of first episode of psychosis on Wisconsin Quality of Life Index (client version). Acta Psychiatrica Scandinavica 2001;103:355e61. Melle I, Friis S, Haahr U, Johannesen JO, Larsen TK, Opjordsmoen S, et al. Measuring quality of life in first-episode psychosis. European Psychiatry 2005;20:474e83. Podsakoff PM, Mackenzie SB, Lee J-Y, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology 2003;88:879e903. Renwick L, Jackson D, Foley S, Owens E, Ramperti N, Behan C, et al. Depression and quality of life in first-episode psychosis. Comprehensive Psychiatry July 2012;53(5):451e5. Ruggeri M, Nosa M, Bonetto C, Cristofalo D, Lasalvia A, Salvi G, et al. Changes and predictors of change in objective and subjective quality of life. The British Journal of Psychiatry 2005;187:121e30. Sainfort F, Becker M, Diamond R. Judgments of quality of life of individuals with severe mental disorders: patient self-report versus provider perspectives. American Journal of Psychiatry 1996;153:497e502. Schmitt N. Path analysis of multitrait-multimethod matrices. Applied Psychological Measurement 1978;2:157e73. Schreiber JB, Nora A, Stage FK, Barlow EA, King J. Reporting structural equation modeling and confirmatory factor analysis results: a review. The Journal of Educational Research 2006;99:323e38. Schumacker RE, Lomax RE. A beginners guide to structural equation modelling. New Jersey: Lawrence Erlbaum Associates; 2004. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin Company; 2002. Skevington SM, Lotfy M, O’Connell KA. The World Health Organization’s WHOQOLBREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Quality of Life Research 2004;13:299e310. Testa MA. Interpretation of quality-of-life outcomes: issues that affect magnitude and meaning. Medical Care 2000;38:II166e74. The Whoqol Group. The World Health Organization quality of life assessment (WHOQOL): development and general psychometric properties. Social Science & Medicine 1998;46:1569e85. Thorup A, Petersen L, Jeppesen P, Nordentoft M. The quality of life among firstepisode psychotic patients in the OPUS trial. Schizophrenia Research 2010;116:27e34. Walters SJ. Quality of life outcomes in clinical trials and health-care evaluation a practical guide to analysis and interpretation. Chichester, U.K.: John Wiley & Sons; 2009. Whitty P, Browne S, Clarke M, Mctigue O, Waddington J, Kinsella T, et al. Systematic comparison of subjective and objective measures of quality of life at 4-year follow-up subsequent to a first episode of psychosis. Journal of Nervous and Mental Disease 2004;192:805e9.
Contents lists available at SciVerse ScienceDirect
Journal of Psychiatric Research journal homepage: www.elsevier.com/locate/psychires
Comparison of generic and disease-specific measures of quality of life in first-episode psychosis Laoise Renwick a, *, John Lyne a, Elizabeth Owens a, Brian O’ Donoghue a, Kevin Madigan a, Niall Turner a, Jonathan Drennan c, Ann Sheridan c, Anthony Kinsella a, Mary Clarke a, b, Eadbhard O’ Callaghan a, b,1 a b c
DETECT Early Psychosis Service, Block 5, Blackrock Business Park, Blackrock, Dublin, Ireland School of Medicine, University College Dublin, Ireland School of Nursing, Midwifery & Health Systems, University College Dublin, Ireland
a r t i c l e i n f o
a b s t r a c t
Article history: Received 18 May 2012 Received in revised form 28 April 2013 Accepted 3 June 2013
Background: Quality of life (QOL) is now recognised as an important measure of outcome that could potentially influence clinical decision-making for those with a first-episode psychosis (FEP). A number of QOL instruments are available however; many differ in their conceptual orientation which may have serious implications for the outcome of QOL studies, interpretation of findings and clinical utility. We aimed to compare two commonly used tools representing both generic and disease-specific constructs to examine whether both tools appraise the same underlying QOL traits and also whether disease-specific tools retain their psychometric properties when used in FEP groups. Methods: We assessed 159 consecutive individuals presenting with FEP in a defined catchment area with two commonly used QOL tools and examined the findings using the multi-trait multi-method matrix. Results: Similarly named domains of QOL between both tools (Psychological Wellbeing, Physical Health, Social Relations) showed good convergent validity using confirmatory factor analysis. However, discriminant validity was not established given that domains loading onto their indicated latent factors were more strongly correlated with their non-corresponding latent factors. Conclusions: A major consideration in undertaking the present study was to assess the extent to which the outcome of QOL studies in FEP were valid and that systematic error did not provide another plausible explanation for findings. Establishing convergent validity demonstrates that either tool could be used satisfactorily to measure the QOL construct identified however; we did not establish discriminant validity. Doing so would have demonstrated that QOL domains are substantively different in that they contain some unique piece of information determining clinical utility. These findings are important for our understanding of multi-dimensional models of QOL. Ó 2013 Elsevier Ltd. All rights reserved.
Keywords: Quality of life First-episode psychosis Generic measures Disease-specific measures
1. Introduction Quality of life (QOL) is now recognized as an important outcome measure, used in the evaluation of both pharmacological and psychosocial treatments during a first episode of psychosis (FEP) (Thorup et al., 2010; Ho et al., 2000). The QOL concept aspires to take a step beyond traditional indicators of outcome, such as hospital recidivism and symptom remission, by reflecting patient-level concerns about their well-being that are recognized as increasingly
* Corresponding author. Tel.: þ353 12791700; fax: þ353 12791799. E-mail address: [email protected] (L. Renwick). 1 Deceased. 0022-3956/$ e see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.jpsychires.2013.06.001
important in clinical decision-making (Cotton et al., 2010; Malla and Payne, 2005). Indeed, the widespread availability of QOL instruments suggests they are popular in both research and in clinical use. Even still, disagreement among experts persists regarding a universal definition of QOL, the perspective from which it should be viewed and its constituent parts. Consequently, measuring clinically meaningful improvement in the quality of patient’s lives is hindered by the lack of consensus on a gold standard approach. Instruments differ in their orientation towards subjective or objective perspectives of QOL (Malla and Payne, 2005), whether QOL comprises a unitary measure or multiple life domains (Awad and Voruganti, 2000) or whether it should reflect the life domains of generic populations or be specific to the influence of specific illnesses on the quality of people’s lives (Melle et al., 2005).
1404
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
As a result, it seems possible that the use of a particular QOL measure may have serious implications for the outcome and tools oriented towards different perspectives may be non-equivalent or biased (Shadish et al., 2002). Inaccurately estimating the quality of people’s lives makes comparison between studies challenging and importantly, has wider implications for the interpretation and application of findings in clinical practice. Multi-trait multi-method matrices (MMTM) are routinely used to evaluate construct validity and the extent to which reliable variance on a measure is due to the method of assessment (Eid et al., 2009; Podsakoff et al., 2003; Campbell and Fiske, 1959). This method requires an examination of all possible correlations between two or more traits that are measured by two or more measures (Schmitt, 1978). Ideally, tests will show that similar traits that are measured by different methods will be correlated strongly and also that different traits measured by either method will not correlate significantly (Walters, 2009). The aim of the present study was to establish and compare the psychometric properties of two frequently used QOL tools in FEP, namely the WHOQOL-Bref (The Whoqol Group, 1998) and the WQLI-Client Version (Becker et al., 1993). Although both instruments endorse the QOL construct as the subjective appraisal of participant with multiple dimensions of life, the former is a generic instrument and the latter is a disease specific instrument. In theory, there should be substantive differences between each tool due to this conceptual distinction in addition to similarities such as, the ability of multidimensional models of QOL to derive clinically relevant results (Ruggeri et al., 2005; Sainfort et al., 1996). A major problem noted with respect to analysing Campbell & Fiske’s criteria for convergent and discriminant validity is the subjectivity in visually inspecting matrices to arrive at valid conclusions (Schmitt, 1978). Thus, several analytic procedures have been utilized with confirmatory factor analysis believed to be most advantageous (Hadorn and Hays, 1991; Kenny and Kashy, 1992). A key aspect of this study was to employ the criteria of Campbell & Fiske in examining the convergent and discriminant validity of QOL tools using confirmatory factor analytic techniques. 2. Methods 2.1. Subjects Between February 2009 and April 2011, we assessed 159 consecutive individuals presenting with FEP aged between 17 and 65 years within a geographically defined catchment area (population approx. 375,000). Participants were excluded if they had prior treatment with anti-psychotic medication for more than 30 days, had an existing learning disability, or had psychosis deemed to be the result of a general medical condition. Individuals satisfying DSM-IV criteria for psychotic disorder diagnoses, primary psychotic disorder (n ¼ 66) or primary mood disorder (n ¼ 20), who reported on their QOL were included in this study. 2.2. Measures Participants’ diagnoses were established using the Structured Clinical Interview for DSM-IV (Association, 2000) and comprised those with both affective and non-affective psychoses at first treatment for psychotic illness. Having a first episode of psychosis referred to patients who were largely untreated within secondary services, however, there may be unsuccessful attempts at gaining entry or attempting to engage a person in treatment. Given this, if they had been prescribed pharmacological treatment for a psychotic illness by a primary care physician for more than 30 days prior to presentation for treatment, they were deemed not suitable
for the study. Onset was denoted by the first noted psychotic symptoms (reality distortion, disorganised speech) by either retrospective assessment with the participant, family or prior healthcare records. Both pharmacological and psychosocial treatments were offered at inception into the clinical service denoting adequate treatment availability as the offset of psychosis. As such, we defined first episode psychosis as the first presentation with psychotic symptoms to secondary mental health services where an adequate trial of pharmacological treatment has not been administered prior to presentation. Given that there are several difficulties in establishing the boundaries of first-episode psychosis and in attempting to ensure that participants are relatively homogenous and representative of the sample from which they were drawn, these operational definitions were utilised in conjunction with weekly consensus meetings chaired by the Professor of Psychiatry or designated equivalent. QOL was assessed using the WHOQOL-Bref and WQLI-Client which are described in the section below. 2.3. Description of QOL instruments Both QOL instruments have been detailed elsewhere and have reported good reliability and validity (Skevington et al., 2004; Becker et al., 1993). See Table 1 for a descriptive comparison of QOL definitions, constructs and operationalisation in both instruments. 2.4. Procedures Data were collected as part of a larger observational study determining outcomes of an epidemiological cohort of participants with FEP. The study raters (post-membership registrars in psychiatry and clinical nurse specialists in psychosis) received training in the use of instruments and were subject to inter-rater reliability testing. Concordance in diagnosing patients with the SCID interview was minimum 90%. Assessments typically commenced within 48 h of receipt of referral for assessment and treatment of FEP, and were conducted sequentially with interviewer assessments first, then self-reported assessments. Self-report assessments were administered after a level of clinical stability had been achieved. Informed consent and ascent was obtained at entry into the study and ethical approval was granted by the St John of God Hospitaller Services Provincial Ethics Committee. 2.5. Statistical analysis MMTM was used to test hypotheses regarding convergent and discriminant validity (Campbell and Fiske, 1959) by assessing variance due to measurement method. As before, using this approach, the domains of QOL are considered traits and the instruments are considered methods and when combined allow for consideration of validity. Convergent validity, defined as the degree of shared variance between indicators of latent constructs, is established by examining the validity diagonals and the columns and rows of the heterotrait-heteromethod triangles. Correlations in the validity diagonal should be significantly different from zero and should be higher than the correlation coefficients in the columns and rows of the corresponding triangles thus indicating a level of agreement between rating similar concepts using different methods (Campbell and Fiske, 1959). In this study, the validity diagonal refers to psychological well-being, physical health and social relations. Conversely, discriminant validity, defined as the extent to which a construct is truly distinct from other constructs (Hair et al., 2010), is established by ascertaining correlation coefficients within the heterotrait-heteromethod triangles show a pattern of inter-
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
1405
Table 1 Descriptive comparison of QOL tools. WQLI-Client version
WHOQOL-Bref
QOL definition
‘a person’s sense of well-being that stems from satisfaction or dissatisfaction with the areas of life that are important to him/her.’
Conceptualisation
Complex, multi-dimensional construct incorporating objective indices and subjective evaluations of same Disease-specific 8 Domains 2 Overall QOL Domains (1 weighted for importance) 117 items General Satisfaction Occupation Psychological Well-being Physical Health Symptoms/Outlook Social Relations Money Activities of Daily Living Goal Attainment 20e30 min Self-administered Four weeks Items within domains rescaled and aggregated to produce domains scores (3 to þ3, positive scoring indicating better QOL)
‘individuals perceptions of their position in life in the context of their culture and value systems in which they live and in relation to their goals, expectations, standards and concerns’ Subjective satisfaction of several life domains that are cross-culturally sensitive Generic 4 Domains 1 Overall QOL Domain (summation of QOL uniscale and satisfaction with health status) 26 items Physical Health Psychological Social Relationships Environment
Orientation No of domains and items
Domains of QOL
Completion time & type of questionnaire Frame of reference Scoring
relationships that are weaker than those for traits that are considered to be similar. The heterotrait-heteromethod triangles contain all other QOL domains assessed by either instrument that should display a pattern of relationships that discriminate between traits (QOL domains). Univariate and bivariate data analyses were conducted using PASW 20. Confirmatory factor analysis was conducted to test the hypothesised structure of comparative QOL methods using maximum likelihood estimation in AMOS 20. The proposed model for examining MMTM, comprising latent variables representing both methods and traits, resulted in an unidentified model or ‘Heywood Case’ where the model failed to converge (Kenny and Kashy, 1992; Blunch, 2013). A series of models were therefore hypothesised to examine three traits measured by two separate methods (Eid et al., 2003). Model 1 to 3 were single-group MMTM designs with each method observed variable loading onto its respective latent factor constituting an attempt to establish convergent validity e.g. psychological well-being in each tool was an indicator (observed variable) of the latent variable psychological well-being. Discriminant validity was determined by replications of this model with successive iterations loading the noncorresponding QOL domains onto each of the hypothesised traits i.e. latent factors. The final model contained no variables that did not correspond with the three latent QOL factors identified in both tools. A necessity prior to model specification and identification and to ensure an unbiased dataset (Schumacker and Lomax, 2004), missing values were imputed having established these data were missing completely at random (MCAR X2 ¼ 104.48, df ¼ 102, p ¼ .413). The fully observed dataset in this instance accounted for 88% of the data thus few imputations were necessary. There are several goodness of fit indices available for each model but commonly reported indices include Chi-square (X2), Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA) and Akaike Information Criterion (AIC), all of which provide global fit measures (Schreiber et al., 2006). To ascertain the reliability and validity, the output also yields information on Construct Reliability (CR) with a value above .7 indicating scale reliability. Estimates of Average Variance Extracted (AVE) and
10e15 min Self-administered Two weeks Mean score of items within each domain produces a domain score (0e100, higher scores indicate greater satisfaction in domain being evaluated)
Maximum and Average Shared Squared Variance (MVA & ASV) are computed to establish convergent and discriminant validity. Evidence of convergent validity is established when CR is greater than AVE and the AVE is above .5, indicating that observed variables within a latent factor explain an acceptable average variance. Evidence of discriminant validity is established when both MSV and ASV are less than AVE, indicating that the latent factor is explained by hypothesised observed variables better than observed variables hypothesised to relate to another latent factor. 3. Results At study entry, there were completed clinician interviews of clinical status for 159 participants and a subgroup had complete information on QOL; WQLI (n ¼ 86) and WHOQOL-Bref (n ¼ 68). There were 66 participants with ratings of QOL on both generic and disease-specific measures comprising the complete sample analysed here and there were no differences in sample characteristics between respondents and non-respondents (Table 2). 3.1. Construct reliability As a measure of internal consistency, analysis provided two separate sets of reliability coefficients for QOL domains measured by each method and latent constructs; Cronbach’s f in MMTM and CR in CFA. Reliabilities in the former set of estimates for the WHOQOL domain scores ranged from .76 to .89 and were between .73 and .85 for the WQLI subscales. Estimates above .7 indicate an acceptable level of internal consistency (Cronbach, 1951). The CR values for each of the latent constructs in the four CFA models are all within acceptable limits (Hair et al., 2010). 3.2. Convergent and discriminant validity As before, convergent validity is derived from the AVE of the indicators of a specific construct. In Model 1 through 3, the AVE is below the recommended level of .5 for each of the latent constructs when additional QOL domains are loaded onto the respective factor.
1406
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
Table 2 Sample characteristics.
Table 4 Goodness of fit indices of multi-method models of QOL. Mean (SD)
Age at presentation
Models
35.1 (11.9) n, (%)
Marital status Not married Married Employed Yes No Diagnosis Schizophrenia spectrum Depression with psychosis Mania with psychosis Admission status In-patient Out-patient Involuntary admission Voluntary admission Medication prescribed (n [ 63) Yes No Administered > 2/52 (n [ 62) Yes No Global assessment of functioning Duration of untreated psychosis (months) Length of unemployment (months, n [ 57) Positive symptoms (SAPS global score) Negative symptoms (SANS global score) Depressive symptoms (CDSS total score)
PW PH SR ALL
49 (74) 17 (26)
50 (76) 7 (10) 9 (14)
df
p
CFI
RMSEA [90%CI]
AIC
94.54 96.27 95.65 8.60
51 51 51 6
.001 .001 .001 .197
.90 .90 .90 8.60
.11 .12 .11 .08
172.54 174.27 173.65 50.60
[.077e.148] [.079e.150] [.078e.149] [.000e.191]
latent factor. There is one exception in Model 4 where the average squared variance is not greater than the AVE but the maximum shared variance remains above the AVE thus discriminant validity cannot be established.
(62) (38) (39) (61)
51 (77) 12 (18) 9 53 32.7 26.6 57.9 6.9 4.5 5.0
X2
Note 2: SR ¼ Social Relations; PW ¼ Psychological Well-being; PH ¼ Physical Health; Model 1 Non-corresponding QOL domains loaded onto PW; Model 2 noncorresponding QOL domains loaded onto PH; Model 3 non-corresponding QOL domains loaded onto SR.
24 (36) 42 (64)
41 25 16 25
Fit indices
4. Discussion
(14) (80) (11.3) (72.7) (84.6) (2.8) (4.4) (6.0)
Conversely, when the additional domains are excluded the AVE is within acceptable limits thus establishing convergent validity for each of the traits measured by different methods (Tables 3 and 4). As observed in the MMTM, the correlations between corresponding QOL domains measuring the same hypothesized traits are not greater than correlations between non-corresponding traits measured by any method. This indicates that discriminant validity has not been established, a finding reproduced in confirmatory factor analysis. Table 5 details test statistics to establish discriminant validity. As can be seen, in Models 1 through 4, the square root of the AVE is less than one the absolute correlations with other factors indicating that domains loading onto their indicated latent factors are more strongly correlated with non-corresponding latent factors. In all Models the AVE is less than the maximum and average square root of the shared variance indicating variance could be further explained by domains not corresponding to their identified
A major consideration in undertaking the present study was to assess the extent to which the outcome of QOL studies in FEP may be partially dependent upon the instrument that is chosen for the task. As such, the study was conducted primarily to assess whether any systematic measurement error provides a plausible alternative explanation for observed relationships in QOL studies in FEP to date. In particular, we aimed to compare two commonly used tools representing both generic and disease-specific constructs to examine whether both tools appraise the same underlying QOL traits and also whether disease-specific tools retain their psychometric properties when used in FEP groups (Melle et al., 2005). A secondary aim was to establish whether scores on QOL domains in either tool were substantively different indicating that, although related by comprising an element of the QOL construct, each would contain some unique piece of information that would produce clinically useful data upon which to base clinical judgements. On the first point, we examined the convergent validity of three corresponding domains in both tools. As a precursory measure, examining the internal consistency of all QOL domains was necessary. With the exception of the occupation domain, coefficients in all domains were above the advised .7 (Cronbach, 1951) and were comparable with previously reported data (Diaz et al., 1999). Weaker internal consistency in the occupation domain could possibly be explained by summing fewer items than other domains but could also be explained by divergence between
Table 3 Multimethod multitrait correlation matrix and Cronbach’s alpha reliability coefficients for the WHOQOL-Bref and the WQLI-Client. WHOQOL-Bref
WHOQOL-Bref
WQLI-Client
PW PH SR EN GS OC PW PH SR SX M ADL
WQLI-Client
PW
PH
SR
EN
GS
OC
PW
PH
SR
SX
M
ADL
.898 .751 .747 .628 .647 .521 .653 .429 .630 .626 .465 .494
.795 .678 .607 .567 .551 .574 .532 .341 .341 .527 .526
.767 .585 .760 .472 .535 .395 .709 .470 .375 .376
.862 .662 .394 .440 .373 .460 .314 .524 .251
.851 .453 .326 .272 .606 .414 .468 .245
.588 .481 .348 .345 .381 .433 .390
.756 .493 .404 .655 .298 .373
.794 .312 .393 .298 .470
.732 .380 .298 .272
.740 .262 .414
.770 .269
797
Note 1: PW ¼ Psychological Well-being; PH ¼ Physical Health; SR ¼ Social Relations; EN ¼ Environment; OC ¼ Occupation; SX ¼ Symptoms; M ¼ Money; ADL ¼ Activities of Daily Living. Correlation coefficients in bold italics indicate stronger relationships for non-corresponding traits than corresponding traits thus discriminant validity has not been established.
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408 Table 5 Convergent and discriminant validity statistics of multi-method models of QOL. Models Model SR PW PH Model SR PW PH Model SR PW PH Model SR PW PH
CR
AVE
MSV
ASV
.838 .871 .726
.722 .466 .581
.861 .861 .828
.754 .845 .737
.838 .807 .857
.722 .680 .434
.817 .906 .906
.811 .858 .862
.873 .808 .732
.471 .682 .590
.867 .867 .769
.789 .818 .740
.838 .810 .737
.722 .685 .597
.794 .794 .741
.704 .768 .678
1
2
3
4
Note 3: SR ¼ Social Relations; PW ¼ Psychological Well-being; PH ¼ Physical Health; Model 1 Non-corresponding QOL domains loaded onto PW; Model 2 noncorresponding QOL domains loaded onto PH; Model 3 non-corresponding QOL domains loaded onto SR.
objective and subjective QOL (Melle et al., 2005) and patient’s satisfaction being less consistent with their level of participation in activities. Nonetheless, it can be deduced that there is substantive evidence of reasonable convergent validity among each of the three QOL domains with corresponding measures and that either method could be used to assess patients subjective QOL in these areas. On the latter point, this study also aimed to establish discriminant validity between multiple QOL domains in each tool. Indeed, multi-dimensional models of QOL have been accepted as necessary (Testa, 2000) and tested as clinically valid by ascertaining logical clinical relationships between QOL domains and clinical variables given their usefulness in planning interventions and assessing a comprehensive range of patient outcomes. However, demonstrating that QOL domains can, to a degree, discriminate between each other is also crucial for establishing the clinical validity of multi-dimensional models of QOL (Katschnig et al., 2006). This study did not confirm that QOL domains in either tool were satisfactorily able to distinguish from each other. Specifically, some domains displayed stronger associations with non-corresponding domains than with their corresponding domains. For example, the social relations domains of the WHOQOL was strongly correlated with its counterpart in the WQLI but was more strongly correlated with the general satisfaction subscale of the WQLI than its counterpart in the WQLI. To our knowledge, this is the first study to assess method variance in QOL assessments in FEP. Research has primarily focused on programme evaluation (Addington et al., 2003; Cotton et al., 2010; Larsen et al., 2011; Malla et al., 2001) and prospectively assessing the determinants of QOL (Gorna et al., 2008; Browne et al., 2000; Malla et al., 2004; Whitty et al., 2004) and have produced inconsistent findings (Renwick et al., 2012; Malla and Payne, 2005). Specifically, comparison across studies has been difficult due to inconsistent findings between studies, although within studies the finding that general psychopathology has been correlated with QOL domains without distinction has been replicated with more consistency (Malla and Payne, 2005; Renwick et al., 2011). Whilst inconsistencies and consistencies may largely be due to methodological nuances between studies, the prospect of utilizing different conceptualizations of QOL that are ill-equipped to provide clinically meaningful information must be considered a possible explanation for these findings. In terms of advancing research on QOL in FEP, Campbell and Fiske (1959) advise that examination of the MMTM will indicate whether methods should be discarded or replaced or whether
1407
concepts should be sharpened in definition to produce a more clinically relevant method of measure a given concept. This is proposed when convergent validity has not been established, however, no similar recommendation has been made to deal with discriminant validity. Hammond et al. (1986) suggest an approach to assessing criterion-related clinical importance, as an extension of the Campbell & Fiske model, to combine convergent and discriminant validity in MMTM with evaluation of differential prediction of clinical variables thus further establishing clinical validity. This would extend this work and assist in establishing much-needed, clinically valid conceptualizations of QOL in the FEP group to adequately assess the psychosocial impact of FEP on QOL. There are obvious limitations to this study, namely, high attrition rates and the use of cross-sectional data which limits the inferences that can be made. Nonetheless, the results of this study indicate that the usefulness and clinical value of multi-dimensional assessments of QOL for FEP merits more detailed consideration. Whilst not detracting from the importance of assessing QOL in FEP, instruments used during the acute phase of the illness may be ill equipped to provide precise information about multiple domains of QOL for targeted interventions. A further consideration is the benefits of QOL assessment for programme evaluation and clinical decision-making in the absence of concurrence on the domains that constitute QOL and an adequate concept that is fit for purpose. Contributors Laoise Renwick is the first author who initiated the study of quality of life in firat episode psychosis and was responsible writing the manuscript. Brian O’Donoghue, Elizabeth Owens, John Lyne, Niall Turner and Kevin Madigan were responsible for data collection, reviewing the manuscript and contributing to ideas within the manuscript. Anthony Kinsella provided statistical advice on the research. Jonathan Drennan and Ann Sheridan supervised the research on quality of life in first-episode psychosis and Mary Clarke is responsible for the design of the DETECT project research. Eadbhard O’Callaghan was the founder of the DETECT project and supervised the research in DETECT. All authors have contributed to and approved the final manuscript. Statement of funding The early psychosis service is funded by the Health Service Executive and formerly St John of God Hospitaller Services. The first author is funded by a Nursing and Midwifery fellowship from the Health Research Board of Ireland. The authors have no ties with industry. Conflict of interest The authors have not ties with industry. Acknowledgements The authors thank all of the people, both clients and clinicians, who participated in this study for generously giving their time. References Addington J, Young J, Addington D. Social outcome in early psychosis. Psychological Medicine 2003;33:1119e24. Association AP. Diagnostic and statistical manual of mental disorders DSM-IV-TR. Virginia: American Psychiatric Association; 2000. Awad AG, Voruganti LN. Intervention research in psychosis: issues related to the assessment of quality of life. Schizophrenia Bulletin 2000;26:557e64. Becker M, Diamond R, Sainfort F. A new patient focused index for measuring quality of life in persons with severe and persistent mental illness. Quality of Life Research 1993;2:239e51.
1408
L. Renwick et al. / Journal of Psychiatric Research 47 (2013) 1403e1408
Blunch NJ. Introduction to structural equation modeling using IBM SPSS statistics and AMOS. London: Sage Publications; 2013. Browne S, Clarke M, Gervin M, Waddington JL, Larkin C, O’Callaghan E. Determinants of quality of life at first presentation with schizophrenia. British Journal of Psychiatry 2000;176:173e6. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin 1959;56:81e105. Cotton SM, Gleeson J, Alvarez-Jimenez M, Mcgorry PD. Quality of life in patients who have remitted from their first episode of psychosis. Schizophrenia Research 2010;117:283. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297e334. Diaz P, Mercier C, Hachey R, Caron J, Boyer G. An evaluation of psychometric properties of the client’s questionnaire of the Wisconsin Quality of Life IndexCanadian version (CaW-QLI). Quality of Life Research 1999;8:509e14. Eid M, Geiser C, Nussbeck FW. Multitrait-multimethod analysis in psychotherapy research: new methodological approaches. Psychotherapy Research 2009;19: 390e6. Eid M, Lischetzke T, Nussbeck FW, Trierweiler LI. Separating trait effects from traitspecific method effects in multitrait-multimethod models: a multiple-indicator CT-C(M-1) model. Psychological Methods 2003;8:38e60. Gorna K, Jaracz K, Rybakowski F, Rybakowski J. Determinants of objective and subjective quality of life in first-time-admission schizophrenic patients in Poland: a longitudinal study. Quality of Life Research 2008;17:237e47. Hadorn DC, Hays RD. Multitrait-multimethod analysis of health-related quality-oflife measures. Medical Care 1991;29:829e40. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. New Jersey: Pearson Prentice Hall; 2010. Hammond KR, Hamm RM, Grassia J. Generalizing over conditions by combining the multitrait-multimethod matrix and the representative design of experiments. Psychological Bulletin 1986;100:257. Ho BC, Andreasen NC, Flaum M, Nopoulos P, Miller D. Untreated initial psychosis: its relation to quality of life and symptom remission in first-episode schizophrenia. American Journal of Psychiatry 2000;157:808e15. Katschnig H, Freeman H, Sartorius N. Quality of life in mental disorders. 2nd ed. Chichester: John Wiley & Sons; 2006. Kenny DA, Kashy DA. Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin 1992;112:165e72. Larsen TK, Melle I, Auestad B, Haahr U, Joa I, Johannessen JO, et al. Early detection of psychosis: positive effects on 5-year outcome. Psychological Medicine 2011;41: 1461e9. Malla A, Payne J. First-episode psychosis: psychopathology, quality of life, and functional outcome. Schizophrenia Bulletin 2005;31:650e71. Malla AK, Norman RM, Mclean TS, Macdonald C, Mcintosh E, Dean-Lashley F, et al. Determinants of quality of life in first-episode psychosis. Acta Psychiatrica Scandinavica 2004;109:46e54.
Malla AK, Norman RM, Mclean TS, Mcintosh E. Impact of phase-specific treatment of first episode of psychosis on Wisconsin Quality of Life Index (client version). Acta Psychiatrica Scandinavica 2001;103:355e61. Melle I, Friis S, Haahr U, Johannesen JO, Larsen TK, Opjordsmoen S, et al. Measuring quality of life in first-episode psychosis. European Psychiatry 2005;20:474e83. Podsakoff PM, Mackenzie SB, Lee J-Y, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology 2003;88:879e903. Renwick L, Jackson D, Foley S, Owens E, Ramperti N, Behan C, et al. Depression and quality of life in first-episode psychosis. Comprehensive Psychiatry July 2012;53(5):451e5. Ruggeri M, Nosa M, Bonetto C, Cristofalo D, Lasalvia A, Salvi G, et al. Changes and predictors of change in objective and subjective quality of life. The British Journal of Psychiatry 2005;187:121e30. Sainfort F, Becker M, Diamond R. Judgments of quality of life of individuals with severe mental disorders: patient self-report versus provider perspectives. American Journal of Psychiatry 1996;153:497e502. Schmitt N. Path analysis of multitrait-multimethod matrices. Applied Psychological Measurement 1978;2:157e73. Schreiber JB, Nora A, Stage FK, Barlow EA, King J. Reporting structural equation modeling and confirmatory factor analysis results: a review. The Journal of Educational Research 2006;99:323e38. Schumacker RE, Lomax RE. A beginners guide to structural equation modelling. New Jersey: Lawrence Erlbaum Associates; 2004. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin Company; 2002. Skevington SM, Lotfy M, O’Connell KA. The World Health Organization’s WHOQOLBREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Quality of Life Research 2004;13:299e310. Testa MA. Interpretation of quality-of-life outcomes: issues that affect magnitude and meaning. Medical Care 2000;38:II166e74. The Whoqol Group. The World Health Organization quality of life assessment (WHOQOL): development and general psychometric properties. Social Science & Medicine 1998;46:1569e85. Thorup A, Petersen L, Jeppesen P, Nordentoft M. The quality of life among firstepisode psychotic patients in the OPUS trial. Schizophrenia Research 2010;116:27e34. Walters SJ. Quality of life outcomes in clinical trials and health-care evaluation a practical guide to analysis and interpretation. Chichester, U.K.: John Wiley & Sons; 2009. Whitty P, Browne S, Clarke M, Mctigue O, Waddington J, Kinsella T, et al. Systematic comparison of subjective and objective measures of quality of life at 4-year follow-up subsequent to a first episode of psychosis. Journal of Nervous and Mental Disease 2004;192:805e9.