- Email: [email protected]
Development of a brief self-report questionnaire for screening putative pre-psychotic states
Schizophrenia Research 143 (2013) 32–37
Contents lists available at SciVerse ScienceDirect
Schizophrenia Research journal homepage: www.elsevier.com/locate/schres
Development of a brief self-report questionnaire for screening putative pre-psychotic states Chen-Chung Liu a, Yin-Jing Tien b, Chun-Houh Chen b, Yen-Nan Chiu a, Yi-Ling Chien a, Ming H. Hsieh a, Chih-Min Liu a, Tzung-Jeng Hwang a, Hai-Gwo Hwu a,⁎ a b
Department of Psychiatry, National Taiwan University Hospital and College of Medicine, National Taiwan University, Taipei, Taiwan Institute of Statistical Science, Academia Sinica, Taipei, Taiwan
a r t i c l e
i n f o
Article history: Received 28 June 2012 Received in revised form 30 August 2012 Accepted 8 October 2012 Available online 24 November 2012 Keywords: Pre-psychotic Schizophrenia Screening Self-report questionnaire
a b s t r a c t Background: Several self-report instruments were developed to capture psychotic prodrome, and were claimed to have good predictive validity. The feasibility of screening is questionable considering the heterogeneity of the targeted populations and the negative ramifications of false positive identification. This study developed a questionnaire using data covering a wide range of clinical characteristics. Methods: One hundred and eleven putative pre-psychotic participants, 129 normal comparison subjects, and 95 non-psychotic psychiatric outpatients completed a 231-item questionnaire comprising a 110-item Wisconsin psychotic prone scale, 74-item schizotypal personality questionnaire, 33-item basic symptoms, and 14-item cognitive symptoms. Items showing the best discriminating power, estimated using chi-square statistics with Bonferroni correction, were extracted to create a brief version. A two-stage cut-off approach emphasizing specific items was applied to maximize sensitivity and specificity. The concurrent validity of the proposed approach was estimated using a ten-fold cross-validation procedure. Results: A 15-item self-report questionnaire was developed. Respondents checking at least eight items, or those checking three to seven items including any of the three referring to feeling stress in crowds, aloofness, and perceptual disturbance, would be considered putatively pre-psychotic with the largest sensitivity+ specificity (0.784+ 0.705=1.489). This cut-off selection was the best estimate by calculating 1000 permutations in the cross-validation procedure. Conclusions: This investigation proposes a different orientation for applying questionnaires to screen putative pre-psychotic states, with less emphasis on attenuated psychotic symptoms and predictive values. Besides providing a handy tool for increasing awareness and referral, the instructions of such a screening questionnaire should be carefully worded. © 2012 Elsevier B.V. All rights reserved.
1. Introduction The prodromal symptoms of schizophrenia are near indistinguishable from other common psychiatric illnesses or stress reactions. The development of newer instruments for clinical assessment has made the predictive validity of prodromal criteria considered acceptable when applied to the help-seeking population (Cannon et al., 2008; Ruhrmann et al., 2010), but screening in the general population is considered infeasible owing to concerns regarding non-specificity and low prediction rates (Jablensky, 2000; Cougnard et al., 2005). Basic symptoms have been reported with high predictive values in a clinical sample (Klosterkotter et al., 2001), but are difficult to apply through self-reporting and in the community (Mojtabai et al., 2003). CASIS, an ⁎ Corresponding author at: Department of Psychiatry, National Taiwan University Hospital and College of Medicine, National Taiwan University, No. 7 Chung Shan S. Rd, Taipei 10043, Taiwan. Tel.: + 886 2 2312 3456x66785; fax: + 886 2 2375 3663. E-mail address: [email protected] (H.-G. Hwu). 0920-9964/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.schres.2012.10.042
acronym meaning cognitive deficits, affective symptoms, social isolation, and school failure, is a non-specific checklist for psychotic prodrome (Cornblatt et al., 2003). The presence of psychotic-like experiences (PLEs) is an important marker for identifying at-risk subjects (Armando et al., 2010; Kelleher and Cannon, 2011); while PLEs are common in the general population (Verdoux and van Os, 2002; Scott et al., 2006), the prevalence of PLEs is highest among adolescents and tends to decrease in reporting frequency after subjects enter their 20s (Rossler et al., 2007). Information campaigns or the provision of special services for suspected psychotic prodrome inevitably draws attention to non-psychotic subjects and will increase the burden for clinical screening (Liu et al., 2009; O'Donoghue et al., 2012). Even mental health professionals from different training backgrounds can be differently inclined in recognizing pre-psychotic phenomena (Liu et al., 2010a). However, for mental health education and for recruiting subjects at-risk of developing psychosis to study and follow-up, it is best to provide an easy, convenient questionnaire to attract people's attention to engage in self evaluation and referral.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
Kline et al. have compared several screening tools, each comprising 12 to 28 items primarily addressing attenuated psychotic symptoms, to demonstrate their good convergent and discriminant validity for attenuated psychosis (Kline et al., 2012). In addition, Kobayashi et al. proposed a 12-item self-report instrument for psychotic prodrome with acceptable sensitivity, specificity, and positive predictive value (Kobayashi et al., 2008), and Muller et al. also suggested a 32-item self-report instrument as a handy screening tool in a large conscript population (Muller et al., 2010). However, whether subjects seek help voluntarily and whether they are distressed strongly affects the predictive power (Loewy et al., 2005, 2007), and caution is needed in applying these instruments to identify at-risk subjects owing to the potential risk of mislabeling and even stigmatizing (Corcoran et al., 2005; Yang et al., 2010). In this paper we present a brief self-report questionnaire, which is derived from a follow-up study on the psychopathological progress of early schizophrenia-like disorder (SOPRES) in Taiwan, for screening putative pre-psychotic states that assigns relatively little emphasis to predictive values for transition to psychosis but covers a broader range of early prodromal symptoms. 2. Methods 2.1. Sample Subjects were participants in the SOPRES study. The rationale and methodology for the SOPRES study have been described elsewhere (Liu et al., 2010b, 2011). Briefly, we employed a low-threshold approach to invite individuals aged between 16 and 32 years old presenting with non-specific symptoms as described using CASIS (Cornblatt et al., 2003) or having newly developed psychotic-like symptoms to an outpatient psychiatric special clinic of the study hospital for thorough assessments (Liu et al., 2010b). All adult participants voluntarily provided written informed consent, and minors gave written assent with informed consent from their parents. This study has been approved by the Institutional Review Board of the study hospital. This investigation conducted clinical assessment using the Thought/ Perception Diagnostic Interview Schedule (TP-DIS) to classify participants into four groups based on level of clinical risk (Liu et al., 2011). The group of first-episode psychosis (FEP) included participants with schizophrenia, schizophreniform disorder, brief psychotic disorder, or schizoaffective disorder meeting the DSM-IV criteria during the previous year. The ultra-high-risk group (UHR) included participants with attenuated psychotic symptoms (APS) or brief limited intermittent psychotic symptoms (BLIPS) (McGorry et al., 2003). The intermediate-risk group (IRG) included participants presenting with odd thinking, feelings, speech, or perceptual experiences, which were not as severe as in the UHR group but met the criteria of schizotypal disorder as specified in the 10th edition of the International Classification of Diseases (ICD-10) without the requirement of two year duration. The marginal-risk group (MRG) included participants with CASIS symptoms that did not meet either the threshold for the IRG or other diagnostic categories. A group of age- and gender-matched healthy volunteers was recruited by public posters, and a group of non-psychotic psychiatric outpatients (NPO) was recruited via referral by their attending psychiatrists. Subjects with IQ below 70, aged younger than 16 years old, with a history of traumatic brain injury, central nervous system illness, a prior psychotic episode lasting for more than one year, or current use of psychoactive stimulants were excluded. The FEP group was excluded from further analysis because the questionnaire is focused on screening pre-psychotic states. 2.2. Instrument At baseline assessment, all participants completed a self-administered 231-item Mandarin version Schizophrenia Proneness Scale comprising a 110-item Wisconsin psychotic prone scale (Chapman et al., 1994), a
33
74-item schizotypal personality questionnaire (Raine, 1991; Chen et al., 1998), 33 items of basic symptoms (Gross, 1989; Klosterkotter et al., 2001), and 14 items of cognitive symptoms (Nakayasu et al., 1999). Participants were encouraged to answer as many questions as possible. All questions were answered using simple “yes” or “no” responses to minimize the response burden. A “yes” represents an affirmative response to the question of a deviant experience. 2.3. Analysis steps and statistical methods Sporadic missing values were treated as negative responses. Each individual item was examined using chi-square test. Items with a p valueb 0.001 based on comparison among five groups were extracted first. Post hoc analysis with Bonferroni correction was used to test the differences between each pair of UHR, IRG, and MRG groups, and none of these items exhibited significant differences in self-reporting frequency; consequently, these three groups were collapsed into a single putative pre-psychotic group (PPP) for further analysis. Furthermore, items with a p valueb 0.05 compared between the PPP and NPO groups were identified to establish a brief self-report questionnaire. Items fulfilling the aforementioned working criteria were scrutinized by two principal clinical investigators (HGH and CCL) to ensure that each item had face validity and that no apparent overlap existed among items extracted from different sets of original questionnaires. With the goal being to maximize sensitivity + specificity, this study employed a two-stage cut-off approach by emphasizing three designated items expected to be more likely to refer to attenuated psychotic symptoms as to increase the sensitivity and specificity. Whether this combination of cut-off and weighted items was the most feasible version for screening was then tested using estimations via a ten-fold leave-one-out cross-validation procedure (Burman, 1989). 3. Results 3.1. Participant demographics A total of 111 putative pre-psychotic participants (PPP, including 41 UHR, 32 IRG, and 38 MRG participants), a normal comparison group (NC) comprising 129 subjects, and 95 non-psychotic psychiatric outpatients (NPO) responded to the original 231-item questionnaires. Table 1 lists the demographic profiles of the respondents. 3.2. Development of a 15-item questionnaire Initially, 14 items fulfilling the selection criteria were derived from the original 231 items, and then two items were removed based on Table 1 Comparison of demographics and frequencies of responses to individual items across three groups.
Age Gender Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Item 14 Item 15
PPP (n = 111)
NPO (n = 95)
NC (n = 129)
Statistics
p
22.5 55 65 61 45 72 59 69 63 69 77 54 48 58 42 45 27
24.15 69 38 31 26 42 33 38 34 38 52 34 14 33 36 27 16
21.33 55 10 10 3 26 14 23 20 26 35 15 10 18 10 9 5
13.807 19.827 71.228 63.072 52.775 49.346 49.943 49.459 44.718 43.983 44.368 39.986 48.126 40.160 37.137 37.963 21.005
b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001
(3.89) (49.5%) (58.6%) (55%) (40.5%) (64.9%) (53.2%) (62.2%) (56.8%) (62.2%) (69.4%) (48.6%) (43.2%) (52.3%) (37.8%) (40.5%) (24.3%)
(4.27) (71.9%) (40%) (32.6%) (27.4%) (44.2%) (34.7%) (40%) (35.8%) (40%) (54.7%) (35.8%) (14.7%) (34.7%) (37.9%) (28.4%) (16.8%)
(3.71) (42.6%) (7.8%) (7.8%) (2.3%) (20.2%) (10.9%) (17.8%) (15.5%) (20.2%) (27.1%) (11.6%) (7.8%) (14%) (7.8%) (7%) (3.9%)
34
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
their apparent similarity to the other two items with smaller p values, yielding a set of 12 items. However, only one of these 12 items referring to mild attenuated psychotic symptoms (Item 6: I sometimes become concerned about the loyalty and trustworthiness of friends or coworkers.),
so the principal investigators manually selected three items which met only the first part of the selection criteria (with very small p values comparing among five groups) to form a 15-item questionnaire. Table 1 listed the distribution of response frequencies among the three groups. This
(C)
(A)
(B)
(D)
Fig. 1. Matrix visualization of 335 participants with 15 items. Left axis: number of participants. Panel (A): The corresponding composition of PPP/NPO/NC within each summary score. Panel (B) The total number of “Yes” of each participant responded to 15 items. Black: cumulative number of Yes; white: cumulative number of No. Panel (C): Jaccard coefficient matrix for 15 items with 335 participants. The tree structure in the right of the Jaccard coefficient matrix represents average linkage hierarchical clustering tree (HCT) for these 15 items. Panel (D): The profiles of response to these 15 items by each participant; each row represents a participant. Items were clustered by average linkage with Jaccard coefficient of these 15 items within 335 participants. Panel (E): The composition of PPP/NPO/NC in each of these 3 groups (nYes ≥8, 3 ≤ nYes b 8, nYes b 3). Group: black: PPP, putative pre-psychotic; gray: NPO, non-psychotic outpatient; white: NC, normal control.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
study used the sum of these 15 variables as a simple score (number of affirmative responses, nYes). This study tries to identify two cut-points to enable the inclusion of putative pre-psychotic subjects not at the expense of including too many subjects from the other two groups. Matrix visualization (Chen, 2002; Chen et al., 2004; Wu et al., 2010) of 335 participants with 15 items is shown in Fig. 1, in which 89 subjects exhibited nYes≥8, including 62, 23, and 4 participants belonging to the PPP, NPO, and NC groups, respectively; furthermore, 138 subjects exhibited nYesb 3, including 16, 27, and 95 participants belonging to the PPP, NPO, and NC groups, respectively; while the remaining 108 participants comprised a mixture of 33, 45 and 30 individuals from the PPP, NPO and NC groups, respectively. Thus it is desirable to seek a solution by adding a condition to this subset (3≤nYesb 8): if the “Yes” response of the participant included any of the three designated items. This study exhausted all the combinations of two cut-points and assigned weightings to one of three items to identify the best solution using this formula: largest sensitivity + specificity. Finally, the higher cut-point was set to nYes ≥ 8 to indicate a positive value for a PPP and the lower cut-point was set to nYes b 3 to indicate a negative value for a PPP, while those participants with nYes ranging from 3 to 7 yet also responding “Yes” to any of the 3 items, I cannot deal with the pressures associated with crowds. (Item 1), I feel I cannot get close to people. (Item 2), Do you hear some sounds, voices, or calls of your name when nobody is around you? (Item 15), are also treated as the PPP group. This procedure yielded the largest value of sensitivity + specificity (0.784 + 0.705 = 1.489) to differentiate PPP from non-PPP (Table 2). The checks of response bias by exploring the composition of PPP/ NPO/NC within each summary score and the composition of PPP/ NPO/NC in each of those 3 groups (nYes ≥ 8, 3 ≤ nYes b 8, nYes b 3) revealed satisfactory results as shown in Column A and Column E of Fig. 1. 3.3. Re-test of sensitivity and specificity Because this formula was not tested on a group of subjects independent of the study population, we tried to estimate its sensitivity and specificity using a ten-fold leave-one-out cross-validation method (Burman, 1989). With the goal of first finding two cut-points and then finding three items if the subject answers “Yes” to any of the three questions as addressed in the previous section, we broke the data into ten subsets of size n/10, conducted training on 9 of the 10 subsets and tested the one remaining subset, and repeated this procedure ten times such that each n/10 samples can be used once as the testing subset, and data was randomly divided into 10 subsets of size n/10 for 100 times. Thus 10 ∗ 100 = 1000 permutations were yielded to test for validations. The two-point cutoff (3, 8) has been selected 931 times, the combination of three items as previously designated has been selected 613 times, and another combination with just 1 item difference from the first combination (Do you often pick up hidden threats or put downs from the words or actions of others? rather than Do you hear some sounds, voices, or calls of your name
35
when nobody is around you?) has been selected 117 times. The estimated averaged sensitivity and specificity of the final screening formula are 0.736 and 0.679, respectively (Table 2), which are slightly lower than the original values (0.784 and 0.705) as expected using the leave-one-out cross-validation method (O'Toole, 2000). 4. Discussions Reduced from the original 231 items, a 15-item self-report questionnaire was developed (Appendix 1). The 15 items fall into four categories (upper right of Fig. 1), including five interpersonal difficulty/social anxiety symptoms (Item 2, 5, 7, 8, and 12), three self-depreciating descriptions (Item 1, 4, and 9), three negative symptoms (Item 3, 10, and 11), and four subthreshold psychotic-like experiences (Item 6, 13, 14, and 15), similar to the four-dimension structure, namely negative schizotypy, positive schizotypy, interpersonal sensitivity, and social isolation/introversion, revealed by analyzing the Structural Interview of Schizotypy (SIS) data in our previous study of non-psychotic relatives of patients with schizophrenia (Lien et al., 2009). This suggests that the screening tool has certain construct validity. The screening tool was tested by comparing the group of putative pre-psychotic participants with a group of non-psychotic psychiatric outpatients and a group of normal controls to examine its discriminant ability. According to the working criteria, participants who had checked at least eight items or who had checked three to seven items, including any one of the three items referring to low tolerance for stress in a crowd, aloofness, and perceptual disturbance, were considered to represent a putative pre-psychotic state. Although the values of sensitivity and specificity are not striking, such a result is reasonable referring to the nature of non-specificity in early prodromal symptoms (Keshavan et al., 2011). Since this study does not attempt to develop a better and more accurate solution for screening ultra-high risk subjects, we feel comfortable using this instrument simply for quick self-evaluation and referral, not specifically emphasizing the high likelihood of the transition to psychosis, but rather addressing a need for clinical attention and potential interest in follow-up based on the relatively raised risk linked to a trajectory of psychosis. Interestingly, prior to performing statistical analysis, this study assumed those subthreshold psychotic-like symptoms would more likely be selected as the three weighted items. However, in the results only the item referring to perceptual disturbance was selected, but not the items referring to suspiciousness, idea of reference and persecutory ideations. This finding might have resulted from a relative low prevalence of attenuated psychotic symptoms in a broader range of prodromal syndromes covered in our study population (Table 1). In fact our proposed screening questionnaire is not focusing on positive prediction of transition rates but with an intention to capture the earlier putative prodromal states, thus making it possible to follow up the formation of psychosis and explore the underlying pathogenesis from the very beginning (Keshavan et al., 2011; Hsieh et al., 2012). Given that the putative pre-psychotic participants cannot be differentiated based on
Table 2 Statistical characteristics using two different cut-off approaches and leave-one-out cross-validation of the two cut-off points approach.
Sensitivity Specificity Overall accuracy Positive predictive value Negative predictive value False positive rate False negative rate Prevalence rate by cut-off
Single cut-off point if nYes ≥ 8
Two cut-off points (3, 8)+(at least one of three items with “Yes” response)a
Leave-one-out cross-validation for two cut-off points+(at least one of three items with “Yes” response)
0.559 0.879 0.773 0.700 0.800 0.121 0.441 0.266
0.784 0.705 0.731 0.569 0.868 0.295 0.216 0.457
0.736 0.679 0.698 0.532 0.839 0.321 0.264 0.458
a These three items are Item 1: I cannot deal with the pressures associated with crowds; Item 2: I feel I cannot get close to people; Item 15: Do you hear some sounds, voices, or calls of your name when nobody is around you?
36
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
their self-report results, the clinical interview can categorize them into three different risk levels. Furthermore, the results of the follow-up study demonstrate that only one-third of patients from the UHR group have progressed to full-blown psychosis, while none of the IRG and MRG subjects converted within 2 years (Liu et al., 2011). Thus a brief self-report questionnaire is not advised to aim at predicting psychosis in the short-term. Nonetheless, this questionnaire can be employed to attract the attention of the lay public to psychotic prodrome, and serves as a handy tool for self-screening and referral, under the premise of carefully wording in instructions while using the questionnaire. The tradeoff between increasing sensitivity at the expense of decreasing specificity has no uniform solution referring to clinical practice (O'Toole, 2000). For example, as demonstrated in Table 2, compared to a simple one cut-off point (nYes≥ 8) approach, the performance of our working criteria can reduce half false negative rates (from 0.441 to 0.216) but the prevalence rate using this approach will increase 72% (from 0.266 to 0.457) and the false positive rate will increase 144% (from 0.121 to 0.295), which implies much heavier demands for providing clinical assessment to subjects identified as positive by screening and potentially much more negative ramification to false positives, such as the risks of being labeled as at risk, stigmatization and self-stigmatization, and unnecessary exposure to the risk of having adverse reactions to early antipsychotic treatment (Corcoran et al., 2005, 2010; Yang et al., 2010). Thus given that our working criteria are statistically the best selection, while applying to real world situations, the choice of different cutoffs still depends on the purpose of screening and the resources available to individual works. The main limitation of this study is not testing the test–retest reliability of the 15-item questionnaire on participants and its validity and reliability on any other reference population. Furthermore, this study did not perform follow-up clinical diagnosis (or no diagnosis) of those who did not convert to psychosis in the PPP group. While acknowledging the limitations of a self-report questionnaire, this study does not attempt to develop an alternative questionnaire with better predicting abilities. Instead, compared to those instruments mainly focused on attenuated psychotic symptoms or psychotic-like experiences to stress its “predictive value” regarding transition rates to frank psychosis (Loewy et al., 2005; Kobayashi et al., 2008; Kelleher et al., 2011; Kline et al., 2012), this proposed questionnaire might be able to capture the early and broadly defined at risk mental status which not only comprises attenuated positive symptoms, but also mood symptoms, negative symptoms and interpersonal difficulties (Keshavan et al., 2011). Creating a brief screening questionnaire with good predictive values is impractical and unnecessary. Indeed a recent large-scale population-based prospective study suggests that self-reported attenuated symptoms are not clinically useful predictors for psychotic disorders later in life (Werbeloff et al., 2012). Thus this brief self-report questionnaire is designed to perform screening for first step risk assessment (Loewy et al., 2011; Ruhrmann et al., 2010). More accurate prediction requires a formal interview and consideration of more clinical variables for risk assessment (Yung et al., 2006; Cannon et al., 2008; Ruhrmann et al., 2010). As the false positive rates can increase rapidly while trying to increase sensitivity, the instructions of a screening tool should be carefully deliberated to avoid misinterpretation of self-report results and subsequent unwanted ramifications. Role of funding source The National Health Research Institutes, Taiwan, provided grants for studying the pre-psychotic state and early psychosis; YJ Tien and CH Chen's work was supported partially by the National Core Facility Program for Biotechnology, Taiwan. Both funding sources had no involvement in study design, data collection, analysis and interpretation, report writing, or the decision to submit the paper for publication.
Contributors CC Liu and HG Hwu reviewed literature and designed the study; CC Liu, YN Chiu, YL Chien, MH Hsieh, CM Liu, TJ Hwang, and HG Hwu contributed to case recruitment and
assessment; YJ Tien and CH Chen did the statistical analyses; all authors contributed to the discussion and interpretation of data; CC Liu wrote the first draft; HG Hwu oversaw the research; all authors contributed to and have approved the final manuscript. Conflict of interest All authors declare no conflicts of interest regarding the work presented in this paper. Acknowledgments This work was supported by the National Health Research Institutes, Taiwan (NHRI-EX95, 96, 97, 98, 99-9511PP) and the National Core Facility Program for Biotechnology, Taiwan (Bioinformatics Consortium of Taiwan, NSC100-2319-B-010-002).
Appendix 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
I cannot deal with the pressures associated with crowds. I feel I cannot get close to people. I feel lethargic whatever I do. I feel mentally insufficient and easily fatigued while thinking or reading. I am mostly quiet when with others. I sometimes become concerned about the loyalty and trustworthiness of friends or coworkers. I tend to keep my feelings to myself. I feel nervous when giving a speech in front of a large group of people. I cannot focus on a task and need to take frequent breaks while working (studying). I always mess up whatever I do. I do not have an expressive and lively way of speaking. I am poor at returning social courtesies and gestures. When you see people talking to each other, do you often wonder if they are talking about you? Do you often pick up hidden threats or put downs from the words or actions of others? Do you hear some sounds, voices, or calls of your name when nobody is around you?
References Armando, M., Nelson, B., Yung, A.R., Ross, M., Birchwood, M., Girardi, P., Fiori Nastro, P., 2010. Psychotic-like experiences and correlation with distress and depressive symptoms in a community sample of adolescents and young adults. Schizophr. Res. 119 (1–3), 258–265. Burman, P., 1989. A comparative study of ordinary cross-validation, v-fold crossvalidation and repeated learning-testing methods. Biometrika 76, 503–514. Cannon, T.D., Cadenhead, K., Cornblatt, B., Woods, S.W., Addington, J., Walker, E., Seidman, L.J., Perkins, D., Tsuang, M., McGlashan, T., Heinssen, R., 2008. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch. Gen. Psychiatry 65 (1), 28–37. Chapman, L.J., Chapman, J.P., Kwapil, T.R., Eckblad, M., Zinser, M.C., 1994. Putatively psychosis-prone subjects 10 years later. J. Abnorm. Psychol. 103 (2), 171–183. Chen, C.H., 2002. Generalized association plots: information visualization via iteratively generated correlation matrices. Statist. Sin. 12, 7–29. Chen, W.J., Liu, S.K., Chang, C.J., Lien, Y.J., Chang, Y.H., Hwu, H.G., 1998. Sustained attention deficit and schizotypal personality features in nonpsychotic relatives of schizophrenic patients. Am. J. Psychiatry 155 (9), 1214–1220. Chen, C.H., Hwu, H.G., Jang, W.J., Kao, C.H., Tien, Y.J., Tzeng, S., Wu, H.M., 2004. Matrix visualization and information mining. Proceedings in Computational Statistics, pp. 85–100. Corcoran, C., Malaspina, D., Hercher, L., 2005. Prodromal interventions for schizophrenia vulnerability: the risks of being “at risk”. Schizophr. Res. 73 (2–3), 173–184. Corcoran, C.M., First, M.B., Cornblatt, B., 2010. The psychosis risk syndrome and its proposed inclusion in the DSM-V: a risk–benefit analysis. Schizophr. Res. 120 (1–3), 16–22. Cornblatt, B.A., Lencz, T., Smith, C.W., Correll, C.U., Auther, A.M., Nakayama, E., 2003. The schizophrenia prodrome revisited: a neurodevelopmental perspective. Schizophr. Bull. 29 (4), 633–651. Cougnard, A., Salmi, L.R., Salamon, R., Verdoux, H., 2005. A decision analysis model to assess the feasibility of the early detection of psychosis in the general population. Schizophr. Res. 74 (1), 27–36. Gross, G., 1989. The ‘basic’ symptoms of schizophrenia. Br. J. Psychiatry (7), 21–25 (Suppl., discussion 37–40). Hsieh, M.H., Shan, J.C., Huang, W.L., Cheng, W.C., Chiu, M.J., Jaw, F.S., Hwu, H.G., Liu, C.C., 2012. Auditory event-related potential of subjects with suspected pre-psychotic state and first-episode psychosis. Schizophr. Res. 140 (1–3), 243–249. Jablensky, A., 2000. Prevalence and incidence of schizophrenia spectrum disorders: implications for prevention. Aust. N. Z. J. Psychiatry 34, S26–S34 (Suppl., discussion S35–28). Kelleher, I., Cannon, M., 2011. Psychotic-like experiences in the general population: characterizing a high-risk group for psychosis. Psychol. Med. 41 (1), 1–6.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37 Kelleher, I., Harley, M., Murtagh, A., Cannon, M., 2011. Are screening instruments valid for psychotic-like experiences? A validation study of screening questions for psychotic-like experiences using in-depth clinical interview. Schizophr. Bull. 37 (2), 362–369. Keshavan, M.S., Delisi, L.E., Seidman, L.J., 2011. Early and broadly defined psychosis risk mental states. Schizophr. Res. 126 (1–3), 1–10. Kline, E., Wilson, C., Ereshefsky, S., Tsuji, T., Schiffman, J., Pitts, S., Reeves, G., 2012. Convergent and discriminant validity of attenuated psychosis screening tools. Schizophr. Res. 134 (1), 49–53. Klosterkotter, J., Hellmich, M., Steinmeyer, E.M., Schultze-Lutter, F., 2001. Diagnosing schizophrenia in the initial prodromal phase. Arch. Gen. Psychiatry 58 (2), 158–164. Kobayashi, H., Nemoto, T., Koshikawa, H., Osono, Y., Yamazawa, R., Murakami, M., Kashima, H., Mizuno, M., 2008. A self-reported instrument for prodromal symptoms of psychosis: testing the clinical validity of the PRIME Screen—Revised (PS-R) in a Japanese population. Schizophr. Res. 106 (2–3), 356–362. Lien, Y.J., Tsuang, H.C., Chiang, A., Liu, C.M., Hsieh, M.H., Hwang, T.J., Liu, S.K., Hsiao, P.C., Faraone, S.V., Tsuang, M.T., Hwu, H.G., Chen, W.J., 2009. The multidimensionality of schizotypy in nonpsychotic relatives of patients with schizophrenia and its applications in ordered subsets linkage analysis of schizophrenia. Am. J. Med. Genet. B Neuropsychiatr. Genet. 153B (1), 1–9. Liu, C.C., Chiu, Y.C., Chiu, Y.N., Lai, M.C., Hwu, H.G., 2009. Mental health literacy: impact of newspaper report on increasing recognition of psychotic prodrome. Taiwan. J. Psychiatry 23, 320–330. Liu, C.C., Chang, L.R., Tseng, H.H., Lai, M.C., Hwu, H.G., 2010a. Differential propensity in recognition of prepsychotic phenomena among psychiatrists, clinical psychologists and school counsellors. Early Interv. Psychiatry 4 (4), 275–282. Liu, C.C., Hwu, H.G., Chiu, Y.N., Lai, M.C., Tseng, H.H., 2010b. Creating a platform to bridge service and research for early psychosis. J. Formos. Med. Assoc. 109 (7), 543–549. Liu, C.C., Lai, M.C., Liu, C.M., Chiu, Y.N., Hsieh, M.H., Hwang, T.J., Chien, Y.L., Chen, W.J., Hua, M.S., Hsiung, P.C., Huang, Y.C., Hwu, H.G., 2011. Follow-up of subjects with suspected pre-psychotic state in Taiwan. Schizophr. Res. 126 (1–3), 65–70. Loewy, R.L., Pearson, R., Vinogradov, S., Bearden, C.E., Cannon, T.D., 2011. Psychosis risk screening with the Prodromal Questionnaire—brief version (PQ-B). Schizophr. Res. 129 (1), 42–46. Loewy, R.L., Bearden, C.E., Johnson, J.K., Raine, A., Cannon, T.D., 2005. The prodromal questionnaire (PQ): preliminary validation of a self-report screening measure for prodromal and psychotic syndromes. Schizophr. Res. 79 (1), 117–125. Loewy, R.L., Johnson, J.K., Cannon, T.D., 2007. Self-report of attenuated psychotic experiences in a college population. Schizophr. Res. 93 (1–3), 144–151. McGorry, P.D., Yung, A.R., Phillips, L.J., 2003. The “close-in” or ultra high-risk model: a safe and effective strategy for research and clinical intervention in prepsychotic mental disorder. Schizophr. Bull. 29 (4), 771–790.
37
Mojtabai, R., Malaspina, D., Susser, E., 2003. The concept of population prevention: application to schizophrenia. Schizophr. Bull. 29 (4), 791–801. Muller, M., Vetter, S., Buchli-Kammermann, J., Stieglitz, R.D., Stettbacher, A., RiecherRossler, A., 2010. The Self-screen-Prodrome as a short screening tool for prepsychotic states. Schizophr. Res. 123 (2–3), 217–224. Nakayasu, N., Seki, Y., Harima, H., 1999. Age of onset, frequency of the development of symptoms, and treatment outcome in the early stage of schizophrenia—in search of guidelines for early diagnosis and treatment of schizophrenia. Seishin Shinkeigaku Zasshi 101 (11), 898–907. O'Donoghue, B., Lyne, J., Renwick, L., Madigan, K., Kinsella, A., Clarke, M., Turner, N., O'Callaghan, E., 2012. A descriptive study of ‘non-cases’ and referral rates to an early intervention for psychosis service. Early Interv. Psychiatry 6 (3), 276–282. O'Toole, B.I., 2000. Screening for low prevalence disorders. Aust. N. Z. J. Psychiatry 34, S39–S46 (Suppl.). Raine, A., 1991. The SPQ: a scale for the assessment of schizotypal personality based on DSM-III-R criteria. Schizophr. Bull. 17 (4), 555–564. Rossler, W., Riecher-Rossler, A., Angst, J., Murray, R., Gamma, A., Eich, D., van Os, J., Gross, V.A., 2007. Psychotic experiences in the general population: a twenty-year prospective community study. Schizophr. Res. 92 (1–3), 1–14. Ruhrmann, S., Schultze-Lutter, F., Salokangas, R.K., Heinimaa, M., Linszen, D., Dingemans, P., Birchwood, M., Patterson, P., Juckel, G., Heinz, A., Morrison, A., Lewis, S., von Reventlow, H.G., Klosterkotter, J., 2010. Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European prediction of psychosis study. Arch. Gen. Psychiatry 67 (3), 241–251. Scott, J., Chant, D., Andrews, G., McGrath, J., 2006. Psychotic-like experiences in the general community: the correlates of CIDI psychosis screen items in an Australian sample. Psychol. Med. 36 (2), 231–238. Verdoux, H., van Os, J., 2002. Psychotic symptoms in non-clinical populations and the continuum of psychosis. Schizophr. Res. 54 (1–2), 59–65. Werbeloff, N., Drukker, M., Dohrenwend, B.P., Levav, I., Yoffe, R., van Os, J., Davidson, M., Weiser, M., 2012. Self-reported attenuated psychotic symptoms as forerunners of severe mental disorders later in life. Arch. Gen. Psychiatry 69 (5), 467–475. Wu, H.M., Tien, Y.J., Chen, C.H., 2010. GAP: a graphical environment for matrix visualization and cluster analysis. Comput. Stat. Data Anal. 54, 767–778. Yang, L.H., Wonpat-Borja, A.J., Opler, M.G., Corcoran, C.M., 2010. Potential stigma associated with inclusion of the psychosis risk syndrome in the DSM-V: an empirical question. Schizophr. Res. 120 (1–3), 42–48. Yung, A.R., Buckby, J.A., Cotton, S.M., Cosgrave, E.M., Killackey, E.J., Stanford, C., Godfrey, K., McGorry, P.D., 2006. Psychotic-like experiences in nonpsychotic help-seekers: associations with distress, depression, and disability. Schizophr. Bull. 32 (2), 352–359.
Contents lists available at SciVerse ScienceDirect
Schizophrenia Research journal homepage: www.elsevier.com/locate/schres
Development of a brief self-report questionnaire for screening putative pre-psychotic states Chen-Chung Liu a, Yin-Jing Tien b, Chun-Houh Chen b, Yen-Nan Chiu a, Yi-Ling Chien a, Ming H. Hsieh a, Chih-Min Liu a, Tzung-Jeng Hwang a, Hai-Gwo Hwu a,⁎ a b
Department of Psychiatry, National Taiwan University Hospital and College of Medicine, National Taiwan University, Taipei, Taiwan Institute of Statistical Science, Academia Sinica, Taipei, Taiwan
a r t i c l e
i n f o
Article history: Received 28 June 2012 Received in revised form 30 August 2012 Accepted 8 October 2012 Available online 24 November 2012 Keywords: Pre-psychotic Schizophrenia Screening Self-report questionnaire
a b s t r a c t Background: Several self-report instruments were developed to capture psychotic prodrome, and were claimed to have good predictive validity. The feasibility of screening is questionable considering the heterogeneity of the targeted populations and the negative ramifications of false positive identification. This study developed a questionnaire using data covering a wide range of clinical characteristics. Methods: One hundred and eleven putative pre-psychotic participants, 129 normal comparison subjects, and 95 non-psychotic psychiatric outpatients completed a 231-item questionnaire comprising a 110-item Wisconsin psychotic prone scale, 74-item schizotypal personality questionnaire, 33-item basic symptoms, and 14-item cognitive symptoms. Items showing the best discriminating power, estimated using chi-square statistics with Bonferroni correction, were extracted to create a brief version. A two-stage cut-off approach emphasizing specific items was applied to maximize sensitivity and specificity. The concurrent validity of the proposed approach was estimated using a ten-fold cross-validation procedure. Results: A 15-item self-report questionnaire was developed. Respondents checking at least eight items, or those checking three to seven items including any of the three referring to feeling stress in crowds, aloofness, and perceptual disturbance, would be considered putatively pre-psychotic with the largest sensitivity+ specificity (0.784+ 0.705=1.489). This cut-off selection was the best estimate by calculating 1000 permutations in the cross-validation procedure. Conclusions: This investigation proposes a different orientation for applying questionnaires to screen putative pre-psychotic states, with less emphasis on attenuated psychotic symptoms and predictive values. Besides providing a handy tool for increasing awareness and referral, the instructions of such a screening questionnaire should be carefully worded. © 2012 Elsevier B.V. All rights reserved.
1. Introduction The prodromal symptoms of schizophrenia are near indistinguishable from other common psychiatric illnesses or stress reactions. The development of newer instruments for clinical assessment has made the predictive validity of prodromal criteria considered acceptable when applied to the help-seeking population (Cannon et al., 2008; Ruhrmann et al., 2010), but screening in the general population is considered infeasible owing to concerns regarding non-specificity and low prediction rates (Jablensky, 2000; Cougnard et al., 2005). Basic symptoms have been reported with high predictive values in a clinical sample (Klosterkotter et al., 2001), but are difficult to apply through self-reporting and in the community (Mojtabai et al., 2003). CASIS, an ⁎ Corresponding author at: Department of Psychiatry, National Taiwan University Hospital and College of Medicine, National Taiwan University, No. 7 Chung Shan S. Rd, Taipei 10043, Taiwan. Tel.: + 886 2 2312 3456x66785; fax: + 886 2 2375 3663. E-mail address: [email protected] (H.-G. Hwu). 0920-9964/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.schres.2012.10.042
acronym meaning cognitive deficits, affective symptoms, social isolation, and school failure, is a non-specific checklist for psychotic prodrome (Cornblatt et al., 2003). The presence of psychotic-like experiences (PLEs) is an important marker for identifying at-risk subjects (Armando et al., 2010; Kelleher and Cannon, 2011); while PLEs are common in the general population (Verdoux and van Os, 2002; Scott et al., 2006), the prevalence of PLEs is highest among adolescents and tends to decrease in reporting frequency after subjects enter their 20s (Rossler et al., 2007). Information campaigns or the provision of special services for suspected psychotic prodrome inevitably draws attention to non-psychotic subjects and will increase the burden for clinical screening (Liu et al., 2009; O'Donoghue et al., 2012). Even mental health professionals from different training backgrounds can be differently inclined in recognizing pre-psychotic phenomena (Liu et al., 2010a). However, for mental health education and for recruiting subjects at-risk of developing psychosis to study and follow-up, it is best to provide an easy, convenient questionnaire to attract people's attention to engage in self evaluation and referral.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
Kline et al. have compared several screening tools, each comprising 12 to 28 items primarily addressing attenuated psychotic symptoms, to demonstrate their good convergent and discriminant validity for attenuated psychosis (Kline et al., 2012). In addition, Kobayashi et al. proposed a 12-item self-report instrument for psychotic prodrome with acceptable sensitivity, specificity, and positive predictive value (Kobayashi et al., 2008), and Muller et al. also suggested a 32-item self-report instrument as a handy screening tool in a large conscript population (Muller et al., 2010). However, whether subjects seek help voluntarily and whether they are distressed strongly affects the predictive power (Loewy et al., 2005, 2007), and caution is needed in applying these instruments to identify at-risk subjects owing to the potential risk of mislabeling and even stigmatizing (Corcoran et al., 2005; Yang et al., 2010). In this paper we present a brief self-report questionnaire, which is derived from a follow-up study on the psychopathological progress of early schizophrenia-like disorder (SOPRES) in Taiwan, for screening putative pre-psychotic states that assigns relatively little emphasis to predictive values for transition to psychosis but covers a broader range of early prodromal symptoms. 2. Methods 2.1. Sample Subjects were participants in the SOPRES study. The rationale and methodology for the SOPRES study have been described elsewhere (Liu et al., 2010b, 2011). Briefly, we employed a low-threshold approach to invite individuals aged between 16 and 32 years old presenting with non-specific symptoms as described using CASIS (Cornblatt et al., 2003) or having newly developed psychotic-like symptoms to an outpatient psychiatric special clinic of the study hospital for thorough assessments (Liu et al., 2010b). All adult participants voluntarily provided written informed consent, and minors gave written assent with informed consent from their parents. This study has been approved by the Institutional Review Board of the study hospital. This investigation conducted clinical assessment using the Thought/ Perception Diagnostic Interview Schedule (TP-DIS) to classify participants into four groups based on level of clinical risk (Liu et al., 2011). The group of first-episode psychosis (FEP) included participants with schizophrenia, schizophreniform disorder, brief psychotic disorder, or schizoaffective disorder meeting the DSM-IV criteria during the previous year. The ultra-high-risk group (UHR) included participants with attenuated psychotic symptoms (APS) or brief limited intermittent psychotic symptoms (BLIPS) (McGorry et al., 2003). The intermediate-risk group (IRG) included participants presenting with odd thinking, feelings, speech, or perceptual experiences, which were not as severe as in the UHR group but met the criteria of schizotypal disorder as specified in the 10th edition of the International Classification of Diseases (ICD-10) without the requirement of two year duration. The marginal-risk group (MRG) included participants with CASIS symptoms that did not meet either the threshold for the IRG or other diagnostic categories. A group of age- and gender-matched healthy volunteers was recruited by public posters, and a group of non-psychotic psychiatric outpatients (NPO) was recruited via referral by their attending psychiatrists. Subjects with IQ below 70, aged younger than 16 years old, with a history of traumatic brain injury, central nervous system illness, a prior psychotic episode lasting for more than one year, or current use of psychoactive stimulants were excluded. The FEP group was excluded from further analysis because the questionnaire is focused on screening pre-psychotic states. 2.2. Instrument At baseline assessment, all participants completed a self-administered 231-item Mandarin version Schizophrenia Proneness Scale comprising a 110-item Wisconsin psychotic prone scale (Chapman et al., 1994), a
33
74-item schizotypal personality questionnaire (Raine, 1991; Chen et al., 1998), 33 items of basic symptoms (Gross, 1989; Klosterkotter et al., 2001), and 14 items of cognitive symptoms (Nakayasu et al., 1999). Participants were encouraged to answer as many questions as possible. All questions were answered using simple “yes” or “no” responses to minimize the response burden. A “yes” represents an affirmative response to the question of a deviant experience. 2.3. Analysis steps and statistical methods Sporadic missing values were treated as negative responses. Each individual item was examined using chi-square test. Items with a p valueb 0.001 based on comparison among five groups were extracted first. Post hoc analysis with Bonferroni correction was used to test the differences between each pair of UHR, IRG, and MRG groups, and none of these items exhibited significant differences in self-reporting frequency; consequently, these three groups were collapsed into a single putative pre-psychotic group (PPP) for further analysis. Furthermore, items with a p valueb 0.05 compared between the PPP and NPO groups were identified to establish a brief self-report questionnaire. Items fulfilling the aforementioned working criteria were scrutinized by two principal clinical investigators (HGH and CCL) to ensure that each item had face validity and that no apparent overlap existed among items extracted from different sets of original questionnaires. With the goal being to maximize sensitivity + specificity, this study employed a two-stage cut-off approach by emphasizing three designated items expected to be more likely to refer to attenuated psychotic symptoms as to increase the sensitivity and specificity. Whether this combination of cut-off and weighted items was the most feasible version for screening was then tested using estimations via a ten-fold leave-one-out cross-validation procedure (Burman, 1989). 3. Results 3.1. Participant demographics A total of 111 putative pre-psychotic participants (PPP, including 41 UHR, 32 IRG, and 38 MRG participants), a normal comparison group (NC) comprising 129 subjects, and 95 non-psychotic psychiatric outpatients (NPO) responded to the original 231-item questionnaires. Table 1 lists the demographic profiles of the respondents. 3.2. Development of a 15-item questionnaire Initially, 14 items fulfilling the selection criteria were derived from the original 231 items, and then two items were removed based on Table 1 Comparison of demographics and frequencies of responses to individual items across three groups.
Age Gender Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Item 14 Item 15
PPP (n = 111)
NPO (n = 95)
NC (n = 129)
Statistics
p
22.5 55 65 61 45 72 59 69 63 69 77 54 48 58 42 45 27
24.15 69 38 31 26 42 33 38 34 38 52 34 14 33 36 27 16
21.33 55 10 10 3 26 14 23 20 26 35 15 10 18 10 9 5
13.807 19.827 71.228 63.072 52.775 49.346 49.943 49.459 44.718 43.983 44.368 39.986 48.126 40.160 37.137 37.963 21.005
b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001 b0.001
(3.89) (49.5%) (58.6%) (55%) (40.5%) (64.9%) (53.2%) (62.2%) (56.8%) (62.2%) (69.4%) (48.6%) (43.2%) (52.3%) (37.8%) (40.5%) (24.3%)
(4.27) (71.9%) (40%) (32.6%) (27.4%) (44.2%) (34.7%) (40%) (35.8%) (40%) (54.7%) (35.8%) (14.7%) (34.7%) (37.9%) (28.4%) (16.8%)
(3.71) (42.6%) (7.8%) (7.8%) (2.3%) (20.2%) (10.9%) (17.8%) (15.5%) (20.2%) (27.1%) (11.6%) (7.8%) (14%) (7.8%) (7%) (3.9%)
34
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
their apparent similarity to the other two items with smaller p values, yielding a set of 12 items. However, only one of these 12 items referring to mild attenuated psychotic symptoms (Item 6: I sometimes become concerned about the loyalty and trustworthiness of friends or coworkers.),
so the principal investigators manually selected three items which met only the first part of the selection criteria (with very small p values comparing among five groups) to form a 15-item questionnaire. Table 1 listed the distribution of response frequencies among the three groups. This
(C)
(A)
(B)
(D)
Fig. 1. Matrix visualization of 335 participants with 15 items. Left axis: number of participants. Panel (A): The corresponding composition of PPP/NPO/NC within each summary score. Panel (B) The total number of “Yes” of each participant responded to 15 items. Black: cumulative number of Yes; white: cumulative number of No. Panel (C): Jaccard coefficient matrix for 15 items with 335 participants. The tree structure in the right of the Jaccard coefficient matrix represents average linkage hierarchical clustering tree (HCT) for these 15 items. Panel (D): The profiles of response to these 15 items by each participant; each row represents a participant. Items were clustered by average linkage with Jaccard coefficient of these 15 items within 335 participants. Panel (E): The composition of PPP/NPO/NC in each of these 3 groups (nYes ≥8, 3 ≤ nYes b 8, nYes b 3). Group: black: PPP, putative pre-psychotic; gray: NPO, non-psychotic outpatient; white: NC, normal control.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
study used the sum of these 15 variables as a simple score (number of affirmative responses, nYes). This study tries to identify two cut-points to enable the inclusion of putative pre-psychotic subjects not at the expense of including too many subjects from the other two groups. Matrix visualization (Chen, 2002; Chen et al., 2004; Wu et al., 2010) of 335 participants with 15 items is shown in Fig. 1, in which 89 subjects exhibited nYes≥8, including 62, 23, and 4 participants belonging to the PPP, NPO, and NC groups, respectively; furthermore, 138 subjects exhibited nYesb 3, including 16, 27, and 95 participants belonging to the PPP, NPO, and NC groups, respectively; while the remaining 108 participants comprised a mixture of 33, 45 and 30 individuals from the PPP, NPO and NC groups, respectively. Thus it is desirable to seek a solution by adding a condition to this subset (3≤nYesb 8): if the “Yes” response of the participant included any of the three designated items. This study exhausted all the combinations of two cut-points and assigned weightings to one of three items to identify the best solution using this formula: largest sensitivity + specificity. Finally, the higher cut-point was set to nYes ≥ 8 to indicate a positive value for a PPP and the lower cut-point was set to nYes b 3 to indicate a negative value for a PPP, while those participants with nYes ranging from 3 to 7 yet also responding “Yes” to any of the 3 items, I cannot deal with the pressures associated with crowds. (Item 1), I feel I cannot get close to people. (Item 2), Do you hear some sounds, voices, or calls of your name when nobody is around you? (Item 15), are also treated as the PPP group. This procedure yielded the largest value of sensitivity + specificity (0.784 + 0.705 = 1.489) to differentiate PPP from non-PPP (Table 2). The checks of response bias by exploring the composition of PPP/ NPO/NC within each summary score and the composition of PPP/ NPO/NC in each of those 3 groups (nYes ≥ 8, 3 ≤ nYes b 8, nYes b 3) revealed satisfactory results as shown in Column A and Column E of Fig. 1. 3.3. Re-test of sensitivity and specificity Because this formula was not tested on a group of subjects independent of the study population, we tried to estimate its sensitivity and specificity using a ten-fold leave-one-out cross-validation method (Burman, 1989). With the goal of first finding two cut-points and then finding three items if the subject answers “Yes” to any of the three questions as addressed in the previous section, we broke the data into ten subsets of size n/10, conducted training on 9 of the 10 subsets and tested the one remaining subset, and repeated this procedure ten times such that each n/10 samples can be used once as the testing subset, and data was randomly divided into 10 subsets of size n/10 for 100 times. Thus 10 ∗ 100 = 1000 permutations were yielded to test for validations. The two-point cutoff (3, 8) has been selected 931 times, the combination of three items as previously designated has been selected 613 times, and another combination with just 1 item difference from the first combination (Do you often pick up hidden threats or put downs from the words or actions of others? rather than Do you hear some sounds, voices, or calls of your name
35
when nobody is around you?) has been selected 117 times. The estimated averaged sensitivity and specificity of the final screening formula are 0.736 and 0.679, respectively (Table 2), which are slightly lower than the original values (0.784 and 0.705) as expected using the leave-one-out cross-validation method (O'Toole, 2000). 4. Discussions Reduced from the original 231 items, a 15-item self-report questionnaire was developed (Appendix 1). The 15 items fall into four categories (upper right of Fig. 1), including five interpersonal difficulty/social anxiety symptoms (Item 2, 5, 7, 8, and 12), three self-depreciating descriptions (Item 1, 4, and 9), three negative symptoms (Item 3, 10, and 11), and four subthreshold psychotic-like experiences (Item 6, 13, 14, and 15), similar to the four-dimension structure, namely negative schizotypy, positive schizotypy, interpersonal sensitivity, and social isolation/introversion, revealed by analyzing the Structural Interview of Schizotypy (SIS) data in our previous study of non-psychotic relatives of patients with schizophrenia (Lien et al., 2009). This suggests that the screening tool has certain construct validity. The screening tool was tested by comparing the group of putative pre-psychotic participants with a group of non-psychotic psychiatric outpatients and a group of normal controls to examine its discriminant ability. According to the working criteria, participants who had checked at least eight items or who had checked three to seven items, including any one of the three items referring to low tolerance for stress in a crowd, aloofness, and perceptual disturbance, were considered to represent a putative pre-psychotic state. Although the values of sensitivity and specificity are not striking, such a result is reasonable referring to the nature of non-specificity in early prodromal symptoms (Keshavan et al., 2011). Since this study does not attempt to develop a better and more accurate solution for screening ultra-high risk subjects, we feel comfortable using this instrument simply for quick self-evaluation and referral, not specifically emphasizing the high likelihood of the transition to psychosis, but rather addressing a need for clinical attention and potential interest in follow-up based on the relatively raised risk linked to a trajectory of psychosis. Interestingly, prior to performing statistical analysis, this study assumed those subthreshold psychotic-like symptoms would more likely be selected as the three weighted items. However, in the results only the item referring to perceptual disturbance was selected, but not the items referring to suspiciousness, idea of reference and persecutory ideations. This finding might have resulted from a relative low prevalence of attenuated psychotic symptoms in a broader range of prodromal syndromes covered in our study population (Table 1). In fact our proposed screening questionnaire is not focusing on positive prediction of transition rates but with an intention to capture the earlier putative prodromal states, thus making it possible to follow up the formation of psychosis and explore the underlying pathogenesis from the very beginning (Keshavan et al., 2011; Hsieh et al., 2012). Given that the putative pre-psychotic participants cannot be differentiated based on
Table 2 Statistical characteristics using two different cut-off approaches and leave-one-out cross-validation of the two cut-off points approach.
Sensitivity Specificity Overall accuracy Positive predictive value Negative predictive value False positive rate False negative rate Prevalence rate by cut-off
Single cut-off point if nYes ≥ 8
Two cut-off points (3, 8)+(at least one of three items with “Yes” response)a
Leave-one-out cross-validation for two cut-off points+(at least one of three items with “Yes” response)
0.559 0.879 0.773 0.700 0.800 0.121 0.441 0.266
0.784 0.705 0.731 0.569 0.868 0.295 0.216 0.457
0.736 0.679 0.698 0.532 0.839 0.321 0.264 0.458
a These three items are Item 1: I cannot deal with the pressures associated with crowds; Item 2: I feel I cannot get close to people; Item 15: Do you hear some sounds, voices, or calls of your name when nobody is around you?
36
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37
their self-report results, the clinical interview can categorize them into three different risk levels. Furthermore, the results of the follow-up study demonstrate that only one-third of patients from the UHR group have progressed to full-blown psychosis, while none of the IRG and MRG subjects converted within 2 years (Liu et al., 2011). Thus a brief self-report questionnaire is not advised to aim at predicting psychosis in the short-term. Nonetheless, this questionnaire can be employed to attract the attention of the lay public to psychotic prodrome, and serves as a handy tool for self-screening and referral, under the premise of carefully wording in instructions while using the questionnaire. The tradeoff between increasing sensitivity at the expense of decreasing specificity has no uniform solution referring to clinical practice (O'Toole, 2000). For example, as demonstrated in Table 2, compared to a simple one cut-off point (nYes≥ 8) approach, the performance of our working criteria can reduce half false negative rates (from 0.441 to 0.216) but the prevalence rate using this approach will increase 72% (from 0.266 to 0.457) and the false positive rate will increase 144% (from 0.121 to 0.295), which implies much heavier demands for providing clinical assessment to subjects identified as positive by screening and potentially much more negative ramification to false positives, such as the risks of being labeled as at risk, stigmatization and self-stigmatization, and unnecessary exposure to the risk of having adverse reactions to early antipsychotic treatment (Corcoran et al., 2005, 2010; Yang et al., 2010). Thus given that our working criteria are statistically the best selection, while applying to real world situations, the choice of different cutoffs still depends on the purpose of screening and the resources available to individual works. The main limitation of this study is not testing the test–retest reliability of the 15-item questionnaire on participants and its validity and reliability on any other reference population. Furthermore, this study did not perform follow-up clinical diagnosis (or no diagnosis) of those who did not convert to psychosis in the PPP group. While acknowledging the limitations of a self-report questionnaire, this study does not attempt to develop an alternative questionnaire with better predicting abilities. Instead, compared to those instruments mainly focused on attenuated psychotic symptoms or psychotic-like experiences to stress its “predictive value” regarding transition rates to frank psychosis (Loewy et al., 2005; Kobayashi et al., 2008; Kelleher et al., 2011; Kline et al., 2012), this proposed questionnaire might be able to capture the early and broadly defined at risk mental status which not only comprises attenuated positive symptoms, but also mood symptoms, negative symptoms and interpersonal difficulties (Keshavan et al., 2011). Creating a brief screening questionnaire with good predictive values is impractical and unnecessary. Indeed a recent large-scale population-based prospective study suggests that self-reported attenuated symptoms are not clinically useful predictors for psychotic disorders later in life (Werbeloff et al., 2012). Thus this brief self-report questionnaire is designed to perform screening for first step risk assessment (Loewy et al., 2011; Ruhrmann et al., 2010). More accurate prediction requires a formal interview and consideration of more clinical variables for risk assessment (Yung et al., 2006; Cannon et al., 2008; Ruhrmann et al., 2010). As the false positive rates can increase rapidly while trying to increase sensitivity, the instructions of a screening tool should be carefully deliberated to avoid misinterpretation of self-report results and subsequent unwanted ramifications. Role of funding source The National Health Research Institutes, Taiwan, provided grants for studying the pre-psychotic state and early psychosis; YJ Tien and CH Chen's work was supported partially by the National Core Facility Program for Biotechnology, Taiwan. Both funding sources had no involvement in study design, data collection, analysis and interpretation, report writing, or the decision to submit the paper for publication.
Contributors CC Liu and HG Hwu reviewed literature and designed the study; CC Liu, YN Chiu, YL Chien, MH Hsieh, CM Liu, TJ Hwang, and HG Hwu contributed to case recruitment and
assessment; YJ Tien and CH Chen did the statistical analyses; all authors contributed to the discussion and interpretation of data; CC Liu wrote the first draft; HG Hwu oversaw the research; all authors contributed to and have approved the final manuscript. Conflict of interest All authors declare no conflicts of interest regarding the work presented in this paper. Acknowledgments This work was supported by the National Health Research Institutes, Taiwan (NHRI-EX95, 96, 97, 98, 99-9511PP) and the National Core Facility Program for Biotechnology, Taiwan (Bioinformatics Consortium of Taiwan, NSC100-2319-B-010-002).
Appendix 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
I cannot deal with the pressures associated with crowds. I feel I cannot get close to people. I feel lethargic whatever I do. I feel mentally insufficient and easily fatigued while thinking or reading. I am mostly quiet when with others. I sometimes become concerned about the loyalty and trustworthiness of friends or coworkers. I tend to keep my feelings to myself. I feel nervous when giving a speech in front of a large group of people. I cannot focus on a task and need to take frequent breaks while working (studying). I always mess up whatever I do. I do not have an expressive and lively way of speaking. I am poor at returning social courtesies and gestures. When you see people talking to each other, do you often wonder if they are talking about you? Do you often pick up hidden threats or put downs from the words or actions of others? Do you hear some sounds, voices, or calls of your name when nobody is around you?
References Armando, M., Nelson, B., Yung, A.R., Ross, M., Birchwood, M., Girardi, P., Fiori Nastro, P., 2010. Psychotic-like experiences and correlation with distress and depressive symptoms in a community sample of adolescents and young adults. Schizophr. Res. 119 (1–3), 258–265. Burman, P., 1989. A comparative study of ordinary cross-validation, v-fold crossvalidation and repeated learning-testing methods. Biometrika 76, 503–514. Cannon, T.D., Cadenhead, K., Cornblatt, B., Woods, S.W., Addington, J., Walker, E., Seidman, L.J., Perkins, D., Tsuang, M., McGlashan, T., Heinssen, R., 2008. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch. Gen. Psychiatry 65 (1), 28–37. Chapman, L.J., Chapman, J.P., Kwapil, T.R., Eckblad, M., Zinser, M.C., 1994. Putatively psychosis-prone subjects 10 years later. J. Abnorm. Psychol. 103 (2), 171–183. Chen, C.H., 2002. Generalized association plots: information visualization via iteratively generated correlation matrices. Statist. Sin. 12, 7–29. Chen, W.J., Liu, S.K., Chang, C.J., Lien, Y.J., Chang, Y.H., Hwu, H.G., 1998. Sustained attention deficit and schizotypal personality features in nonpsychotic relatives of schizophrenic patients. Am. J. Psychiatry 155 (9), 1214–1220. Chen, C.H., Hwu, H.G., Jang, W.J., Kao, C.H., Tien, Y.J., Tzeng, S., Wu, H.M., 2004. Matrix visualization and information mining. Proceedings in Computational Statistics, pp. 85–100. Corcoran, C., Malaspina, D., Hercher, L., 2005. Prodromal interventions for schizophrenia vulnerability: the risks of being “at risk”. Schizophr. Res. 73 (2–3), 173–184. Corcoran, C.M., First, M.B., Cornblatt, B., 2010. The psychosis risk syndrome and its proposed inclusion in the DSM-V: a risk–benefit analysis. Schizophr. Res. 120 (1–3), 16–22. Cornblatt, B.A., Lencz, T., Smith, C.W., Correll, C.U., Auther, A.M., Nakayama, E., 2003. The schizophrenia prodrome revisited: a neurodevelopmental perspective. Schizophr. Bull. 29 (4), 633–651. Cougnard, A., Salmi, L.R., Salamon, R., Verdoux, H., 2005. A decision analysis model to assess the feasibility of the early detection of psychosis in the general population. Schizophr. Res. 74 (1), 27–36. Gross, G., 1989. The ‘basic’ symptoms of schizophrenia. Br. J. Psychiatry (7), 21–25 (Suppl., discussion 37–40). Hsieh, M.H., Shan, J.C., Huang, W.L., Cheng, W.C., Chiu, M.J., Jaw, F.S., Hwu, H.G., Liu, C.C., 2012. Auditory event-related potential of subjects with suspected pre-psychotic state and first-episode psychosis. Schizophr. Res. 140 (1–3), 243–249. Jablensky, A., 2000. Prevalence and incidence of schizophrenia spectrum disorders: implications for prevention. Aust. N. Z. J. Psychiatry 34, S26–S34 (Suppl., discussion S35–28). Kelleher, I., Cannon, M., 2011. Psychotic-like experiences in the general population: characterizing a high-risk group for psychosis. Psychol. Med. 41 (1), 1–6.
C.-C. Liu et al. / Schizophrenia Research 143 (2013) 32–37 Kelleher, I., Harley, M., Murtagh, A., Cannon, M., 2011. Are screening instruments valid for psychotic-like experiences? A validation study of screening questions for psychotic-like experiences using in-depth clinical interview. Schizophr. Bull. 37 (2), 362–369. Keshavan, M.S., Delisi, L.E., Seidman, L.J., 2011. Early and broadly defined psychosis risk mental states. Schizophr. Res. 126 (1–3), 1–10. Kline, E., Wilson, C., Ereshefsky, S., Tsuji, T., Schiffman, J., Pitts, S., Reeves, G., 2012. Convergent and discriminant validity of attenuated psychosis screening tools. Schizophr. Res. 134 (1), 49–53. Klosterkotter, J., Hellmich, M., Steinmeyer, E.M., Schultze-Lutter, F., 2001. Diagnosing schizophrenia in the initial prodromal phase. Arch. Gen. Psychiatry 58 (2), 158–164. Kobayashi, H., Nemoto, T., Koshikawa, H., Osono, Y., Yamazawa, R., Murakami, M., Kashima, H., Mizuno, M., 2008. A self-reported instrument for prodromal symptoms of psychosis: testing the clinical validity of the PRIME Screen—Revised (PS-R) in a Japanese population. Schizophr. Res. 106 (2–3), 356–362. Lien, Y.J., Tsuang, H.C., Chiang, A., Liu, C.M., Hsieh, M.H., Hwang, T.J., Liu, S.K., Hsiao, P.C., Faraone, S.V., Tsuang, M.T., Hwu, H.G., Chen, W.J., 2009. The multidimensionality of schizotypy in nonpsychotic relatives of patients with schizophrenia and its applications in ordered subsets linkage analysis of schizophrenia. Am. J. Med. Genet. B Neuropsychiatr. Genet. 153B (1), 1–9. Liu, C.C., Chiu, Y.C., Chiu, Y.N., Lai, M.C., Hwu, H.G., 2009. Mental health literacy: impact of newspaper report on increasing recognition of psychotic prodrome. Taiwan. J. Psychiatry 23, 320–330. Liu, C.C., Chang, L.R., Tseng, H.H., Lai, M.C., Hwu, H.G., 2010a. Differential propensity in recognition of prepsychotic phenomena among psychiatrists, clinical psychologists and school counsellors. Early Interv. Psychiatry 4 (4), 275–282. Liu, C.C., Hwu, H.G., Chiu, Y.N., Lai, M.C., Tseng, H.H., 2010b. Creating a platform to bridge service and research for early psychosis. J. Formos. Med. Assoc. 109 (7), 543–549. Liu, C.C., Lai, M.C., Liu, C.M., Chiu, Y.N., Hsieh, M.H., Hwang, T.J., Chien, Y.L., Chen, W.J., Hua, M.S., Hsiung, P.C., Huang, Y.C., Hwu, H.G., 2011. Follow-up of subjects with suspected pre-psychotic state in Taiwan. Schizophr. Res. 126 (1–3), 65–70. Loewy, R.L., Pearson, R., Vinogradov, S., Bearden, C.E., Cannon, T.D., 2011. Psychosis risk screening with the Prodromal Questionnaire—brief version (PQ-B). Schizophr. Res. 129 (1), 42–46. Loewy, R.L., Bearden, C.E., Johnson, J.K., Raine, A., Cannon, T.D., 2005. The prodromal questionnaire (PQ): preliminary validation of a self-report screening measure for prodromal and psychotic syndromes. Schizophr. Res. 79 (1), 117–125. Loewy, R.L., Johnson, J.K., Cannon, T.D., 2007. Self-report of attenuated psychotic experiences in a college population. Schizophr. Res. 93 (1–3), 144–151. McGorry, P.D., Yung, A.R., Phillips, L.J., 2003. The “close-in” or ultra high-risk model: a safe and effective strategy for research and clinical intervention in prepsychotic mental disorder. Schizophr. Bull. 29 (4), 771–790.
37
Mojtabai, R., Malaspina, D., Susser, E., 2003. The concept of population prevention: application to schizophrenia. Schizophr. Bull. 29 (4), 791–801. Muller, M., Vetter, S., Buchli-Kammermann, J., Stieglitz, R.D., Stettbacher, A., RiecherRossler, A., 2010. The Self-screen-Prodrome as a short screening tool for prepsychotic states. Schizophr. Res. 123 (2–3), 217–224. Nakayasu, N., Seki, Y., Harima, H., 1999. Age of onset, frequency of the development of symptoms, and treatment outcome in the early stage of schizophrenia—in search of guidelines for early diagnosis and treatment of schizophrenia. Seishin Shinkeigaku Zasshi 101 (11), 898–907. O'Donoghue, B., Lyne, J., Renwick, L., Madigan, K., Kinsella, A., Clarke, M., Turner, N., O'Callaghan, E., 2012. A descriptive study of ‘non-cases’ and referral rates to an early intervention for psychosis service. Early Interv. Psychiatry 6 (3), 276–282. O'Toole, B.I., 2000. Screening for low prevalence disorders. Aust. N. Z. J. Psychiatry 34, S39–S46 (Suppl.). Raine, A., 1991. The SPQ: a scale for the assessment of schizotypal personality based on DSM-III-R criteria. Schizophr. Bull. 17 (4), 555–564. Rossler, W., Riecher-Rossler, A., Angst, J., Murray, R., Gamma, A., Eich, D., van Os, J., Gross, V.A., 2007. Psychotic experiences in the general population: a twenty-year prospective community study. Schizophr. Res. 92 (1–3), 1–14. Ruhrmann, S., Schultze-Lutter, F., Salokangas, R.K., Heinimaa, M., Linszen, D., Dingemans, P., Birchwood, M., Patterson, P., Juckel, G., Heinz, A., Morrison, A., Lewis, S., von Reventlow, H.G., Klosterkotter, J., 2010. Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European prediction of psychosis study. Arch. Gen. Psychiatry 67 (3), 241–251. Scott, J., Chant, D., Andrews, G., McGrath, J., 2006. Psychotic-like experiences in the general community: the correlates of CIDI psychosis screen items in an Australian sample. Psychol. Med. 36 (2), 231–238. Verdoux, H., van Os, J., 2002. Psychotic symptoms in non-clinical populations and the continuum of psychosis. Schizophr. Res. 54 (1–2), 59–65. Werbeloff, N., Drukker, M., Dohrenwend, B.P., Levav, I., Yoffe, R., van Os, J., Davidson, M., Weiser, M., 2012. Self-reported attenuated psychotic symptoms as forerunners of severe mental disorders later in life. Arch. Gen. Psychiatry 69 (5), 467–475. Wu, H.M., Tien, Y.J., Chen, C.H., 2010. GAP: a graphical environment for matrix visualization and cluster analysis. Comput. Stat. Data Anal. 54, 767–778. Yang, L.H., Wonpat-Borja, A.J., Opler, M.G., Corcoran, C.M., 2010. Potential stigma associated with inclusion of the psychosis risk syndrome in the DSM-V: an empirical question. Schizophr. Res. 120 (1–3), 42–48. Yung, A.R., Buckby, J.A., Cotton, S.M., Cosgrave, E.M., Killackey, E.J., Stanford, C., Godfrey, K., McGorry, P.D., 2006. Psychotic-like experiences in nonpsychotic help-seekers: associations with distress, depression, and disability. Schizophr. Bull. 32 (2), 352–359.