- Email: [email protected]
Factorial validity of problematic Internet use scales
Computers in Human Behavior 25 (2009) 1335–1342
Contents lists available at ScienceDirect
Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh
Factorial validity of problematic Internet use scales Ronnie Jia a,*, Heather H. Jia b,1 a b
Southern Illinois University, Carbondale, IL 62901, USA Eastern Illinois University, Charleston, IL 61920, USA
a r t i c l e
i n f o
Article history: Available online 15 July 2009 Keywords: Problematic Internet use Internet addiction Computer attitudes Validation Factor analysis
a b s t r a c t There exists a number of multidimensional measurement scales for problematic Internet use (PIU) with varying factor structures. This study reviews the factor analytic techniques used to develop these measures and discusses their implications for the factorial validity, particularly discriminant validity, of these PIU scales. To further illustrate these points, we reformulate the four-factor Online Cognition Scale into a more parsimonious two-factor measure (i.e., dependency and distraction) and demonstrate its factorial validity as well as robustness across student and working adult samples. Contributions of this research are discussed. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Research in the past decade or so has documented varying types and degrees of dysfunctional use of the Internet. Though different labels have been used for the phenomenon (e.g., Internet addiction, Young, 1996; Internet dependency, Scherer, 1997; problematic Internet use, Davis, 2001; Davis, Flett, & Besser, 2002), there is a general agreement in the literature over the nature of the phenomenon itself (Chou, Condron, & Belland, 2005; Davis et al., 2002) along with its various manifestations (e.g., Beard & Wolf, 2001; Chou, 2001; Chou et al., 2005; Kraut et al., 1998; Nalwa & Anand, 2003; Treuer, Fabian, & Furedi, 2001; Whang, Lee, & Chang, 2003) as well as psychological and occupational consequences (e.g., Brenner, 1997; Davis et al., 2002; Kraut et al., 1998; Widyanto & McMurran, 2004; Young, 1996). As one of the first theoretical frameworks in this area, Davis (2001) proposed a cognitive-behavioral view that conceptualizes problematic Internet use (PIU) as behaviors and cognitions associated with Internet use that result in negative personal and professional consequences for the user. A number of PIU measurement scales have been developed, including six multidimensional measures2: the four-factor Online Cognition Scales (OCS; Davis et al., 2002), the seven-factor Generalized Problematic Internet Use Scale (GPIUS; Caplan, 2002), the fivefactor Internet Addiction Scale for Taiwanese High School Students (IAST; Lin & Tsai, 2002), and three more recent three-factor scales: Problematic Internet Usage Questionnaire (PIUQ; Thatcher &
* Corresponding author. Tel.: +1 618 453 7253; fax: +1 618 453 7254. E-mail addresses: [email protected] (R. Jia), [email protected] (H.H. Jia). 1 Tel.: +1 217 581 6381; fax: +1 217 581 6642. 2 Unidimensional measures, e.g., Young’s (1996) Internet Addition Test, are outside the scope of our discussion. 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.06.004
Goolam, 2005), Problematic Internet Usage Scale (PIUS; Ceyhan, Ceyhan, & Gürcan, 2007), and the revised Internet Addiction Test (revised IAT; Chang & Law, 2008). Factorial complexity of these measures varies widely, ranging from as few as three to as complex as seven, which prompts the question: What are the causes of such diverse factor structures across studies? There are at least two causes for such diverse factor structures for the PIU construct. First, the construct itself has not been uniformly defined across studies. Though there has been some attempts at theory building (e.g., Davis, 2001), there is still a lack of commonly adopted construct definition or theoretical view. Achieving a consensus definition is a critical step before its true factor structure can be discovered because the definition would determine the domain of the construct and the content of the item pool (Tobacyk, 1995). Since this is an nascent area of research where much work is still descriptive in nature, achieving a unified view requires more cumulative research, which is beyond what can be achieved in one study like this. Another potential cause for the wide variations in PIU factor structure is methodological in nature – the factor analytic techniques and decision heuristics used in developing these scales can have a direct impact on the factor structure obtained. For example, certain frequently used statistical criteria in factor analysis have recently been found to lead to significant overfactoring (i.e., selecting too many factors) in cognitive ability tests (Frazier & Youngstrom, 2007). If methodological issues are not addressed, more PIU scales of diverse factor structures are likely to be developed in future research even after the adoption of a consensus definition, making it difficult to build a cumulative tradition in this literature. Thus, this study focuses on the factor analytic issues associated with PIU scale development. Because instrument validation is inherently a process driven by theoretical/conceptual as
1336
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
well as empirical considerations, rigorous methodological approaches can in turn inform our effort toward a consensus definition. In this paper, we review the factor analytic techniques and decision rules used in the development of the existing multidimensional PIU measures and explore their implications to factorial validity, particularly discriminant validity. To further illustrate these points, using empirical data from two samples (students and working adults), the four-factor Online Cognition Scales will be reformulated and validated as a factorially more parsimonious, but psychometrically more satisfactory measure. The paper concludes with a summary of our findings and contributions. We first introduce the existing multidimensional PIU measures.
2. Existing multidimensional PIU measures Based on the cognitive-behavioral view (Davis, 2001), Davis et al. (2002) developed one of the first multidimensional measure for the PIU construct, called the Online Cognition Scale (OCS). Consisting of four factors (i.e., loneliness/depression, diminished impulse control, social comfort, and distraction), the 36-item instrument was tested using confirmatory factor analysis. Also built on the cognitive-behavioral view was the 29-item Generalized Problematic Internet Use Scale (GPIUS; Caplan, 2002), which consists of seven factors extracted from exploratory factor analysis, including mood alteration, perceived social benefits, negative outcomes, compulsive use, excessive time online, withdrawal, and perceived social control. Based on diagnostic criteria for Internet addiction in prior literature, Lin and Tsai (1999, 2002) developed the five-factor Internet Addiction Scale for Taiwanese High School Students (IAST) using principle component analysis. Consisting of 29 items, the five factors include tolerance, compulsive use, withdrawal, related family, school and health problems, and related interpersonal and financial problems. Building on Young’s (1996) Internet addiction criteria and other relevant literature, Thatcher and Goolam (2005) offered a three-factor, 20-item Problematic Internet Usage Questionnaire (PIUQ) based on a principle component analysis. The three factors are online preoccupation, adverse effects, and social interactions. Also using principle component analysis was Ceyhan et al.’s (2007) three-factor Problematic Internet Usage Scale (PIUS), consisting of negative consequences, social benefit/social comfort, and excessive use. Most recently, in a reexamination of Young’s (1996) single-factor Internet Addiction Test, Chang and Law (2008) extracted three principle components from the measure, including withdrawal and social problems, time management and performance, and reality substitute. The reformulated three-factor IAT was subsequently assessed with confirmatory factor analysis. Table 1 summarizes the main characteristics of these six PIU measures, such as factor structures, validation techniques used and results reported. Three types of factor analytic techniques have been employed to develop these six measures: principal components analysis (PCA), exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). Next, we review these methods and their implications for scale development. 2.1. Factor analytic techniques employed We begin with PCA and EFA since five of the six PIU studies have relied on either technique. 2.1.1. PCA and EFA Though both PCA and EFA summarize the relationships between sets of measured variables, letting the data drive the
analysis, the two techniques are conceptually and mathematically distinct (Frazier & Youngstrom, 2007). PCA is used to combine measured variables into a small number of principle components, which are simply linear combinations of the original measured variables, rather than latent factors. Since no distinction is made between common and unique sources of variance in the measured variables, PCA has been described as strictly a data reduction technique (Frazier & Youngstrom, 2007). EFA aims to extract latent (common) factors that could reproduce the correlations among the observed variables based on the assumption that variation in a measured variable is due to variation in the common factor influencing that measured variable. In contrast to PCA, EFA parses unique and common sources of variance and is thus thought to be a more appropriate technique for identifying latent factors (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Gorsuch, 1983; Widaman, 1993). Given these conceptual and mathematical differences, research has shown that principle components loadings tend to be overestimates of corresponding factor loadings (Widaman, 1993). Therefore, though PCA was used in the development of five of the six PIU scales, EFA was the more appropriate tool given the objective of extracting latent factors, rather than data reduction. Other than the choice between PCA and EFA, the determination of the number of factors to retain is probably the most crucial decision in either approach because retaining an incorrect number of factors can compromise the validity of the factor model and the resulting factor loading estimates (Brown, 2006). Having observed significant overfactoring in cognitive ability tests, Frazier and Youngstrom, 2007 pointed out that one major cause for the factorially complex scales is a ‘‘heavy reliance on liberal statistical criteria for determining factor structure,” such as Kaiser’s criterion (i.e., eigenvalue greater than one), Cattell’s scree test, and chi-square statistic resulting from maximum likelihood factor analysis. Though frequently used, these heuristics have been found to lead to significant overfactoring in the case of the Kaiser criterion and the chi-square statistic, or inconsistently recover the true number of factors in the case of scree test (e.g., Fabrigar et al., 1999; Frazier & Youngstrom, 2007). Thus, other more accurate criteria, such as Horn’s parallel analysis (HPA; Horn, 1965) and Minimum Average Partial (MAP; Velicer, 1976) analysis, have been recommended in the psychometric literature as preferred criteria for factor extraction (Frazier & Youngstrom, 2007; Velicer, Eaton, & Fava, 2000; Zwick & Velicer, 1986). Interested readers are referred to these studies for details about these two criteria. However, as elaborated later in this section, regardless of which decision rules are used in EFA/PCA for factor retention, both are exploratory or descriptive in nature, and neither can conclusively establish factorial validity. As authors of the PIUQ (Thatcher & Goolam, 2005) acknowledged, validity is only ‘‘partially established” using exploratory factor analysis (p. 805). It is thus essential to use analytic techniques that are confirmatory in nature. 2.1.2. CFA Confirmatory factor analysis was employed in the development of two of the six PIU measures (i.e., the revised three-factor IAT and the OCS). CFA differs conceptually from EFA and PCA in that the number of factors is specified prior to the analysis (Frazier & Youngstrom, 2007). CFA has been recommended by many psychometricians (e.g., MacCallum, Roznowski, & Necowitz, 1992) because it allows for alternative a priori models differing in factor structure and complexity be specified and evaluated to determine the model with the best fit, and consequently, the number of factors measured by the data (Frazier & Youngstrom, 2007). Such tests of alternative CFA models can be effectively used to assess factorial validity.
1337
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342 Table 1 Existing multidimensional PIU measures. Measure
Authors
Basis
Subjects
Scale length
Factors
Validation technique
Inter-item correlation
Factor loading
Inter-factor correlation
Online Cognition Scale (OCS)
Davis et al. (2002)
Davis’ (2001) cognitivebehavioral model
College students
36
1. 2. 3. 4.
Diminished impulsive control Loneliness/depression Social comfort Distraction
CFA
Not reported
Reported
Reported
Generalized Problematic Internet Use Scale (GPIUS)
Caplan (2002)
Davis’ (2001) cognitivebehavioral model
College students
29
1. 2. 3. 4. 5. 6. 7.
Mood alteration Perceived social benefits Negative outcomes Compulsive use Excessive time online Withdrawal Perceived social control
EFA
Not reported
Not reported
Reported (all < .60)
Internet Addiction Scale for Taiwanese High School Students (IAST)
Lin and Tsai (2002)
Prior literature on diagnostic criteria for Internet addiction
College students
29
1. Tolerance 2. Compulsive use 3. Withdrawal 4. Related problems – family, school and health 5. Related problems – interpersonal and financial
PCA
Not reported
Not reported
Not reported
Problematic Internet Usage Questionnaire (PIUQ)
Thatcher and Goolam (2005)
Young’s (1996) criteria for Internet addiction and other literature
College students; working adults
20
1. Online preoccupation 2. Adverse effects 3. Social interactions
PCA
Not reported
Reported
Reported (all < .63)
Problematic Internet Usage Scale (PIUS)
Ceyhan et al. (2007)
Expert opinions
College students
33
1. Negative consequences 2. Social benefit/social comfort 3. Excessive use
PCA
Not reported
Not reported
Not reported
The Revised Internet Addiction Test (IAT)
Chang and Law (2008)
Young’s (1996) Internet Addiction Test
College students
18
1. Withdrawal and social problems 2. Time management and performance 3. Reality substitute
PCA and CFA
Not reported
Reported
Reported (.83, .88, .88)
Table 2 Appropriate uses of factor analytic techniques. Data reduction
Latent factor extraction
Latent factor/model validation
Exploratory
PCA EFA
Yes No
No Yes
No No
Confirmatory
CFA
No
No
Yes
Table 2 summarizes the appropriate uses of the PCA, EFA, and CFA techniques. The following conclusions can be drawn from the above discussion: 1. Though PCA was frequently used in PIU scale development, EFA was the more appropriate tool given the objective of latent factor extraction, rather than data reduction. 2. Overfactoring is likely a threat to scales developed using EFA/ PCA along with conventional decision heuristics for factor retention. 3. EFA and PCA are exploratory or descriptive in nature and are not effective tools to comprehensively establish instrument validity. 4. CFA is a useful technique because it can evaluate alternative a priori models to identify the factor structure with the best fit. 2.2. Implications for factorial validity of the existing PIU scales Factorial validity typically requires evidence for convergent validity/unidimensionality, and discriminant validity. Though evi-
dence for convergent validity is frequently reported or can be inferred for these six PIU measures, discriminant validity is often overlooked, which at least in part resulted from the heavy use of exploratory techniques like EFA/PCA, which cannot assess discriminant validity. Discriminant validity requires that the factors within a multidimensional measure must be unique from one another. In the context of EFA/PCA, weak discriminant validity is often a result of overfactoring. For example, when measurement items that underlie the same latent factor, Z, are treated as tapping separate factors, Z1 and Z2, the domains of Z1 and Z2 will have considerable overlap, leading to high inter-factor correlation and low discriminant validity between the two. However, inter-factor correlations may not always be high enough to threaten discriminant validity as gauged by prevailing statistical guidelines. In other words, only more severe cases of overfactoring threaten discriminant validity. (However, low discriminant validity is not uniquely related to EFA/PCA studies; one could end up with overlapping factors in a CFA study if not all appropriate tests are performed.) If the use of conventional decision heuristics in exploratory techniques has led to significant overfactoring in cognitive ability tests (Frazier & Youngstrom, 2007), did the same heuristics lead to overfactored PIU scales that were also developed with EFA/ PCA? Without the original datasets used in these studies, it was not possible to reanalyze the data using alternative factor retention criteria to answer this question conclusively. However, four of the six PIU scales have reported relevant results that can be further examined to shed some light on this question. There are two prevailing methods to assess discriminant validity. A straight-forward way, as alluded to above, is to examine the
1338
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
levels of inter-factor correlations. MacKenzie, Podsakoff, and Jarvis (2005) noted that discriminant validity becomes problematic as factor correlations approach .71 (thus 50% of variance is shared). In the absence of the original dataset, this method requires the use of inter-factor correlation matrix (or the inter-item correlation matrix from which the inter-factor correlations can be derived). The other approach is to test a pair of CFA models, one with the two latent factors allowed to freely covary, and the other with their covariance constrained to one. If the unconstrained model represents significantly better fit than the constrained model, then there exists evidence for discriminant validity between the two latent factors. Thus, without the original dataset, this method requires the use of the inter-item correlation matrix. We next examine the factorial validity of the existing PIU scales. Because convergent validity is frequently reported or can be inferred, and is oftentimes less of a concern, the focus of this discussion will be on their discriminant validity. 2.2.1. IAST and PIUS As summarized in Table 1, no inter-item or inter-factor correlation matrix was reported for the five-factor IAST and the three-factor PIUS, both developed with PCA using conventional decision rules. Though overfactoring and discriminant validity can be an issue for both, a reanalysis of the original data with more accurate factor extraction criteria, or future replication studies would be necessary to find conclusive evidence. 2.2.2. GPIUS and PIUQ Moderate levels of inter-factor correlations were reported for the seven-factor GPIUS and the three-factor PIUQ (Table 1), both developed with EFA/PCA techniques using conventional decision rules. Though the degrees of correlations are below MacKenzie et al.’s recommended upper threshold of .71 and thus demonstrates evidence of discriminant validity, possibility of overfactoring in these two measures cannot be ruled out (since discriminant validity only becomes problematic in more severe cases of overfactoring). Similarly, reanalyses of the original data with more accurate factor extraction criteria, or future replication studies are needed. 2.2.3. The revised IAT Using PCA along with Kasier’s criterion and the scree plot, Chang and Law (2008) extracted three principle components from Young’s (1996) single-factor Internet Addiction Test. A subsequent CFA test of the three-factor model found its overall fit to be satisfactory; chi-square difference tests of CFA models also provided some support for discriminant validity (i.e., Dv2 values of 47.05, 4.95, and 8.47, all significant at p = .05 but not at p = .025). However, the high inter-factor correlations (i.e., .83, .88, and .88) suggest considerable overlaps amongst the three principle components extracted (as much as 77.44% of variance), calling into question the adequacy of its discriminant validity based on MacKenzie et al.’s (2005) criterion. This example illustrates how exploratory techniques, when used in conjunction with the conventional factor retention heuristics, can lead to overfactoring and weak discriminant validity. What appear to be conceptually distinct factors may not be empirically distinguishable. One may conclude that Young’s (1996) IAT is indeed a unidimensional measure. 2.2.4. OCS CFA results of the four-factor OCS suggest that the overall model has satisfactory fit (Davis et al., 2002). Though discriminant validity of the scale was not explicitly tested, the inter-factor correlation matrix reported in the study (Table 3) can be examined for evidence.
Table 3 Factor correlations of the OCS instrument (Davis et al., 2002, p. 337). PIU factor
Number of items
LD
DIC
SC
DIS
Loneliness/depression Diminished impulse control Social comfort Distraction
6 10 13 7
1 .71 .70 .59
1 .76 .66
1 .58
1
With correlations ranging from .70 to .76, significant overlaps exist amongst the loneliness/depression (LD), diminished impulse control (DIC), and social comfort (SC) factors. Since discriminant validity becomes problematic as factor correlations approach .71 (MacKenzie et al., 2005), these three factors are not sufficiently unique from one another to exhibit strong discriminant validity. Since they all manifest the user’s dependency on Internet use (which is distinct from the fourth factor, distraction, which refers to the user’s avoidance of responsibilities through Internet use), it can be argued that the measurement items for these three factors tap the same latent construct, which can be labeled dependency. The OCS can thus be reformulated into a two-factor model, including dependency (DEP) and distraction (DIS), subject to empirical validation. In sum, in our examination of the six PIU scales using results reported in these studies, evidence related to overfactoring and/or weak discriminant validity was found for the three-factor revised IAT (which should remain unidimensional as originally proposed) as well as the four-factor OCS (which can be reformulated into a two-factor measure with three overlapping factors combined). Though limited reporting of results for the other four PIU measures prevented an in-depth examination, overfactoring is a possible threat since all four measures were developed with EFA/PCA using conventional factor retention criteria. They need to be further examined in future studies. Frazier and Youngstrom (2007) argued that, in addition to the use of liberal statistical criteria, ‘‘desires to provide clinically useful assessment instruments with greater interpretive value” also contributed to overfactoring in the development of cognitive ability tests. This observation is likely applicable in PIU scale development as well. As Nelson, Canivez, Lindstrom, and Hatt (2007) pointed out, though a possible benefit of a comprehensive measure is its ability to produce unique subtest profile patterns for different populations, which could inform intervention approaches, an instrument consisting of fewer factors can be similarly effective but much more efficient, if the researcher is not concerned about profile analysis. However, regardless of whether a more or less comprehensive measure is preferred in a specific setting, instrument developers must not compromise on the psychometric properties of the measure being developed. Practices contrary to this principle can undermine the purpose of scientific measurement. In the rest of the paper, we use data to illustrate how CFA can be employed to establish factorial validity through validating the new two-factor PIU measure reformulated from Davis et al.’s OCS. Our methodology is discussed next. 3. Methods To validate the reformulated two-factor measure, significant item elimination from the original 36-item OCS instrument was necessary to eliminate redundancy since the newly combined dependency (DEP) scale contained 29 items. Thus, the original OCS instrument was first pilot tested to a small sample of students to identify opportunities for refinement. The refined scales were then distributed to a larger student sample for validation. As with many other PIU measures (Table 1), the OCS has so far only been tested with and applied to student subjects (Davis et al.,
1339
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
2002; Ebeling-Witte, Frank, & Lester, 2007). It is thus important to establish the applicability of the measure to other types of subjects. To that end, the instrument was then subjected to another round of testing with a sample of working adults. This two-sample design will lend us further confidence in the factorial validity and robustness of the measure. All items were measured by 7-point Likert-type scales, ranging from ‘‘Strongly Disagree” to ‘‘Strongly Agree.” The research was approved by human subjects committee. Results from the pilot sample are discussed first.
4.1. Pilot sample A group of 25 undergraduate students from a junior-level class in a public university completed the 36-item PIU instrument, forming the pilot sample. Similar to the correlation patterns in Davis et al.’s study (Table 3), data from this pilot sample showed high correlations amongst items from the previous LD, DIC and SC scales (exceeding 0.7), along with moderate correlations with the DIS items, thus providing initial support for their combination. To reduce redundancy in the DEP and DIS scales and ensure convergent and discriminant validity, items were examined for possible elimination based on their wording and inter-item correlations from this pilot sample as well as Davis et al.’s results. Similarly worded and highly correlated items were dropped as appropriate to reduce redundancy. To ensure convergent and discriminant validity, inter-item correlations were further examined to ensure that each item is more highly correlated with other items within the same scale than with those in the other scale. Throughout the elimination process, it was made sure that the domain coverage of the construct dimensions did not suffer as a result. This process resulted in a refined set of 10 items, including seven items in the Dependency scale and three items in the Distraction scale (Table 4), to be validated with a large sample.
Table 4 The refined two-factor PIU measure. DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 Distraction
DIS03 DIS06 DIS07
An anonymous survey containing the refined PIU instrument was distributed to 288 students in another junior-level undergraduate course, who were invited to participate for extra course credit. In order to assess the nomological validity of the new PIU instrument, the subjects were also asked to indicate the extent to which they engage in the following three types of online activities on scales of 1 (Never) through 7 (All the time): (1) Chat/instant messaging. (2) Visit adult websites. (3) Play online games.
4. Results
Dependency
4.2. Student sample
When I am on the Internet, I often feel a kind of ‘‘rush” or emotional high People complain that I use the Internet too much When I am not online, I often think about the Internet I am less lonely when I am online I feel helpless when I don’t have access to the Internet I am most comfortable online The Internet is more ‘‘real” than real life I find that I go online more when I have something else I am supposed to do I often use the Internet to avoid doing unpleasant things Using the Internet is a way to forget about the things I must do but don’t really want to do
A total of 267 students, including 173 males (65%) and 94 females (35%), returned useable responses. The inter-item correlation matrix is shown in Table 5. 4.2.1. Goodness-of-fit tests of alternative models LISREL 8.80 was used to evaluate the goodness-of-fit of the three alternative CFA models (Models 1–3 in Fig. 1a–c) in relation to the hypothesized two-factor PIU model (Model 4 in Fig. 1d). As shown in Table 6, of all four models evaluated, only the hypothesized second-order model (Model 4 in Fig. 1d) demonstrated satisfactory fit. Fig. 2 presents the estimates of parameters in this model. 4.2.2. Convergent validity/unidimensionality To demonstrate convergent validity/unidimensionality, one single latent variable must underlie each PIU scale. Separate CFA runs were conducted for the two scales. Results in Table 7 suggest an overall good fit for the Dependency scale since four fit indices (CFI, NFI, GFI, and AGFI) were more favorable than the recommended thresholds (see Table 6), though v2/d.f. and RMSEA values were slightly higher than the thresholds. The CFA model for the Distraction scale was saturated because of the number of indicators. The fit indices, calculated from a two-factor model including both scales, met all model fit thresholds. It was thus concluded that the reformulated two-factor PIU measure has demonstrated satisfactory convergent validity/unidimensionality. 4.2.3. Discriminant validity To establish discriminant validity, the two PIU factors must be unique from each other. It was assessed in a pair of LISREL models, one with the two latent constructs allowed to freely covary (unconstrained model), and the other with their covariance constrained to one (constrained model). The unconstrained model represents significantly better fit than the constrained model (Dv2 = 42.90, p < .01). In addition, the correlation between the DEP and DIS factors is .61, below the recommended upper threshold of .71 (MacKenzie et al., 2005). The two-factor instrument thus possesses satisfactory discriminant validity.
Table 5 Inter-item correlation matrix for the student sample (n = 267).
1 2 3 4 5 6 7 8 9 10
DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 DIS03 DIS06 DIS07
1
2
3
4
5
6
7
8
9
10
1 0.503 0.503 0.604 0.392 0.440 0.378 0.205 0.222 0.388
1 0.563 0.402 0.440 0.384 0.441 0.271 0.210 0.474
1 0.386 0.487 0.450 0.463 0.256 0.170 0.399
1 0.302 0.407 0.377 0.201 0.229 0.315
1 0.325 0.385 0.230 0.127 0.216
1 0.337 0.176 0.227 0.318
1 0.174 0.188 0.321
1 0.451 0.492
1 0.469
1
1340
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
a
b Factor 1
Item 1
Item 1
Factor 2
Item 2
Item 2
Factor 3
Item 3
Item 3 PIU
Factor 4
Item 4
Item 4
Factor 5
Item 5
Item 5
Factor 6
Item 6
Item 6
…
…
…
c
d Item 1
Item 1
DEP
DEP
Item 2
Item 2
…
…
PIU
Item 8
Item 8
DIS
DIS
Item 9
Item 9
…
…
Fig. 1. (a) Model 1: Null. (b) Model 2: One first-order factor. (c) Model 3: Uncorrelated two-factors. (d) Model 4: Second-order Model.
Table 6 Goodness-of-fit tests of alternative models (n = 267). Criteria
Threshold
v2 d.f. v2/d.f. RMSEA CFI NFI GFI AGFI
(<3.00) (<0.08) (>0.90) (>0.90) (>0.90) (>0.80)
.47
4.2.5. Nomological validity Though nomological validity is not an aspect of factorial validity, we correlated PIU ratings from the new measure with subjects’ responses to questions regarding the three types of online activities (i.e., chat/instant messaging, visit adult websites, and play online games). As shown in Table 8, PIU is significantly correlated with all three online activities (p < .01), thus demonstrating its nomological validity. Consistent with Davis et al.’s (2002) results, for college students, PIU is more highly related to chat/instant messaging than other online activities. In sum, the reformulated two-factor PIU measure has demonstrated reliability, convergent validity/unidimensionality, discriminant validity and nomological validity with this sample of undergraduate students. To assess its robustness to other types of subjects, the instrument was next tested with a sample of working adults.
.48
4.3. Working adult sample
.46
An anonymous online survey was used to collect data from a diverse sample of working adults. Because the survey concerns problematic usage behavior, data collection through an employersanctioned survey is likely to be subjected to social desirability bias as employees may be reluctant to participate or answer truthfully. Participants were therefore recruited through StudyResponse, a nonprofit online research facilitator at Syracuse University, which maintains a large pool of research participants (over 95,000 individuals as of August 2005). StudyResponse forwarded our email invitation with a link to the online survey to 1000 working adults randomly selected from the participant pool. To encourage participation, the respondents were entered into a random drawing to receive gift certificates from an online merchant. In addition to the PIU scales, the survey also included the same questions regarding the three types of online activities. Useable responses were received from 184 working adults, including 86 males (47%) and 98 females (53%). The average participant was 37 years old (range of 18–68) with 11 years of work experience and a bachelor’s degree. Using the inter-item correlation matrix in Table 9, a CFA test was performed to assess the overall model fit of the instrument.
Model 1 Null
Model 2 One first-order factor
Model 3 Two uncorrelated first-order factors
Model 4 Secondorder Model
1624.43 35 46.41 0.41 0.47 0.46 0.45 0.14
178.94 35 5.11 0.12 0.92 0.90 0.88 0.81
134.60 35 3.85 0.10 0.93 0.91 0.91 0.86
78.03 33 2.36 0.07 0.97 0.95 0.94 0.91
DIC02 .73
DIC04
.72 .74 DEP
.63
DIC06 .61 LD02
.58 .59 .68
.67 LD05
.59 SC01
PIU
SC11
.65 .65 .62
.62
.90 DIS
DIS03 .65
.59 DIS06 .82
.32 DIS07
Fig. 2. Parameter estimates (Model 4, n = 267).
4.2.4. Reliability The new measure has also exhibited satisfactory reliability. Cronbach’s alpha value is .84 for the Dependency scale, .73 for the Distraction scale, and .85 for the overall measure.
1341
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342 Table 7 Convergent validity/unidimensionality (student sample n = 267). Factor
Number of indicators
v2
d.f.
v2/d.f.
RMSEA
CFI
NFI
GFI
AGFI
DEP DIS*
7 3
45.08 78.03
14 34
3.22 2.30
0.091 0.070
0.97 0.97
0.96 0.95
0.95 0.94
0.91 0.91
Note. *This model is saturated because of the number of indicators. Fit indices are thus not available. Fit indices presented here were calculated from a two-factor model including DEP and DIS.
Table 8 Descriptive statistics and correlation matrix for the student sample (n = 267).
1 2 3 4
PIU (refined) Chat/instant messaging Adult websites Online games
Mean
Std. dev.
1
2
3.22 3.46 3.12 3.85
1.04 1.85 1.89 1.79
1 .406 .292 .256
1 .169 .277
3
Table 10 Model goodness-of-fit for the employee sample (n = 184). 4
1 .261
Factor
v2
d.f.
v2/d.f.
RMSEA
CFI
NFI
GFI
AGFI
PIU
89.42
33
2.71
0.097
0.97
0.96
0.91
0.85
1 Table 11 Descriptive statistics and correlation matrix for the employee sample (n = 184).
Results in Table 10 show that the fit was generally satisfactory with four fit indices (CFI, NFI, GFI, and AGFI) more favorable than the thresholds (see Table 6). The v2/d.f. and RMSEA values, though slightly higher than ideal, were deemed acceptable. Cronbach’s alpha reliability for the overall measure was .91 (with the Dependency scale at .88 and the Distraction scale at .87). As shown in Table 11, PIU is significantly correlated (p < .01) with all three types of online activities in the employee sample as well. However, different from undergraduate students, PIU is more highly related to visiting adult websites and playing online games than instant messaging for working adults. These results suggest that problematic use may involve different Internet activities across various user groups. In sum, it was concluded that the new two-factor PIU measure has demonstrated satisfactory validity and robustness across student and working adult samples. 5. Conclusions Six multidimensional PIU measures with different factorial structure and complexity have been developed in the literature using PCA, EFA and CFA methods. After reviewing these techniques and exploring their implications for factorial validity, more extensive use of CFA was recommended for future PIU instrument development since it allows for testing of alternative a priori models and more conclusive assessment of factorial validity. Where an exploratory technique is necessary, EFA will be more appropriate than PCA for the purpose of extracting latent factors, and more accurate factor retention criteria (e.g., MAP, HPA) should be adopted because conventional rules are known to lead to overfactoring (Frazier & Youngstrom, 2007). It is hoped that this study will heighten future researchers’ awareness of these issues.
1 2 3 4
PIU (refined) Chat/instant messaging Adult websites Online games
Mean
Std. dev.
1
2
3
4
3.25 2.34 1.72 1.94
1.40 1.67 1.20 1.33
1 .229 .393 .317
1 .557 .577
1 .568
1
In view of these methodological considerations, the six published PIU measures were examined for their factorial validity, particularly discriminant validity. Other than the four measures whose limited reporting of results prevented close examinations, the remaining two measures were found to be overfactored and/ or to have weak discriminant validity. To illustrate the use of CFA for instrument validation, the fourfactor, 36-item OCS instrument was reformulated into a two-factor measure and validated through a set of CFA tests. This new tenitem PIU measure is not only factorially simpler and more efficient, but also exhibits satisfactory factorial validity. Having tested the instrument with both student and working adult samples, it is also one of the few PIU measures shown to be robust to non-student subjects. As PIU research advances, it is hoped that this new instrument will pave the way for more future studies. Unifying our data analytic approaches is of vital importance in this area of research. Since instrument validation is a process driven by both theory and empirical data, the use of rigorous factor analytic techniques will inform our endeavor toward a consensus view of the PIU phenomenon and contribute to a cumulative tradition in this literature. Acknowledgement This research was supported in part by grants from Pontikes Center for Management of Information at Southern Illinois University.
Table 9 Inter-item correlation matrix for the employee sample (n = 184).
1 2 3 4 5 6 7 8 9 10
DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 DIS03 DIS06 DIS07
1
2
3
4
5
6
7
8
9
10
1 0.476 0.510 0.595 0.449 0.492 0.337 0.477 0.399 0.416
1 0.587 0.452 0.497 0.452 0.396 0.506 0.399 0.479
1 0.569 0.561 0.527 0.474 0.569 0.526 0.519
1 0.545 0.620 0.524 0.468 0.501 0.538
1 0.679 0.408 0.458 0.481 0.425
1 0.493 0.518 0.475 0.391
1 0.516 0.467 0.541
1 0.653 0.723
1 0.717
1
1342
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
References Beard, K. W., & Wolf, E. M. (2001). Modification in the proposed diagnosis criteria for Internet addiction. CyberPsychology & Behavior, 4, 377–383. Brenner, V. (1997). Psychology of computer use: XLVII. Parameters of Internet use, abuse and addiction: The first 90 days of the Internet usage survey. Psychological Reports, 80(87), 9–882. Brown, T. A. (2006). Confirmatory factor analysis for applied research (p. 23). Guilford Press. Caplan, S. E. (2002). Problematic Internet use and psychosocial well-being: Development of a theory-based cognitive-behavioral measurement instrument. Computers in Human Behavior, 18(5), 553–575. Ceyhan, E., Ceyhan, A. A., & Gürcan, A. (2007). The validity and reliability of the problematic Internet usage scale. Educational Science: Theory and Practice, 7(1), 411–416. Chang, M. K., & Law, S. P. M. (2008). Factor structure for Young’s Internet addiction test: A confirmatory study. Computers in Human Behavior, 24, 2597–2619. Chou, C. (2001). Internet heavy use and addiction among Taiwanese college students: An online interview study. CyberPsychology & Behavior, 4, 573–585. Chou, C., Condron, L., & Belland, J. C. (2005). A review of the review on Internet addiction. Educational Psychology Review, 17, 363–368. Davis, R. A. (2001). A cognitive-behavioral model of pathological Internet use. Computers in Human Behavior, 17, 187–195. Davis, R. A., Flett, G. L., & Besser, A. (2002). Validation of a new scale for measuring problematic Internet use: Implications for pre-employment screening. CyberPsychology & Behavior, 5, 331–345. Ebeling-Witte, S., Frank, M. L., & Lester, D. (2007). Shyness, Internet use, and personality. CyberPsychology & Behavior, 10, 713–716. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. Frazier, T. W., & Youngstrom, E. A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169–182. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. Kraut, R., Patterson, M., Lundmark, V., Kiesler, S., Mukopadhyay, T., & Scherlis, W. (1998). Internet paradox: A social technology that reduces social involvement and psychological well-being? American Psychologist, 53, 1017–1031. Lin, S. S. J., & Tsai, C. C. (1999). Internet addiction among high schoolers in Taiwan. Presented at the Annual Meeting of the American Psychological Association. Boston, MA.
Lin, S. S. J., & Tsai, C. C. (2002). Sensation seeking and internet dependence of Taiwanese high school adolescents. Computers in Human Behavior, 18(4), 411–426. MacCallum, R., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504. MacKenzie, S. B., Podsakoff, P. M., & Jarvis, C. B. (2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710–730. Nalwa, K., & Anand, A. P. (2003). Internet addiction in students: A cause of concern. CyberPsychology & Behavior, 6, 653–656. Nelson, J. M., Canivez, G. L., Lindstrom, W. L., & Hatt, C. V. (2007). Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439–456. Scherer, K. (1997). College life on-line: Healthy and unhealthy Internet use. Journal of College Student Development, 38, 655–664. Thatcher, A., & Goolam, S. (2005). Development and psychometric properties of the Problematic Internet Use Questionnaire. South African Journal of Psychology, 35(4), 793–809. Tobacyk, J. J. (1995). Final thoughts on issues in the measurement of paranormal beliefs. Journal of Parapsychology, 59, 141–145. Treuer, T., Fabian, Z., & Furedi, J. (2001). Internet addiction associated with features of impulse control disorder: Is it a real psychiatric disorder? Journal of Affective Disorders, 66, 283. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321–327. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at Seventy (pp. 41 71). Whang, L. S., Lee, S., & Chang, G. (2003). Internet over-users’ psychological profiles: A behavior sampling analysis on Internet addiction. CyberPsychology & Behavior, 6, 143–150. Widaman, K. F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263–311. Widyanto, L., & McMurran, M. (2004). The psychometric properties of the Internet Addiction Test. CyberPsychology & Behavior, 7, 443–450. Young, K. S. (1996). Psychology of computer use: XL. Addictive use of the Internet: A case that breaks the stereotype. Psychological Reports, 79, 899–902. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442.
Contents lists available at ScienceDirect
Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh
Factorial validity of problematic Internet use scales Ronnie Jia a,*, Heather H. Jia b,1 a b
Southern Illinois University, Carbondale, IL 62901, USA Eastern Illinois University, Charleston, IL 61920, USA
a r t i c l e
i n f o
Article history: Available online 15 July 2009 Keywords: Problematic Internet use Internet addiction Computer attitudes Validation Factor analysis
a b s t r a c t There exists a number of multidimensional measurement scales for problematic Internet use (PIU) with varying factor structures. This study reviews the factor analytic techniques used to develop these measures and discusses their implications for the factorial validity, particularly discriminant validity, of these PIU scales. To further illustrate these points, we reformulate the four-factor Online Cognition Scale into a more parsimonious two-factor measure (i.e., dependency and distraction) and demonstrate its factorial validity as well as robustness across student and working adult samples. Contributions of this research are discussed. Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction Research in the past decade or so has documented varying types and degrees of dysfunctional use of the Internet. Though different labels have been used for the phenomenon (e.g., Internet addiction, Young, 1996; Internet dependency, Scherer, 1997; problematic Internet use, Davis, 2001; Davis, Flett, & Besser, 2002), there is a general agreement in the literature over the nature of the phenomenon itself (Chou, Condron, & Belland, 2005; Davis et al., 2002) along with its various manifestations (e.g., Beard & Wolf, 2001; Chou, 2001; Chou et al., 2005; Kraut et al., 1998; Nalwa & Anand, 2003; Treuer, Fabian, & Furedi, 2001; Whang, Lee, & Chang, 2003) as well as psychological and occupational consequences (e.g., Brenner, 1997; Davis et al., 2002; Kraut et al., 1998; Widyanto & McMurran, 2004; Young, 1996). As one of the first theoretical frameworks in this area, Davis (2001) proposed a cognitive-behavioral view that conceptualizes problematic Internet use (PIU) as behaviors and cognitions associated with Internet use that result in negative personal and professional consequences for the user. A number of PIU measurement scales have been developed, including six multidimensional measures2: the four-factor Online Cognition Scales (OCS; Davis et al., 2002), the seven-factor Generalized Problematic Internet Use Scale (GPIUS; Caplan, 2002), the fivefactor Internet Addiction Scale for Taiwanese High School Students (IAST; Lin & Tsai, 2002), and three more recent three-factor scales: Problematic Internet Usage Questionnaire (PIUQ; Thatcher &
* Corresponding author. Tel.: +1 618 453 7253; fax: +1 618 453 7254. E-mail addresses: [email protected] (R. Jia), [email protected] (H.H. Jia). 1 Tel.: +1 217 581 6381; fax: +1 217 581 6642. 2 Unidimensional measures, e.g., Young’s (1996) Internet Addition Test, are outside the scope of our discussion. 0747-5632/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.chb.2009.06.004
Goolam, 2005), Problematic Internet Usage Scale (PIUS; Ceyhan, Ceyhan, & Gürcan, 2007), and the revised Internet Addiction Test (revised IAT; Chang & Law, 2008). Factorial complexity of these measures varies widely, ranging from as few as three to as complex as seven, which prompts the question: What are the causes of such diverse factor structures across studies? There are at least two causes for such diverse factor structures for the PIU construct. First, the construct itself has not been uniformly defined across studies. Though there has been some attempts at theory building (e.g., Davis, 2001), there is still a lack of commonly adopted construct definition or theoretical view. Achieving a consensus definition is a critical step before its true factor structure can be discovered because the definition would determine the domain of the construct and the content of the item pool (Tobacyk, 1995). Since this is an nascent area of research where much work is still descriptive in nature, achieving a unified view requires more cumulative research, which is beyond what can be achieved in one study like this. Another potential cause for the wide variations in PIU factor structure is methodological in nature – the factor analytic techniques and decision heuristics used in developing these scales can have a direct impact on the factor structure obtained. For example, certain frequently used statistical criteria in factor analysis have recently been found to lead to significant overfactoring (i.e., selecting too many factors) in cognitive ability tests (Frazier & Youngstrom, 2007). If methodological issues are not addressed, more PIU scales of diverse factor structures are likely to be developed in future research even after the adoption of a consensus definition, making it difficult to build a cumulative tradition in this literature. Thus, this study focuses on the factor analytic issues associated with PIU scale development. Because instrument validation is inherently a process driven by theoretical/conceptual as
1336
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
well as empirical considerations, rigorous methodological approaches can in turn inform our effort toward a consensus definition. In this paper, we review the factor analytic techniques and decision rules used in the development of the existing multidimensional PIU measures and explore their implications to factorial validity, particularly discriminant validity. To further illustrate these points, using empirical data from two samples (students and working adults), the four-factor Online Cognition Scales will be reformulated and validated as a factorially more parsimonious, but psychometrically more satisfactory measure. The paper concludes with a summary of our findings and contributions. We first introduce the existing multidimensional PIU measures.
2. Existing multidimensional PIU measures Based on the cognitive-behavioral view (Davis, 2001), Davis et al. (2002) developed one of the first multidimensional measure for the PIU construct, called the Online Cognition Scale (OCS). Consisting of four factors (i.e., loneliness/depression, diminished impulse control, social comfort, and distraction), the 36-item instrument was tested using confirmatory factor analysis. Also built on the cognitive-behavioral view was the 29-item Generalized Problematic Internet Use Scale (GPIUS; Caplan, 2002), which consists of seven factors extracted from exploratory factor analysis, including mood alteration, perceived social benefits, negative outcomes, compulsive use, excessive time online, withdrawal, and perceived social control. Based on diagnostic criteria for Internet addiction in prior literature, Lin and Tsai (1999, 2002) developed the five-factor Internet Addiction Scale for Taiwanese High School Students (IAST) using principle component analysis. Consisting of 29 items, the five factors include tolerance, compulsive use, withdrawal, related family, school and health problems, and related interpersonal and financial problems. Building on Young’s (1996) Internet addiction criteria and other relevant literature, Thatcher and Goolam (2005) offered a three-factor, 20-item Problematic Internet Usage Questionnaire (PIUQ) based on a principle component analysis. The three factors are online preoccupation, adverse effects, and social interactions. Also using principle component analysis was Ceyhan et al.’s (2007) three-factor Problematic Internet Usage Scale (PIUS), consisting of negative consequences, social benefit/social comfort, and excessive use. Most recently, in a reexamination of Young’s (1996) single-factor Internet Addiction Test, Chang and Law (2008) extracted three principle components from the measure, including withdrawal and social problems, time management and performance, and reality substitute. The reformulated three-factor IAT was subsequently assessed with confirmatory factor analysis. Table 1 summarizes the main characteristics of these six PIU measures, such as factor structures, validation techniques used and results reported. Three types of factor analytic techniques have been employed to develop these six measures: principal components analysis (PCA), exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). Next, we review these methods and their implications for scale development. 2.1. Factor analytic techniques employed We begin with PCA and EFA since five of the six PIU studies have relied on either technique. 2.1.1. PCA and EFA Though both PCA and EFA summarize the relationships between sets of measured variables, letting the data drive the
analysis, the two techniques are conceptually and mathematically distinct (Frazier & Youngstrom, 2007). PCA is used to combine measured variables into a small number of principle components, which are simply linear combinations of the original measured variables, rather than latent factors. Since no distinction is made between common and unique sources of variance in the measured variables, PCA has been described as strictly a data reduction technique (Frazier & Youngstrom, 2007). EFA aims to extract latent (common) factors that could reproduce the correlations among the observed variables based on the assumption that variation in a measured variable is due to variation in the common factor influencing that measured variable. In contrast to PCA, EFA parses unique and common sources of variance and is thus thought to be a more appropriate technique for identifying latent factors (Fabrigar, Wegener, MacCallum, & Strahan, 1999; Gorsuch, 1983; Widaman, 1993). Given these conceptual and mathematical differences, research has shown that principle components loadings tend to be overestimates of corresponding factor loadings (Widaman, 1993). Therefore, though PCA was used in the development of five of the six PIU scales, EFA was the more appropriate tool given the objective of extracting latent factors, rather than data reduction. Other than the choice between PCA and EFA, the determination of the number of factors to retain is probably the most crucial decision in either approach because retaining an incorrect number of factors can compromise the validity of the factor model and the resulting factor loading estimates (Brown, 2006). Having observed significant overfactoring in cognitive ability tests, Frazier and Youngstrom, 2007 pointed out that one major cause for the factorially complex scales is a ‘‘heavy reliance on liberal statistical criteria for determining factor structure,” such as Kaiser’s criterion (i.e., eigenvalue greater than one), Cattell’s scree test, and chi-square statistic resulting from maximum likelihood factor analysis. Though frequently used, these heuristics have been found to lead to significant overfactoring in the case of the Kaiser criterion and the chi-square statistic, or inconsistently recover the true number of factors in the case of scree test (e.g., Fabrigar et al., 1999; Frazier & Youngstrom, 2007). Thus, other more accurate criteria, such as Horn’s parallel analysis (HPA; Horn, 1965) and Minimum Average Partial (MAP; Velicer, 1976) analysis, have been recommended in the psychometric literature as preferred criteria for factor extraction (Frazier & Youngstrom, 2007; Velicer, Eaton, & Fava, 2000; Zwick & Velicer, 1986). Interested readers are referred to these studies for details about these two criteria. However, as elaborated later in this section, regardless of which decision rules are used in EFA/PCA for factor retention, both are exploratory or descriptive in nature, and neither can conclusively establish factorial validity. As authors of the PIUQ (Thatcher & Goolam, 2005) acknowledged, validity is only ‘‘partially established” using exploratory factor analysis (p. 805). It is thus essential to use analytic techniques that are confirmatory in nature. 2.1.2. CFA Confirmatory factor analysis was employed in the development of two of the six PIU measures (i.e., the revised three-factor IAT and the OCS). CFA differs conceptually from EFA and PCA in that the number of factors is specified prior to the analysis (Frazier & Youngstrom, 2007). CFA has been recommended by many psychometricians (e.g., MacCallum, Roznowski, & Necowitz, 1992) because it allows for alternative a priori models differing in factor structure and complexity be specified and evaluated to determine the model with the best fit, and consequently, the number of factors measured by the data (Frazier & Youngstrom, 2007). Such tests of alternative CFA models can be effectively used to assess factorial validity.
1337
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342 Table 1 Existing multidimensional PIU measures. Measure
Authors
Basis
Subjects
Scale length
Factors
Validation technique
Inter-item correlation
Factor loading
Inter-factor correlation
Online Cognition Scale (OCS)
Davis et al. (2002)
Davis’ (2001) cognitivebehavioral model
College students
36
1. 2. 3. 4.
Diminished impulsive control Loneliness/depression Social comfort Distraction
CFA
Not reported
Reported
Reported
Generalized Problematic Internet Use Scale (GPIUS)
Caplan (2002)
Davis’ (2001) cognitivebehavioral model
College students
29
1. 2. 3. 4. 5. 6. 7.
Mood alteration Perceived social benefits Negative outcomes Compulsive use Excessive time online Withdrawal Perceived social control
EFA
Not reported
Not reported
Reported (all < .60)
Internet Addiction Scale for Taiwanese High School Students (IAST)
Lin and Tsai (2002)
Prior literature on diagnostic criteria for Internet addiction
College students
29
1. Tolerance 2. Compulsive use 3. Withdrawal 4. Related problems – family, school and health 5. Related problems – interpersonal and financial
PCA
Not reported
Not reported
Not reported
Problematic Internet Usage Questionnaire (PIUQ)
Thatcher and Goolam (2005)
Young’s (1996) criteria for Internet addiction and other literature
College students; working adults
20
1. Online preoccupation 2. Adverse effects 3. Social interactions
PCA
Not reported
Reported
Reported (all < .63)
Problematic Internet Usage Scale (PIUS)
Ceyhan et al. (2007)
Expert opinions
College students
33
1. Negative consequences 2. Social benefit/social comfort 3. Excessive use
PCA
Not reported
Not reported
Not reported
The Revised Internet Addiction Test (IAT)
Chang and Law (2008)
Young’s (1996) Internet Addiction Test
College students
18
1. Withdrawal and social problems 2. Time management and performance 3. Reality substitute
PCA and CFA
Not reported
Reported
Reported (.83, .88, .88)
Table 2 Appropriate uses of factor analytic techniques. Data reduction
Latent factor extraction
Latent factor/model validation
Exploratory
PCA EFA
Yes No
No Yes
No No
Confirmatory
CFA
No
No
Yes
Table 2 summarizes the appropriate uses of the PCA, EFA, and CFA techniques. The following conclusions can be drawn from the above discussion: 1. Though PCA was frequently used in PIU scale development, EFA was the more appropriate tool given the objective of latent factor extraction, rather than data reduction. 2. Overfactoring is likely a threat to scales developed using EFA/ PCA along with conventional decision heuristics for factor retention. 3. EFA and PCA are exploratory or descriptive in nature and are not effective tools to comprehensively establish instrument validity. 4. CFA is a useful technique because it can evaluate alternative a priori models to identify the factor structure with the best fit. 2.2. Implications for factorial validity of the existing PIU scales Factorial validity typically requires evidence for convergent validity/unidimensionality, and discriminant validity. Though evi-
dence for convergent validity is frequently reported or can be inferred for these six PIU measures, discriminant validity is often overlooked, which at least in part resulted from the heavy use of exploratory techniques like EFA/PCA, which cannot assess discriminant validity. Discriminant validity requires that the factors within a multidimensional measure must be unique from one another. In the context of EFA/PCA, weak discriminant validity is often a result of overfactoring. For example, when measurement items that underlie the same latent factor, Z, are treated as tapping separate factors, Z1 and Z2, the domains of Z1 and Z2 will have considerable overlap, leading to high inter-factor correlation and low discriminant validity between the two. However, inter-factor correlations may not always be high enough to threaten discriminant validity as gauged by prevailing statistical guidelines. In other words, only more severe cases of overfactoring threaten discriminant validity. (However, low discriminant validity is not uniquely related to EFA/PCA studies; one could end up with overlapping factors in a CFA study if not all appropriate tests are performed.) If the use of conventional decision heuristics in exploratory techniques has led to significant overfactoring in cognitive ability tests (Frazier & Youngstrom, 2007), did the same heuristics lead to overfactored PIU scales that were also developed with EFA/ PCA? Without the original datasets used in these studies, it was not possible to reanalyze the data using alternative factor retention criteria to answer this question conclusively. However, four of the six PIU scales have reported relevant results that can be further examined to shed some light on this question. There are two prevailing methods to assess discriminant validity. A straight-forward way, as alluded to above, is to examine the
1338
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
levels of inter-factor correlations. MacKenzie, Podsakoff, and Jarvis (2005) noted that discriminant validity becomes problematic as factor correlations approach .71 (thus 50% of variance is shared). In the absence of the original dataset, this method requires the use of inter-factor correlation matrix (or the inter-item correlation matrix from which the inter-factor correlations can be derived). The other approach is to test a pair of CFA models, one with the two latent factors allowed to freely covary, and the other with their covariance constrained to one. If the unconstrained model represents significantly better fit than the constrained model, then there exists evidence for discriminant validity between the two latent factors. Thus, without the original dataset, this method requires the use of the inter-item correlation matrix. We next examine the factorial validity of the existing PIU scales. Because convergent validity is frequently reported or can be inferred, and is oftentimes less of a concern, the focus of this discussion will be on their discriminant validity. 2.2.1. IAST and PIUS As summarized in Table 1, no inter-item or inter-factor correlation matrix was reported for the five-factor IAST and the three-factor PIUS, both developed with PCA using conventional decision rules. Though overfactoring and discriminant validity can be an issue for both, a reanalysis of the original data with more accurate factor extraction criteria, or future replication studies would be necessary to find conclusive evidence. 2.2.2. GPIUS and PIUQ Moderate levels of inter-factor correlations were reported for the seven-factor GPIUS and the three-factor PIUQ (Table 1), both developed with EFA/PCA techniques using conventional decision rules. Though the degrees of correlations are below MacKenzie et al.’s recommended upper threshold of .71 and thus demonstrates evidence of discriminant validity, possibility of overfactoring in these two measures cannot be ruled out (since discriminant validity only becomes problematic in more severe cases of overfactoring). Similarly, reanalyses of the original data with more accurate factor extraction criteria, or future replication studies are needed. 2.2.3. The revised IAT Using PCA along with Kasier’s criterion and the scree plot, Chang and Law (2008) extracted three principle components from Young’s (1996) single-factor Internet Addiction Test. A subsequent CFA test of the three-factor model found its overall fit to be satisfactory; chi-square difference tests of CFA models also provided some support for discriminant validity (i.e., Dv2 values of 47.05, 4.95, and 8.47, all significant at p = .05 but not at p = .025). However, the high inter-factor correlations (i.e., .83, .88, and .88) suggest considerable overlaps amongst the three principle components extracted (as much as 77.44% of variance), calling into question the adequacy of its discriminant validity based on MacKenzie et al.’s (2005) criterion. This example illustrates how exploratory techniques, when used in conjunction with the conventional factor retention heuristics, can lead to overfactoring and weak discriminant validity. What appear to be conceptually distinct factors may not be empirically distinguishable. One may conclude that Young’s (1996) IAT is indeed a unidimensional measure. 2.2.4. OCS CFA results of the four-factor OCS suggest that the overall model has satisfactory fit (Davis et al., 2002). Though discriminant validity of the scale was not explicitly tested, the inter-factor correlation matrix reported in the study (Table 3) can be examined for evidence.
Table 3 Factor correlations of the OCS instrument (Davis et al., 2002, p. 337). PIU factor
Number of items
LD
DIC
SC
DIS
Loneliness/depression Diminished impulse control Social comfort Distraction
6 10 13 7
1 .71 .70 .59
1 .76 .66
1 .58
1
With correlations ranging from .70 to .76, significant overlaps exist amongst the loneliness/depression (LD), diminished impulse control (DIC), and social comfort (SC) factors. Since discriminant validity becomes problematic as factor correlations approach .71 (MacKenzie et al., 2005), these three factors are not sufficiently unique from one another to exhibit strong discriminant validity. Since they all manifest the user’s dependency on Internet use (which is distinct from the fourth factor, distraction, which refers to the user’s avoidance of responsibilities through Internet use), it can be argued that the measurement items for these three factors tap the same latent construct, which can be labeled dependency. The OCS can thus be reformulated into a two-factor model, including dependency (DEP) and distraction (DIS), subject to empirical validation. In sum, in our examination of the six PIU scales using results reported in these studies, evidence related to overfactoring and/or weak discriminant validity was found for the three-factor revised IAT (which should remain unidimensional as originally proposed) as well as the four-factor OCS (which can be reformulated into a two-factor measure with three overlapping factors combined). Though limited reporting of results for the other four PIU measures prevented an in-depth examination, overfactoring is a possible threat since all four measures were developed with EFA/PCA using conventional factor retention criteria. They need to be further examined in future studies. Frazier and Youngstrom (2007) argued that, in addition to the use of liberal statistical criteria, ‘‘desires to provide clinically useful assessment instruments with greater interpretive value” also contributed to overfactoring in the development of cognitive ability tests. This observation is likely applicable in PIU scale development as well. As Nelson, Canivez, Lindstrom, and Hatt (2007) pointed out, though a possible benefit of a comprehensive measure is its ability to produce unique subtest profile patterns for different populations, which could inform intervention approaches, an instrument consisting of fewer factors can be similarly effective but much more efficient, if the researcher is not concerned about profile analysis. However, regardless of whether a more or less comprehensive measure is preferred in a specific setting, instrument developers must not compromise on the psychometric properties of the measure being developed. Practices contrary to this principle can undermine the purpose of scientific measurement. In the rest of the paper, we use data to illustrate how CFA can be employed to establish factorial validity through validating the new two-factor PIU measure reformulated from Davis et al.’s OCS. Our methodology is discussed next. 3. Methods To validate the reformulated two-factor measure, significant item elimination from the original 36-item OCS instrument was necessary to eliminate redundancy since the newly combined dependency (DEP) scale contained 29 items. Thus, the original OCS instrument was first pilot tested to a small sample of students to identify opportunities for refinement. The refined scales were then distributed to a larger student sample for validation. As with many other PIU measures (Table 1), the OCS has so far only been tested with and applied to student subjects (Davis et al.,
1339
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
2002; Ebeling-Witte, Frank, & Lester, 2007). It is thus important to establish the applicability of the measure to other types of subjects. To that end, the instrument was then subjected to another round of testing with a sample of working adults. This two-sample design will lend us further confidence in the factorial validity and robustness of the measure. All items were measured by 7-point Likert-type scales, ranging from ‘‘Strongly Disagree” to ‘‘Strongly Agree.” The research was approved by human subjects committee. Results from the pilot sample are discussed first.
4.1. Pilot sample A group of 25 undergraduate students from a junior-level class in a public university completed the 36-item PIU instrument, forming the pilot sample. Similar to the correlation patterns in Davis et al.’s study (Table 3), data from this pilot sample showed high correlations amongst items from the previous LD, DIC and SC scales (exceeding 0.7), along with moderate correlations with the DIS items, thus providing initial support for their combination. To reduce redundancy in the DEP and DIS scales and ensure convergent and discriminant validity, items were examined for possible elimination based on their wording and inter-item correlations from this pilot sample as well as Davis et al.’s results. Similarly worded and highly correlated items were dropped as appropriate to reduce redundancy. To ensure convergent and discriminant validity, inter-item correlations were further examined to ensure that each item is more highly correlated with other items within the same scale than with those in the other scale. Throughout the elimination process, it was made sure that the domain coverage of the construct dimensions did not suffer as a result. This process resulted in a refined set of 10 items, including seven items in the Dependency scale and three items in the Distraction scale (Table 4), to be validated with a large sample.
Table 4 The refined two-factor PIU measure. DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 Distraction
DIS03 DIS06 DIS07
An anonymous survey containing the refined PIU instrument was distributed to 288 students in another junior-level undergraduate course, who were invited to participate for extra course credit. In order to assess the nomological validity of the new PIU instrument, the subjects were also asked to indicate the extent to which they engage in the following three types of online activities on scales of 1 (Never) through 7 (All the time): (1) Chat/instant messaging. (2) Visit adult websites. (3) Play online games.
4. Results
Dependency
4.2. Student sample
When I am on the Internet, I often feel a kind of ‘‘rush” or emotional high People complain that I use the Internet too much When I am not online, I often think about the Internet I am less lonely when I am online I feel helpless when I don’t have access to the Internet I am most comfortable online The Internet is more ‘‘real” than real life I find that I go online more when I have something else I am supposed to do I often use the Internet to avoid doing unpleasant things Using the Internet is a way to forget about the things I must do but don’t really want to do
A total of 267 students, including 173 males (65%) and 94 females (35%), returned useable responses. The inter-item correlation matrix is shown in Table 5. 4.2.1. Goodness-of-fit tests of alternative models LISREL 8.80 was used to evaluate the goodness-of-fit of the three alternative CFA models (Models 1–3 in Fig. 1a–c) in relation to the hypothesized two-factor PIU model (Model 4 in Fig. 1d). As shown in Table 6, of all four models evaluated, only the hypothesized second-order model (Model 4 in Fig. 1d) demonstrated satisfactory fit. Fig. 2 presents the estimates of parameters in this model. 4.2.2. Convergent validity/unidimensionality To demonstrate convergent validity/unidimensionality, one single latent variable must underlie each PIU scale. Separate CFA runs were conducted for the two scales. Results in Table 7 suggest an overall good fit for the Dependency scale since four fit indices (CFI, NFI, GFI, and AGFI) were more favorable than the recommended thresholds (see Table 6), though v2/d.f. and RMSEA values were slightly higher than the thresholds. The CFA model for the Distraction scale was saturated because of the number of indicators. The fit indices, calculated from a two-factor model including both scales, met all model fit thresholds. It was thus concluded that the reformulated two-factor PIU measure has demonstrated satisfactory convergent validity/unidimensionality. 4.2.3. Discriminant validity To establish discriminant validity, the two PIU factors must be unique from each other. It was assessed in a pair of LISREL models, one with the two latent constructs allowed to freely covary (unconstrained model), and the other with their covariance constrained to one (constrained model). The unconstrained model represents significantly better fit than the constrained model (Dv2 = 42.90, p < .01). In addition, the correlation between the DEP and DIS factors is .61, below the recommended upper threshold of .71 (MacKenzie et al., 2005). The two-factor instrument thus possesses satisfactory discriminant validity.
Table 5 Inter-item correlation matrix for the student sample (n = 267).
1 2 3 4 5 6 7 8 9 10
DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 DIS03 DIS06 DIS07
1
2
3
4
5
6
7
8
9
10
1 0.503 0.503 0.604 0.392 0.440 0.378 0.205 0.222 0.388
1 0.563 0.402 0.440 0.384 0.441 0.271 0.210 0.474
1 0.386 0.487 0.450 0.463 0.256 0.170 0.399
1 0.302 0.407 0.377 0.201 0.229 0.315
1 0.325 0.385 0.230 0.127 0.216
1 0.337 0.176 0.227 0.318
1 0.174 0.188 0.321
1 0.451 0.492
1 0.469
1
1340
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
a
b Factor 1
Item 1
Item 1
Factor 2
Item 2
Item 2
Factor 3
Item 3
Item 3 PIU
Factor 4
Item 4
Item 4
Factor 5
Item 5
Item 5
Factor 6
Item 6
Item 6
…
…
…
c
d Item 1
Item 1
DEP
DEP
Item 2
Item 2
…
…
PIU
Item 8
Item 8
DIS
DIS
Item 9
Item 9
…
…
Fig. 1. (a) Model 1: Null. (b) Model 2: One first-order factor. (c) Model 3: Uncorrelated two-factors. (d) Model 4: Second-order Model.
Table 6 Goodness-of-fit tests of alternative models (n = 267). Criteria
Threshold
v2 d.f. v2/d.f. RMSEA CFI NFI GFI AGFI
(<3.00) (<0.08) (>0.90) (>0.90) (>0.90) (>0.80)
.47
4.2.5. Nomological validity Though nomological validity is not an aspect of factorial validity, we correlated PIU ratings from the new measure with subjects’ responses to questions regarding the three types of online activities (i.e., chat/instant messaging, visit adult websites, and play online games). As shown in Table 8, PIU is significantly correlated with all three online activities (p < .01), thus demonstrating its nomological validity. Consistent with Davis et al.’s (2002) results, for college students, PIU is more highly related to chat/instant messaging than other online activities. In sum, the reformulated two-factor PIU measure has demonstrated reliability, convergent validity/unidimensionality, discriminant validity and nomological validity with this sample of undergraduate students. To assess its robustness to other types of subjects, the instrument was next tested with a sample of working adults.
.48
4.3. Working adult sample
.46
An anonymous online survey was used to collect data from a diverse sample of working adults. Because the survey concerns problematic usage behavior, data collection through an employersanctioned survey is likely to be subjected to social desirability bias as employees may be reluctant to participate or answer truthfully. Participants were therefore recruited through StudyResponse, a nonprofit online research facilitator at Syracuse University, which maintains a large pool of research participants (over 95,000 individuals as of August 2005). StudyResponse forwarded our email invitation with a link to the online survey to 1000 working adults randomly selected from the participant pool. To encourage participation, the respondents were entered into a random drawing to receive gift certificates from an online merchant. In addition to the PIU scales, the survey also included the same questions regarding the three types of online activities. Useable responses were received from 184 working adults, including 86 males (47%) and 98 females (53%). The average participant was 37 years old (range of 18–68) with 11 years of work experience and a bachelor’s degree. Using the inter-item correlation matrix in Table 9, a CFA test was performed to assess the overall model fit of the instrument.
Model 1 Null
Model 2 One first-order factor
Model 3 Two uncorrelated first-order factors
Model 4 Secondorder Model
1624.43 35 46.41 0.41 0.47 0.46 0.45 0.14
178.94 35 5.11 0.12 0.92 0.90 0.88 0.81
134.60 35 3.85 0.10 0.93 0.91 0.91 0.86
78.03 33 2.36 0.07 0.97 0.95 0.94 0.91
DIC02 .73
DIC04
.72 .74 DEP
.63
DIC06 .61 LD02
.58 .59 .68
.67 LD05
.59 SC01
PIU
SC11
.65 .65 .62
.62
.90 DIS
DIS03 .65
.59 DIS06 .82
.32 DIS07
Fig. 2. Parameter estimates (Model 4, n = 267).
4.2.4. Reliability The new measure has also exhibited satisfactory reliability. Cronbach’s alpha value is .84 for the Dependency scale, .73 for the Distraction scale, and .85 for the overall measure.
1341
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342 Table 7 Convergent validity/unidimensionality (student sample n = 267). Factor
Number of indicators
v2
d.f.
v2/d.f.
RMSEA
CFI
NFI
GFI
AGFI
DEP DIS*
7 3
45.08 78.03
14 34
3.22 2.30
0.091 0.070
0.97 0.97
0.96 0.95
0.95 0.94
0.91 0.91
Note. *This model is saturated because of the number of indicators. Fit indices are thus not available. Fit indices presented here were calculated from a two-factor model including DEP and DIS.
Table 8 Descriptive statistics and correlation matrix for the student sample (n = 267).
1 2 3 4
PIU (refined) Chat/instant messaging Adult websites Online games
Mean
Std. dev.
1
2
3.22 3.46 3.12 3.85
1.04 1.85 1.89 1.79
1 .406 .292 .256
1 .169 .277
3
Table 10 Model goodness-of-fit for the employee sample (n = 184). 4
1 .261
Factor
v2
d.f.
v2/d.f.
RMSEA
CFI
NFI
GFI
AGFI
PIU
89.42
33
2.71
0.097
0.97
0.96
0.91
0.85
1 Table 11 Descriptive statistics and correlation matrix for the employee sample (n = 184).
Results in Table 10 show that the fit was generally satisfactory with four fit indices (CFI, NFI, GFI, and AGFI) more favorable than the thresholds (see Table 6). The v2/d.f. and RMSEA values, though slightly higher than ideal, were deemed acceptable. Cronbach’s alpha reliability for the overall measure was .91 (with the Dependency scale at .88 and the Distraction scale at .87). As shown in Table 11, PIU is significantly correlated (p < .01) with all three types of online activities in the employee sample as well. However, different from undergraduate students, PIU is more highly related to visiting adult websites and playing online games than instant messaging for working adults. These results suggest that problematic use may involve different Internet activities across various user groups. In sum, it was concluded that the new two-factor PIU measure has demonstrated satisfactory validity and robustness across student and working adult samples. 5. Conclusions Six multidimensional PIU measures with different factorial structure and complexity have been developed in the literature using PCA, EFA and CFA methods. After reviewing these techniques and exploring their implications for factorial validity, more extensive use of CFA was recommended for future PIU instrument development since it allows for testing of alternative a priori models and more conclusive assessment of factorial validity. Where an exploratory technique is necessary, EFA will be more appropriate than PCA for the purpose of extracting latent factors, and more accurate factor retention criteria (e.g., MAP, HPA) should be adopted because conventional rules are known to lead to overfactoring (Frazier & Youngstrom, 2007). It is hoped that this study will heighten future researchers’ awareness of these issues.
1 2 3 4
PIU (refined) Chat/instant messaging Adult websites Online games
Mean
Std. dev.
1
2
3
4
3.25 2.34 1.72 1.94
1.40 1.67 1.20 1.33
1 .229 .393 .317
1 .557 .577
1 .568
1
In view of these methodological considerations, the six published PIU measures were examined for their factorial validity, particularly discriminant validity. Other than the four measures whose limited reporting of results prevented close examinations, the remaining two measures were found to be overfactored and/ or to have weak discriminant validity. To illustrate the use of CFA for instrument validation, the fourfactor, 36-item OCS instrument was reformulated into a two-factor measure and validated through a set of CFA tests. This new tenitem PIU measure is not only factorially simpler and more efficient, but also exhibits satisfactory factorial validity. Having tested the instrument with both student and working adult samples, it is also one of the few PIU measures shown to be robust to non-student subjects. As PIU research advances, it is hoped that this new instrument will pave the way for more future studies. Unifying our data analytic approaches is of vital importance in this area of research. Since instrument validation is a process driven by both theory and empirical data, the use of rigorous factor analytic techniques will inform our endeavor toward a consensus view of the PIU phenomenon and contribute to a cumulative tradition in this literature. Acknowledgement This research was supported in part by grants from Pontikes Center for Management of Information at Southern Illinois University.
Table 9 Inter-item correlation matrix for the employee sample (n = 184).
1 2 3 4 5 6 7 8 9 10
DIC02 DIC04 DIC06 LD02 LD05 SC01 SC11 DIS03 DIS06 DIS07
1
2
3
4
5
6
7
8
9
10
1 0.476 0.510 0.595 0.449 0.492 0.337 0.477 0.399 0.416
1 0.587 0.452 0.497 0.452 0.396 0.506 0.399 0.479
1 0.569 0.561 0.527 0.474 0.569 0.526 0.519
1 0.545 0.620 0.524 0.468 0.501 0.538
1 0.679 0.408 0.458 0.481 0.425
1 0.493 0.518 0.475 0.391
1 0.516 0.467 0.541
1 0.653 0.723
1 0.717
1
1342
R. Jia, H.H. Jia / Computers in Human Behavior 25 (2009) 1335–1342
References Beard, K. W., & Wolf, E. M. (2001). Modification in the proposed diagnosis criteria for Internet addiction. CyberPsychology & Behavior, 4, 377–383. Brenner, V. (1997). Psychology of computer use: XLVII. Parameters of Internet use, abuse and addiction: The first 90 days of the Internet usage survey. Psychological Reports, 80(87), 9–882. Brown, T. A. (2006). Confirmatory factor analysis for applied research (p. 23). Guilford Press. Caplan, S. E. (2002). Problematic Internet use and psychosocial well-being: Development of a theory-based cognitive-behavioral measurement instrument. Computers in Human Behavior, 18(5), 553–575. Ceyhan, E., Ceyhan, A. A., & Gürcan, A. (2007). The validity and reliability of the problematic Internet usage scale. Educational Science: Theory and Practice, 7(1), 411–416. Chang, M. K., & Law, S. P. M. (2008). Factor structure for Young’s Internet addiction test: A confirmatory study. Computers in Human Behavior, 24, 2597–2619. Chou, C. (2001). Internet heavy use and addiction among Taiwanese college students: An online interview study. CyberPsychology & Behavior, 4, 573–585. Chou, C., Condron, L., & Belland, J. C. (2005). A review of the review on Internet addiction. Educational Psychology Review, 17, 363–368. Davis, R. A. (2001). A cognitive-behavioral model of pathological Internet use. Computers in Human Behavior, 17, 187–195. Davis, R. A., Flett, G. L., & Besser, A. (2002). Validation of a new scale for measuring problematic Internet use: Implications for pre-employment screening. CyberPsychology & Behavior, 5, 331–345. Ebeling-Witte, S., Frank, M. L., & Lester, D. (2007). Shyness, Internet use, and personality. CyberPsychology & Behavior, 10, 713–716. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. Frazier, T. W., & Youngstrom, E. A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169–182. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. Kraut, R., Patterson, M., Lundmark, V., Kiesler, S., Mukopadhyay, T., & Scherlis, W. (1998). Internet paradox: A social technology that reduces social involvement and psychological well-being? American Psychologist, 53, 1017–1031. Lin, S. S. J., & Tsai, C. C. (1999). Internet addiction among high schoolers in Taiwan. Presented at the Annual Meeting of the American Psychological Association. Boston, MA.
Lin, S. S. J., & Tsai, C. C. (2002). Sensation seeking and internet dependence of Taiwanese high school adolescents. Computers in Human Behavior, 18(4), 411–426. MacCallum, R., Roznowski, M., & Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111(3), 490–504. MacKenzie, S. B., Podsakoff, P. M., & Jarvis, C. B. (2005). The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. Journal of Applied Psychology, 90, 710–730. Nalwa, K., & Anand, A. P. (2003). Internet addiction in students: A cause of concern. CyberPsychology & Behavior, 6, 653–656. Nelson, J. M., Canivez, G. L., Lindstrom, W. L., & Hatt, C. V. (2007). Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439–456. Scherer, K. (1997). College life on-line: Healthy and unhealthy Internet use. Journal of College Student Development, 38, 655–664. Thatcher, A., & Goolam, S. (2005). Development and psychometric properties of the Problematic Internet Use Questionnaire. South African Journal of Psychology, 35(4), 793–809. Tobacyk, J. J. (1995). Final thoughts on issues in the measurement of paranormal beliefs. Journal of Parapsychology, 59, 141–145. Treuer, T., Fabian, Z., & Furedi, J. (2001). Internet addiction associated with features of impulse control disorder: Is it a real psychiatric disorder? Journal of Affective Disorders, 66, 283. Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321–327. Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at Seventy (pp. 41 71). Whang, L. S., Lee, S., & Chang, G. (2003). Internet over-users’ psychological profiles: A behavior sampling analysis on Internet addiction. CyberPsychology & Behavior, 6, 143–150. Widaman, K. F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263–311. Widyanto, L., & McMurran, M. (2004). The psychometric properties of the Internet Addiction Test. CyberPsychology & Behavior, 7, 443–450. Young, K. S. (1996). Psychology of computer use: XL. Addictive use of the Internet: A case that breaks the stereotype. Psychological Reports, 79, 899–902. Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432–442.